Data Engineer Lead

Role & responsibilities

About the Role

This role will be responsible for architecting and delivering high-performance, governed data environments for large-scale enterprise clientsdriving data reliability, compliance, and analytics readiness.

Key Responsibilities

• Design and implement scalable, reusable data pipelines using

  Databricks (PySpark, Delta Lake)

  and

  Azure Data Factory

  with high throughput and low latency.
• Develop and maintain

  Common Data Platforms (CDP)

  and

  Medallion Architecture

  –based data lakes to unify legacy systems and improve data accessibility.
• Build backend data services using

  Python, Spark, or C#

  , ensuring performance, reliability, and scalability.
• Create metadata-driven, schema-evolution–capable data ingestion frameworks supporting multi-source integration.
• Lead data migration and integration initiatives using

  Talend, Ab Initio, Alteryx, and Informatica

  , modernizing legacy ETL processes into cloud-native pipelines.
• Implement data quality, lineage, and governance frameworks such as

  Unity Catalog, Data Stewardship, Data Catalog, and MDM

  .
• Ensure data processing is compliant with global privacy standards including

  GDPR and CCPA

  .
• Build and optimize

  Feature Store Data Marts

  to support AI/ML and BI teams.
• Collaborate with cross-functional teams, including Data Scientists, BI Developers, and Product Owners.
• Mentor junior engineers in data engineering best practices, CI/CD, and IaC principles.
• Utilize

  Redis

  and caching strategies to enhance performance and reduce analytic workload latency.
• Drive end-to-end ownership of assigned modules including delivery, documentation, and performance optimization.

Preferred candidate profile

• Cloud Platforms:

  Azure (ADF, Synapse, ADLS, Databricks), AWS (S3, EC2, Glue, Redshift)

• Data Engineering Tools:

  Databricks (DLT, Auto Loader, Unity Catalog), Talend, Ab Initio, Informatica, Alteryx

• Languages:

  Python, PySpark, SQL, Unix Shell Scripting

• Data Modeling:

  Dimensional, Relational, Medallion Architecture

• CI/CD & DevOps:

  GitHub Actions, Azure DevOps, Databricks Asset Bundles (DABs)

• Governance & Security:

  RBAC/ABAC, MDM, GDPR/CCPA compliance

• Orchestration & Monitoring:

  Airflow, Logic Apps, Azure Monitor

Required Qualifications

• 12+ years of experience in

  Data Engineering, Data Architecture, or ETL Development

  .
• Strong leadership experience in Azure + Databricks modernization/migration programs.
• Hands-on experience with

  Talend, Informatica, Ab Initio, Alteryx

  and metadata-driven pipeline frameworks.
• Proficiency in

  PySpark, SQL, Azure Data Services

  .
• Strong knowledge of data modeling, data quality, and governance.
• Experience with real-time and batch ingestion (Kafka, Kinesis, Event Hub).
• Excellent communication, problem-solving, and stakeholder management skills.

Preferred Qualifications

• Bachelor’s degree in Computer Science or similar field.
• Azure Data Engineer Certification.
• Experience with AI/ML data preparation and Feature Store architectures.
• Experience with Databricks cluster optimization and cost governance.