Data Engineer Lead

About the Role

Data Engineer Lead

high-performance, governed data architectures

Key Responsibilities

• Design and implement

  scalable and reusable

  data pipelines

  using

  Databricks (PySpark, Delta Lake)

  and

  Azure Data Factory

  , ensuring high throughput and low latency.
• Develop and maintain

  Common Data Platforms (CDP)

  and

  Medallion Architecture-based data lakes

  to consolidate legacy systems and improve data availability.
• Create and manage

  backend data services

  using

  Python, Spark, or C#

  , ensuring performance, reliability, and scalability.

• Develop data ingestion frameworks

  using metadata-driven and schema-drift–handling designs to support multi-source data integration.
• Lead

  data integration and migration projects

  using

  Talend, Ab Initio, Alteryx, and Informatica

  , modernizing traditional ETL workflows into cloud-native pipelines.

• Implement data quality, lineage, and governance frameworks

  (Unity Catalog, Data Stewardship, Data Catalog, MDM).
• Design and maintain

  GDPR/CCPA-compliant data processing

  and ensure adherence to global data privacy standards.
• Build and optimize

  Feature Store Data Marts

  to support AI/ML analytics teams and downstream BI consumers.
• Collaborate with cross-functional teams (Data Scientists, BI Developers, Product Owners) to define and deliver data-driven business outcomes.
• Mentor junior engineers in

  best practices of data engineering

  , CI/CD, and infrastructure-as-code deployments.
• Utilize

  Redis

  and caching mechanisms to optimize performance and reduce load latency in analytic workloads.
• Take end-to-end ownership of assigned modules ensuring timely delivery, documentation, and performance optimization.

Technical Environment

• Cloud Platforms:

  Microsoft Azure (Data Factory, Synapse, Data Lake, Databricks), AWS (S3, EC2, Glue, Redshift)

• Data Engineering Tools:

  Databricks (DLT, Auto Loader, Unity Catalog), Talend, Ab Initio, Informatica, Alteryx

• Programming Languages:

  Python, PySpark, SQL, Unix Shell Scripting

• Data Modeling:

  Dimensional, Relational, and Medallion architecture

• DevOps / CI-CD:

  GitHub Actions, Azure DevOps, Databricks Asset Bundles (DABs)

• Data Governance & Security:

  RBAC/ABAC, Data Stewardship, MDM, GDPR/CCPA compliance

• Monitoring & Scheduling:

  Airflow, Logic Apps, Azure Monitor

Required Qualifications

• 12+ years of experience in

  Data Engineering, Data Architecture, or ETL Development

  .
• Proven experience leading

  Azure + Databricks

  –based modernization or migration programs.
• Deep understanding of

  data integration tools

  (Talend, Informatica, Ab Initio, Alteryx) and

  metadata-driven pipeline design

  .
• Hands-on experience with

  PySpark, SQL, and Azure Data Services

  .
• Strong understanding of

  data modeling, data quality, and governance frameworks

  .
• Experience with

  real-time and batch data ingestion

  , including Kafka, Kinesis, or Event Hub.
• Excellent problem-solving, communication, and stakeholder management skills.

Preferred Qualifications

• Bachelor’s degree in computer science, or a related field
• Strong analytical and problem-solving skills
• Azure Data Engineer Certification or equivalent
• Familiarity with

  AI/ML data preparation and Feature Store design

• Experience implementing

  cost optimization and cluster governance

  on Databricks

Why Join Us

• Be part of a

  data modernization journey

  with a global enterprise brand.
• Work with cutting-edge

  cloud and data engineering tools

  .
• Collaborate with a

  highly skilled, cross-functional team

  of architects, analysts, and data scientists.
• Opportunity to lead and innovate within a large-scale, multi-domain

  data transformation initiative

  .