Lead Data Engineer

Role & responsibilities

Data Engineer Lead

The ideal candidate will combine hands-on technical leadership with strong domain understanding, governance awareness, and stakeholder collaboration.

Key Responsibilities

Technical Leadership

• Lead the 

  architecture, design, and implementation

   of enterprise-scale data pipelines using 

  Databricks (Spark, Delta Lake, Unity Catalog)

  .
• Own end-to-end data engineering solutions including 

  ingestion, transformation, orchestration, and serving layers

  .
• Define best practices for 

  data modeling, performance optimization, cost optimization, and scalability

  .
• Review and guide development of PySpark, SQL, and Delta Lake implementations.
• Ensure adherence to 

  data quality, lineage, and observability standards

  .

Platform & Engineering

• Build and maintain 

  batch and streaming pipelines

   using Databricks, Kafka/Event Hubs, and cloud-native services.
• Integrate diverse data sources: clinical trials, EHR/EMR, claims, pharmacovigilance, lab data, commercial and third-party datasets.
• Implement CI/CD pipelines for data engineering workloads.
• Partner with cloud teams on 

  security, networking, and infrastructure optimization

  .

Domain & Compliance

• Apply pharma/healthcare domain knowledge across:
  
• Clinical Development
• Medical Affairs
• Pharmacovigilance
• Commercial Analytics
• Real World Evidence (RWE)
• Ensure compliance with 

  GxP, HIPAA, GDPR, and data privacy regulations

  .
• Collaborate with governance teams on metadata management, access controls, and audit readiness.

Stakeholder & Team Leadership

• Lead and mentor a team of data engineers (onshore/offshore).
• Work closely with 

  data scientists, analysts, product owners, and business stakeholders

  .
• Translate business requirements into technical data solutions.
• Provide effort estimation, delivery planning, and risk management.

Required Skills & Qualifications

Technical Skills

• Strong hands-on experience with 

  Databricks

   (Spark, Delta Lake, Unity Catalog).
• Advanced proficiency in 

  PySpark and SQL

  .
• Experience with 

  cloud platforms

  : Azure (preferred), AWS, or GCP.
• Knowledge of 

  data orchestration tools

   (ADF, Airflow, Dagster, etc.).
• Experience with 

  data warehousing concepts

   and lakehouse architectures.
• Familiarity with 

  streaming technologies

   (Kafka, Event Hubs, Spark Streaming).
• Strong understanding of 

  data security, encryption, and access control models

  .

Domain Experience

• 2+ years of experience in 

  Pharmaceutical or Healthcare data environments

  .
• Hands-on exposure to 

  clinical, safety, medical, or commercial datasets

  .
• Understanding of 

  regulated environments and validation requirements

  .

Leadership & Soft Skills

• Proven experience leading data engineering teams.
• Strong communication and stakeholder management skills.
• Ability to balance 

  hands-on development with strategic oversight

  .
• Experience working in 

  Agile / SAFe delivery models

  .