Data Engineer Lead

Job Overview

Senior Data Engineer

PySpark, Databricks, AWS Glue, S3, and SQL

Key Responsibilities

• Leadership & Project Management

• Lead and mentor a team of data engineers across multiple data initiatives.
  
• Manage end-to-end delivery of projects, ensuring adherence to timelines and quality standards.
  
• Collaborate with cross-functional teams including data scientists, BI developers, analysts, and business stakeholders.
  

• Data Pipeline Development

• Design, develop, and maintain

  ETL/ELT pipelines

  using

  Databricks, PySpark, AWS Glue

  , and

  S3

  for ingesting, transforming, and storing large-scale data.
• Implement best practices for pipeline reliability, scalability, and performance optimization.
  

• Data Architecture & Modeling

• Design and document data models, schemas, and metadata standards for consistent and reusable data structures.
  
• Architect cloud-native, scalable data solutions supporting both batch and streaming data.
  

• Data Quality & Governance

• Develop and implement robust

  data validation, testing, and quality control

  mechanisms.
• Ensure all data and processes comply with internal governance, security, and compliance frameworks (e.g., GxP, GDPR).
  

• Monitoring & Optimization

• Continuously monitor pipeline performance and proactively resolve bottlenecks or failures.
  
• Implement performance tuning, cost optimization, and automation where applicable.
  

• Collaboration & Stakeholder Communication

• Translate complex technical concepts into clear business insights for non-technical stakeholders.
  
• Identify potential risks and communicate mitigation plans effectively.
  

• Technical Innovation

• Stay updated on the latest data engineering technologies and best practices.
  
• Recommend and adopt new tools or approaches to improve data platform efficiency and capabilities.
  

Required Skills & Qualifications

• Experience:

  Minimum 6+ years in Data Engineering with hands-on experience in

  Big Data

  and

  Cloud

  environments.

• Technical Expertise:

• Strong proficiency in

  PySpark, AWS Glue, S3, SQL, and Python

  .
• Proven experience with

  Databricks-based architectures

  .
• Expertise in developing, optimizing, and maintaining scalable ETL/ELT pipelines.
  

• Certifications:

• Mandatory:

  AWS Certified Data Analytics or Databricks Certified Data Engineer.

• Domain Knowledge:

  Experience in

  pharmaceutical data

  is a strong plus.