Associate Manager-Data Engineer

Job Summary

Associate Manager Data Engineering

Key Responsibilities

Data Architecture & Engineering

• Architect and implement end-to-end data solutions using

  Databricks, Apache Spark, Delta Lake

  , and cloud platforms (

  AWS, Azure, GCP

  )
• Lead data migration and transformation from legacy systems to modern

  Lakehouse architectures

• Design, build, and optimize data pipelines using

  PySpark, SQL, and Databricks Workflows

• Implement scalable and high-performance data models and ETL frameworks
  

Governance, Security & Quality

• Ensure strong data governance, security, and compliance using

  Unity Catalog, RBAC

  , and best practices
• Implement observability, monitoring, and performance optimization strategies
  
• Maintain high standards of data quality, reliability, and documentation
  

Technical Leadership & Collaboration

• Collaborate with business and technical stakeholders to gather requirements and translate them into technical designs
  
• Provide technical leadership, code reviews, and mentorship to data engineering teams
  
• Guide teams in

  CI/CD, DevOps, and DataOps

  practices using

  Databricks Asset Bundles

  or similar tools
• Evaluate emerging technologies and contribute to continuous improvement and innovation
  

Delivery & Enablement

• Create and maintain technical documentation and training materials
  
• Support post-deployment adoption and ongoing enhancements
  
• Communicate progress, risks, and outcomes to stakeholders and leadership
  

Required Skills & Qualifications

Primary Skills

• Databricks

• Python

• SQL

• Data Modelling / Data Modeling

Secondary Skills

• PySpark

• ETL / Data Pipelines

• Spark, Delta Lake, DLT

• Lakehouse Architecture
  

Desired Profile

• 7+ years

  of experience in Data Engineering or Cloud Architecture

• 3+ years of hands-on Databricks experience

• Experience working with

  commercial datasets

  such as

  IQVIA sales data, Veeva CRM, HCP profiles, Marketing datasets

• Strong proficiency in

  Python, PySpark, SQL

  (Scala is a plus)
• Experience with cloud platforms:

  AWS, Azure, or GCP

• Strong understanding of

  CI/CD, DevOps, and DataOps workflows

• Familiarity with data governance, observability, and performance tuning
  

• Databricks certifications

  (Data Engineer Associate/Professional, Lakehouse Architect) preferred
• Exposure to

  MLflow, TensorFlow, or ML frameworks

  is an added advantage
• Excellent communication, leadership, and stakeholder management skills
  

Nice to Have

• Experience leading or mentoring teams
  
• Exposure to advanced analytics or machine learning pipelines
  
• Experience in pharma / healthcare / commercial analytics domains