Lead Azure Databricks Data Engineer

Location: Bhopal, MP, India

Experience: 7+ Years

Key Responsibilities

• Design and build ETL pipelines in Azure Databricks (PySpark, Delta Lake) to load, clean, and deliver data across Bronze, Silver, and Gold layers.
• Implement Data Lakehouse Architecture on Azure Data Lake Gen2 with partitioning, schema management, and performance optimization.
• Develop data models (dimensional/star schema) for reporting in Synapse and Power BI.
• Integrate Databricks with Azure services – ADF, Key Vault, Event Hub, Synapse Analytics, Purview, and Logic Apps.
• Build and manage CI/CD pipelines in Azure DevOps (YAML, Git repos, pipelines).
• Optimize performance through cluster tuning, caching, Z-ordering, Delta optimization, and job parallelization.
• Ensure data security and compliance (row-level security, PII masking, GDPR/HIPAA, audit logging).
• Collaborate with data architects and analysts to translate business needs into technical solutions.

Required Skills

• Strong experience in Azure Databricks (Python, PySpark, SQL).
• Proficiency with Delta Lake (ACID transactions, schema evolution, incremental loads).
• Hands-on with Azure ecosystem – Data Factory, ADLS Gen2, Key Vault, Event Hub, Synapse.
• Knowledge of data governance & lineage tools (Purview, Unity Catalog is a plus).
• Strong understanding of data warehouse design and star schema.
• Azure DevOps (YAML, Git repos, pipelines) experience.
• Good debugging skills for performance tuning & schema drift issues.

Good to Have

• Experience with healthcare or financial data.
• Familiarity with FHIR, OMOP, OpenEHR (for healthcare projects).
• Exposure to AI/ML integration using Databricks ML runtime.
• Experience with Unity Catalog for governance across workspaces.

Deliverables / Outcomes

• Automated snapshot + incremental pipelines in Databricks.
• Delta Lake architecture with partitioning, Z-ordering, and schema drift handling.
• Metadata-driven ingestion framework (YAML configs).
• Power BI datasets connected to Gold layer.
• CI/CD pipelines for deployment across.

Are you ready to take the lead in building scalable data solutions with Azure Databricks? Apply now!

Job Type: Full-time

Pay: From ₹20,000.34 per month

Work Location: In person