Senior Data Engineer (Azure Databricks / PySpark)

We are looking for a hands-on Data Engineer to design, develop, and optimize data pipelines on Azure Databricks, converting Informatica and Kalido logic into scalable PySpark notebooks.

• Convert source-to-target mappings into

  PySpark Databricks notebooks

  .
• Build

  end-to-end data pipelines

  using ADF + Databricks Jobs with proper logging and monitoring.
• Optimize code for

  performance and cost-efficiency

  (cluster sizing, caching).
• Implement

  unit tests and data validation

  to ensure like-to-like logic migration.
• Collaborate with DevOps to enable

  CI/CD pipelines

  and version control.
• Support SIT/UAT defect resolution and production cutover readiness.

Required Skills:

• 5+ years data engineering experience (2+ years on Azure Databricks).
• Expert in

  PySpark, Delta Lake, and SQL performance tuning

  .
• Strong knowledge of

  ADF, ADLS Gen2, and data orchestration patterns

  .
• Experience with

  Git, code reviews, and Agile development

  .
• Familiarity with

  ETL conversion projects

  (Informatica to cloud preferred).