Lead Data Engineer

Job Summary

We are looking for a skilled Databricks Developer with experience in data engineering, data transformation, and building scalable data pipelines using Azure Databricks and Apache Spark.

Key Responsibilities

• Design, develop, and maintain data pipelines using Azure Databricks and PySpark.
• Develop ETL/ELT processes for data ingestion, transformation, and integration.
• Work with Python, SQL, Apache Spark, and Delta Lake.
• Implement data lakehouse solutions and optimize data processing performance.
• Integrate data from multiple sources and ensure data quality.
• Collaborate with data engineers, architects, and business teams to deliver data solutions.
• Troubleshoot pipeline failures and optimize jobs for performance and cost.
• Implement data security, governance, and access controls.

Required Skills

• Hands-on experience with Azure Databricks and PySpark.
• Strong knowledge of Python and SQL.
• Experience with Apache Spark, Delta Lake, and ETL/ELT pipelines.
• Understanding data warehousing and lakehouse architecture.
• Experience with Azure Data Factory, ADLS, or other cloud data services.
• Knowledge of Git, CI/CD, and workflow orchestration.
• Strong analytical and problem-solving skills.

Preferred Qualifications

• Experience with Databricks Unity Catalog and Delta Live Tables (Lakeflow Declarative Pipelines).
• Knowledge of Azure cloud services and data governance.
• Experience in performance tuning and production deployment of data pipelines.