Data Engineer – Azure Databricks

Job Description

We are looking for a skilled Data Engineer with Azure Databricks expertise to design and develop scalable data pipelines and architectures using cloud-native tools. You will work closely with data scientists, analysts, and business teams to deliver high-quality data solutions that drive insights and innovation. Responsibilities Design and implement ETL/ELT pipelines on Azure Databricks using PySpark and Spark SQL. Integrate data from diverse sources into Azure Data Lake Storage (ADLS Gen2) and structure it for analytics in Delta Lake. Develop notebooks, workflows, and jobs in Databricks Workspaces. Collaborate with data architects to model data warehouse/lakehouse structures (Delta Lake). Optimize data processing jobs for cost and performance. Implement CI/CD pipelines for Databricks using Azure DevOps or GitHub Actions. Ensure data quality, security, and compliance across data pipelines. Use Unity Catalog, Azure Purview, or Data Catalog for data governance. Requirements 3–6+ years of data engineering experience, with 2+ years on Azure Databricks. Proficiency in PySpark, SQL, and Spark. Strong experience with Azure Data Lake, Azure Data Factory, and Delta Lake. Familiarity with Azure Synapse Analytics, Event Hubs, and Stream Analytics is a plus. Knowledge of Databricks cluster configuration, job orchestration, and monitoring. Experience with CI/CD, Terraform, or Infrastructure as Code (IaC) is a bonus. Azure Data Engineer Associate certification is a plus. Job Type: Full-time Pay: ₹800,000.00 - ₹1,000,000.00 per year Work Location: In person