Azure Databricks

As an Azure Databricks Engineer, your primary role will involve designing, developing, and optimizing end-to-end ETL/ELT pipelines on Azure Databricks using PySpark and Delta Lake to support analytics and ML use-cases. You will be responsible for implementing scalable data ingestion patterns from batch and streaming sources, integrating with Azure Data Factory, ADLS Gen2, and event streams. Additionally, you will tune Spark jobs, manage cluster configurations, and drive cost-performance optimizations for production workloads. Your tasks will also include developing Databricks SQL models, notebooks, and CI/CD pipelines to automate deployment, testing, and versioning of data artifacts. Collaboration with data scientists, analysts, and platform teams will be crucial to operationalize ML training/inference and ensure data quality, lineage, and governance. You will create monitoring, observability, and runbook procedures for job failures, SLAs, and performance alerts. Key Responsibilities: - Design, develop, and optimize ETL/ELT pipelines using PySpark and Delta Lake on Azure Databricks - Implement scalable data ingestion patterns from batch and streaming sources - Integrate with Azure Data Factory, ADLS Gen2, and event streams - Tune Spark jobs, manage cluster configurations, and optimize cost-performance for production workloads - Develop Databricks SQL models, notebooks, and CI/CD pipelines for automation - Collaborate with cross-functional teams to operationalize ML training/inference and ensure data quality and governance - Create monitoring and observability procedures for job failures, SLAs, and performance alerts Qualifications Required: Must-Have: - Azure Databricks experience - Apache Spark and PySpark proficiency - Delta Lake knowledge - Familiarity with Azure Data Factory and Databricks SQL Preferred: - Experience with Terraform - Knowledge of Azure DevOps/CI-CD for Databricks - Exposure to streaming platforms like Kafka/Event Hubs In addition to the technical skills and responsibilities, you will have the opportunity to work on enterprise-scale data platforms and influence architecture decisions. You will be part of a collaborative, fast-paced environment with exposure to analytics and ML teams. The company offers competitive compensation and a clear growth path for data engineering professionals.,