Azure Databricks

As an Azure Databricks Engineer at our fast-growing cloud data and analytics consultancy, you will play a crucial role in designing high-throughput lakehouse platforms, predictive analytics, and AI services on Microsoft Azure. Your responsibilities will include: - Designing, developing, and optimizing end-to-end data pipelines on Azure Databricks using PySpark/Scala and Delta Lake. - Building scalable ETL workflows to ingest structured and semi-structured data from Azure Data Lake, SQL, and API sources. - Implementing lakehouse architectures, partitioning, and performance tuning to ensure sub-second query response. - Collaborating with Data Scientists to prepare feature stores and accelerate model training and inference. - Automating deployment with Azure DevOps, ARM/Bicep, and Databricks CLI for secure, repeatable releases. - Monitoring pipeline health, cost, and governance, applying best practices for security, lineage, and data quality. Qualifications required for this role include: - 3+ years of experience building large-scale Spark or Databricks workloads in production. - Expertise in PySpark/Scala, Delta Lake, and SQL optimization. - Deep knowledge of Azure services including Data Lake Storage Gen2, Data Factory/Synapse, Key Vault, and Event Hub. - Proficiency in CI/CD, Git, and automated testing for data engineering. - Understanding of data modeling, partitioning, and performance tuning strategies. Preferred qualifications include exposure to MLflow, feature store design, or predictive model serving, experience implementing role-based access controls and GDPR/PCI compliance on Azure, and certification such as Microsoft DP-203 or Databricks Data Engineer Professional. Joining our team will offer you the opportunity to work on cutting-edge Azure Databricks projects with Fortune 500 clients. You will be part of a flat, learning-centric culture that supports certifications and conference passes. Additionally, we offer a hybrid leave policy, comprehensive health cover, and performance bonuses.,