Senior Azure Databricks Engineer

Senior Azure Databricks Engineer

About the role:

We are looking for a Senior Azure Databricks Engineer with strong hands-on experience in both Databricks development and platform operations.

The ideal candidate should have a good end-to-end understanding of Databricks, from developing Python/PySpark solutions to configuring clusters, managing jobs, and supporting production environments.

Responsibilities

Design, develop, and optimize data pipelines using Azure Databricks, Python, PySpark, Spark SQL, and Delta Lake.

Develop and maintain Databricks notebooks, workflows, and scheduled jobs.

Configure and manage Databricks clusters, cluster policies, autoscaling, runtime versions, and libraries.

Support Databricks workspace configuration, infrastructure management, and platform operations.

Implement Unity Catalog access controls, data governance, and data lineage.

Support Azure security, Key Vault integration, and secrets management.

Implement CI/CD pipelines and automate Databricks deployments.

Monitor and troubleshoot Databricks jobs, clusters, connectivity, and production issues.

Optimize Spark performance, compute resources, and platform costs.

Collaborate with Data Engineering, DevOps, and Infrastructure teams.

Requirements

5+ years of experience in Data Engineering or cloud data platforms, including at least 3 years with Azure Databricks.

Strong Python fundamentals, including data structures, functions, error handling, and problem-solving.

Strong hands-on experience with Azure Databricks, PySpark, Spark SQL, SQL, ADLS Gen2, and Delta Lake.

Practical experience developing Databricks notebooks, workflows, and job scheduling.

Hands-on experience with Databricks cluster management, autoscaling, runtime configuration, and cluster policies.

Experience with Unity Catalog, access management, and data lineage implementation and monitoring.

Good understanding of Databricks infrastructure, workspace configuration, and Azure cloud services.

Experience with CI/CD pipelines, deployment automation, and DevOps practices.

Knowledge of Azure networking, security, Key Vault, and secrets management.

Experience troubleshooting production Databricks jobs, clusters, and platform issues.

Experience with Terraform for provisioning, configuring, and managing Azure Databricks infrastructure, including workspaces, clusters, Unity Catalog, and related Azure resources.

Good understanding of Spark architecture and performance optimization.

Strong communication and problem-solving skills.

Nice to Have

Experience with Bicep, ARM Templates, or other Infrastructure as Code tools.

Experience with Databricks Asset Bundles (DABs).

Experience with Azure Monitor, Log Analytics, and cost optimization.

Experience provisioning Databricks workspaces from scratch.

Experience in banking, insurance, or other regulated industries.

Azure or Databricks certifications.