Senior Databricks Administrator (Azure Expertise)

Job Description

We are seeking a highly skilled Senior Databricks Administrator with extensive experience in managing and optimizing Databricks environments on Microsoft Azure. The ideal candidate will be responsible for administration, performance tuning, security, and troubleshooting of Databricks clusters, ensuring seamless integration with Azure cloud services. This role requires deep expertise in Azure Databricks, Spark, and cloud-based data engineering. Required Skills & Qualifications: Bachelors or Masters degree in Computer Science, Engineering, or a related field. 5+ years of experience in managing Databricks environments, with a strong focus on Azure Databricks. Expertise in Apache Spark, including performance tuning and optimization. Strong knowledge of Azure services, including Azure Data Factory, Azure Synapse, Azure Blob Storage, and Azure Active Directory. Experience with Infrastructure as Code (IaC) tools like Terraform, ARM templates, or Bicep. Strong scripting and automation skills using Python, PowerShell, or Bash. Understanding of data governance, security policies, and compliance in cloud-based environments. Experience integrating Databricks with enterprise authentication and authorization mechanisms. Familiarity with CI/CD pipelines and DevOps methodologies for Databricks deployment. Excellent problem-solving, communication, and stakeholder management skills. Preferred Qualifications: Databricks certifications such as Databricks Certified Associate/Professional. Experience working with Delta Lake and Unity Catalog. Knowledge of Azure networking, firewall configurations, and VNET integration. Hands-on experience with MLflow and AI/ML workloads within Databricks. Experience in multi-cloud environments and hybrid cloud architectures.