Senior DevOps Engineer

Senior DevOps Engineer Summary The Senior DevOps Engineer will design and maintain automation frameworks that underpin Databricks-based data warehouse environments and associated Azure services. This role focuses on establishing robust CI/CD pipelines, automating infrastructure provisioning, and ensuring seamless deployment of code and configurations across development, test, and production environments. Emphasis is placed on reliability, repeatability, and security in deployments, enabling rapid delivery of updates while maintaining compliance and high quality.

Responsibilities

• Develop and manage CI/CD pipelines for Databricks notebooks, libraries, and Azure Data Factory (ADF) workflows.
• Implement Infrastructure-as-Code (IaC) using Terraform or Bicep to provision and manage Azure resources.
• Automate Databricks job and notebook deployments using CLI, REST APIs, and integration with Git-based workflows.
• Establish branching and merging strategies (e.g., GitFlow, trunk-based development) and enforce code review processes.
• Integrate automated testing, data validation, and quality gates into CI/CD pipelines.
• Manage release processes for Databricks jobs, ADF pipelines, and tabular models in Azure Analysis Services.
• Monitor pipeline runs, investigate failures, and resolve issues in collaboration with engineering teams.
• Secure CI/CD processes by leveraging Key Vault for secrets management, service principals, and role-based access controls.
• Document pipeline processes and provide training to development and analytics teams.

Required Qualifications

• 5+ years of DevOps or build/release engineering experience in cloud environments.
• Proficiency in Azure DevOps or GitHub Actions, with hands-on experience automating CI/CD pipelines.
• Strong knowledge of Databricks (clusters, jobs, CLI, APIs) and Azure Data Factory.
• Experience with Terraform, ARM templates, or Bicep for cloud automation.
• Strong scripting skills (Python, Bash, PowerShell).
• Proficiency with Git and modern source control workflows.
• Experience implementing automated testing and validation for data pipelines.

Preferred Qualifications

• Prior experience supporting data and analytics platforms (Databricks, Synapse, Power BI, AAS).
• Familiarity with monitoring and alerting solutions such as Azure Monitor and Log Analytics.
• Knowledge of containerization (Docker, Kubernetes) and related deployment strategies.
• Exposure to data warehouse migration projects or legacy-to-cloud modernization.

Certifications

• Databricks Certified Data Engineer (Associate or Professional).
• Microsoft Certified: Azure DevOps Engineer Expert.
• Microsoft Certified: Azure Administrator Associate (AZ-104).
• HashiCorp Certified: Terraform Associate (optional).