Job
Description
About The Role
Project Role :DevOps Engineer
Project Role Description :Responsible for building and setting up new development tools and infrastructure utilizing knowledge in continuous integration, delivery, and deployment (CI/CD), Cloud technologies, Container Orchestration and Security. Build and test end-to-end CI/CD pipelines, ensuring that systems are safe against security threats.
Must have skills :Microsoft Azure DevOps
Good to have skills :NA
Minimum 5 year(s) of experience is required
Educational Qualification :15 years full time education
Summary:Youll be the engineer who makes data move reliably. Youll build Azure DevOps CI/CD that orchestrates ADF and Databricks Workflows to deliver data into a Medallion (Bronze/Silver/Gold) architecture, and youll own Terraform-based IaC that provisions secure, scalable Azure and Databricks resources across environments. Roles & Responsibilities:-Build CI/CD for data ingestionoAuthor and maintain Azure DevOps (ADO) YAML pipelines to version, validate, and deploy ADF pipelines and Databricks Workflows/Jobs.oImplement parameterized, reusable pipeline templates (multi-stage, approvals, gates) for dev/test/prod with artifact promotion.oOrchestrate batch and streaming patterns (e.g., ADF triggers + Databricks Jobs/Auto Loader) into Bronze ? Silver ? Gold layers.oIntegrate code quality, unit tests, and data checks (pytest, dbx/tests, Great Expectations or equivalent) in CI.Provision cloud & lakehouse with IaCoDesign Terraform modules and workspaces to provision Azure (RGs, VNets/PE, Key Vault, Storage, Event Hubs, Azure SQL, Monitor/Log Analytics) and Databricks (workspaces, UC metastore/catalogs/schemas, grants, clusters/cluster policies, Jobs, Repos/Asset Bundles).oManage Terraform state (remote backends), environment promotion, and drift detection; enforce Policy-as-Code (Azure Policy, Sentinel/OPA).oImplement secure networking (VNet injection, Private Link, firewall rules), identity (AAD groups, service principals, workload identities), and secrets management (Key Vault ? Databricks secret scopes).Operate and improve the platformoSet up observability for pipelines (Azure Monitor, Log Analytics, Databricks metrics & audit logs), SLOs, alerting, and incident runbooks.oOptimize spend (autoscaling, spot/On-Demand mix, job cluster vs warehouse usage, caching, cluster policies).oChampion Git strategies (trunk/PR), code reviews, and release tagging; document standards and how-to guides.Professional & Technical
Skills:
5+ years in DevOps/Platform Engineering supporting data platforms on Azure.Strong hands-on with:oAzure DevOps:Repos, multi-stage YAML pipelines, variable groups, service connections.oAzure Data Factory (ADF):authoring, parameterization, linked services, triggers, integration runtimes.oDatabricks:Workflows/Jobs, Repos/Asset Bundles, Delta Lake (Bronze/Silver/Gold), notebooks, job clusters & cluster policies; familiarity with Unity Catalog (catalogs/schemas/grants).oTerraform (AzAPI/AzureRM + Databricks providers):modules, workspaces, remote state, pipelines.oAzure networking & security:VNet/Private Endpoints, RBAC, Key Vault, MSI/service principals.Proficiency in Python and Bash/PowerShell for automation and CI tasks.Solid understanding of data ingestion patterns (batch/streaming, CDC/Auto Loader, file- and API-based sources).
Qualification 15 years full time education
