Azure Databricks Engineer

Position:

Role Overview

Azure Databricks Engineer

Key Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines on

  Azure Databricks

  .
• Work with structured, semi-structured, and unstructured data from diverse sources.
  
• Implement

  Delta Lake

  ,

  Unity Catalog

  , and

  Lakehouse architecture

  for data governance and performance optimization.
• Collaborate with data architects, analysts, and business teams to translate business needs into technical solutions.
  
• Integrate Databricks with other

  Azure services

  such as Azure Data Lake Storage (ADLS), Azure Synapse Analytics, Azure Data Factory (ADF), Azure Event Hub, and Azure Functions.
• Implement

  data quality frameworks, monitoring, and logging

  for pipelines.
• Tune and optimize

  Spark jobs, cluster configurations, and costs

  .
• Ensure

  security, compliance, and best practices

  in cloud-based data processing.
• Participate in

  CI/CD implementation

  for data pipelines using Azure DevOps/GitHub Actions.
• Support and troubleshoot production workloads with high availability and reliability.
  

Required Skills & Qualifications

• Strong experience in

  Azure Databricks

  and

  Apache Spark (PySpark/Scala/SQL)

  .
• Hands-on expertise with

  Azure Data Lake Storage, ADF, Synapse, Event Hub, Service Bus

  .
• Experience with

  Delta Lake, Unity Catalog, and Lakehouse implementations

  .
• Solid understanding of

  data modeling, partitioning, and performance optimization

  .
• Proficiency in

  SQL

  and at least one programming language (Python/Scala/Java).
• Knowledge of

  CI/CD pipelines, Git, and DevOps practices

  .
• Familiarity with

  data governance, security, and compliance frameworks

  .
• Strong problem-solving and analytical skills with ability to handle large-scale datasets.
  

Preferred Qualifications

• Azure certifications (e.g.,

  DP-203: Data Engineer Associate

  ,

  AZ-305

  , or equivalent).
• Experience with

  streaming data pipelines

  using Spark Structured Streaming, Kafka, or Event Hub.
• Knowledge of

  machine learning pipelines

  using MLflow or Azure Machine Learning.