AZURE Data Bricks

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks and Apache Spark.
  
• Build data ingestion, transformation, and processing frameworks for structured and unstructured data.
  
• Work with Azure services including ADLS Gen2, Azure Data Factory (ADF), Azure Synapse, Azure SQL, Delta Lake, and Event Hubs.
  
• Optimize and tune Spark jobs for performance, cost efficiency, and scalability.
  
• Implement advanced analytics and machine learning workflows using Databricks notebooks.
  
• Ensure data quality, data governance, and compliance throughout the data lifecycle.
  
• Collaborate with data architects, analysts, and business stakeholders to understand requirements and deliver data solutions.
  
• Manage CI/CD pipelines for data engineering workloads using Azure DevOps or GitHub Actions.
  
• Troubleshoot performance, security, and operational issues in Databricks environments.
  
• Document architecture, data flows, and technical designs.
  

Required Skills

• experience in Data Engineering or Big Data environments.
  
• Strong hands-on experience with

  Azure Databricks

  ,

  Apache Spark

  , and

  Delta Lake

  .
• Proficiency in Python, PySpark, SQL, and data transformation logic.
  
• Experience with Azure Storage solutions (ADLS Gen1/Gen2).
  
• Strong understanding of Azure Data Factory pipelines, triggers, and orchestration.
  
• Experience with DevOps practices, version control (Git), and CI/CD pipelines.
  
• Good understanding of data modeling, data warehousing, and data lake architectures.
  
• Strong problem-solving, debugging, and performance optimization skills.
  

Preferred Skills

• Experience with Azure Synapse Analytics, Azure Stream Analytics, or Power BI.
  
• Knowledge of machine learning frameworks (MLflow, Databricks AutoML).
  
• Understanding of cloud security, IAM, networking, and data governance (Purview).
  
• Databricks or Azure Data Engineering certifications.