Azure Data Engineer (Databricks)

Great Place to Work

Role Overview

An Azure Data Engineer specializing in Databricks is responsible for designing, building, and maintaining scalable data solutions on the Azure cloud platform, with a focus on leveraging Databricks and related big data technologies. The role involves close collaboration with data scientists, analysts, and software engineers to ensure efficient data processing, integration, and delivery for analytics and business intelligence needs 2 4 5 .

Key Responsibilities

• Design, develop, and maintain robust and scalable data pipelines using Azure Databricks, Azure Data Factory, and other Azure services.
• Build and optimize data architectures to support large-scale data processing and analytics.
• Collaborate with cross-functional teams to gather requirements and deliver data solutions tailored to business needs.
• Ensure data quality, integrity, and security across various data sources and pipelines.
• Implement data governance, compliance, and best practices for data security (e. g. , encryption, RBAC).
• Monitor, troubleshoot, and optimize data pipeline performance, ensuring reliability and scalability.
• Document technical specifications, data pipeline processes, and architectural decisions
• Support and troubleshoot data workflows, ensuring consistent data delivery and availability for analytics and reporting
• Automate data tasks and deploy production-ready code using CI/CD practices
• Stay updated with the latest Azure and Databricks features, recommending improvements and adopting new tools as appropriate

Required Skills and Qualifications

• Bachelor s degree in computer science, Engineering, or a related field
• 5+ years of experience in data engineering, with hands-on expertise in Azure and Databricks environments
• Proficiency in Databricks, Apache Spark, and Spark SQL
• Strong programming skills in Python and/or Scala
• Advanced SQL skills and experience with relational and NoSQL databases
• Experience with ETL processes, data warehousing concepts, and big data technologies (e. g. , Hadoop, Kafka)
• Familiarity with Azure services: Azure Data Lake Storage (ADLS), Azure Data Factory, Azure SQL Data Warehouse, Cosmos DB, Azure Stream Analytics, Azure Functions
• Understanding of data modeling, schema design, and data integration best practices
• Strong analytical, problem-solving, and troubleshooting abilities
• Experience with source code control systems (e. g. , GIT) and technical documentation tools
• Excellent communication and collaboration skills; ability to work both independently and as part of a team

Preferred Skills

• Experience with automation, unit testing, and CI/CD pipelines
• Certifications in Azure Data Engineering or Databricks are advantageous

Soft Skills

• Flexible, self-starter, and proactive in learning and adopting new technologies
• Ability to manage multiple priorities and work to tight deadlines
• Strong stakeholder management and teamwork capabilities