Role Overview
We are seeking an experienced
Azure Data Engineer
to design, develop, and optimize scalable data pipelines and analytics solutions on Microsoft Azure. The ideal candidate will have strong hands-on expertise with Azure Databricks
, Azure Data Factory
, Python
, and PySpark
, along with a solid understanding of data modeling, ETL/ELT processes, and best practices in data engineering.
Key Responsibilities
- Design, build, and maintain
end-to-end data pipelines
using Azure Data Factory (ADF)
and Azure Databricks
. - Develop
ETL/ELT frameworks
and data transformation workflows using Python
and PySpark
. - Implement scalable
data ingestion
, data cleansing
, data validation
, and data processing
solutions. - Work with structured and unstructured datasets to build robust data engineering processes.
- Optimize Databricks notebooks, jobs, clusters, and workflows for
performance and cost efficiency
. - Implement
Delta Lake
, data versioning, and ACID transaction support for reliable pipelines. - Collaborate with data architects, BI teams, and analytics stakeholders to understand data requirements.
- Ensure
data quality
, data governance
, security
, and compliance across the data lifecycle. - Perform
root-cause analysis
, troubleshoot issues, and optimize pipeline reliability. - Create and maintain technical documentation, design specifications, and operational procedures.
Required Skills & Qualifications
- Strong hands-on experience with
Azure Databricks
and ADB architecture
, including notebooks, workflows, Unity Catalog, and Delta Lake. - Expertise in
Azure Data Factory
for orchestrating data pipelines. - Proficiency in
Python
and PySpark
for data transformation and large-scale processing. - Solid understanding of
ETL/ELT concepts
, data modeling, and distributed data processing. - Experience working with
Azure Storage (ADLS Gen2)
, Azure SQL
, and other Azure data components. - Familiarity with CI/CD for data pipelines using Git, Azure DevOps, or similar tools.
- Strong debugging, performance tuning, and problem-solving abilities.
- Excellent communication and collaboration skills.
Preferred Qualifications
- Experience with
Azure Synapse Analytics
, SQL Warehousing, or Data Lakes. - Exposure to
DataBricks Workflows
, MLFlow
, and Advanced Spark optimization
. - Knowledge of
Power BI
, Data Governance
, and Data Quality frameworks
. - Certifications such as
DP-203: Azure Data Engineer Associate
or equivalent.
Location:
PAN India Start Date:
Immediate Work Mode:
WFO