Azure Data Engineer

Responsibilities:

• Design, build, and maintain scalable and reliable data pipelines using Azure Data Factory, Azure Synapse, and Databricks. • Develop and optimize large-scale data processing jobs using PySpark and Spark SQL on Azure Databricks or Synapse Spark pools. • Manage and work with large datasets stored in data lakes (ADLS Gen2) and integrate with enterprise data warehouses (e.g., SQL Server, Synapse). • Implement robust data transformation, cleansing, and aggregation logic to support analytics and reporting use cases. • Collaborate with BI developers, analysts, and data scientists to provision clean, reliable, and timely datasets. • Optimize data flows for performance and cost efficiency on Azure cloud platforms. • Implement data governance, lineage, and security practices across the data architecture. • Troubleshoot and resolve data pipeline failures and ensure high availability and fault-tolerant design. • Participate in code reviews, adhere to version control practices, and maintain high coding standards.

Mandatory skill sets:

• Azure Data Factory (ADF), Azure Synapse Analytics, Azure Databricks • Apache Spark, PySpark • SQL Server / T-SQL / Synapse SQL • Azure Data Lake Storage Gen2 • Big Data ecosystem knowledge (Parquet, Delta Lake, etc.) • Git, DevOps pipelines for data engineering • Performance tuning of Spark jobs and SQL queries

Preferred skill sets:

• Python for data engineering workflows • Azure Monitor, Log Analytics for pipeline observability • Power BI (data modeling, DAX) • Delta Lake • Experience with CI/CD for data pipelines (YAML pipelines, ADF integration) • Knowledge of data quality tools and frameworks