Azure Data Engineer

Role

Experience

Location

Primary Objective

Key Responsibilities

• ETL Architecture & Design-

  Define and implement ETL strategies for data ingestion, transformation, and loading into Azure-based data lakes or warehouses. Design reusable ETL frameworks and patterns for structured and unstructured data.

• Cloud Platform Expertise -

  Utilize Azure services such as ADF for ETL workflows and workflow scheduling. Implement best practices for security, cost optimization, and performance tuning.

• Data Integration -

  Integrate data from multiple sources (on-prem, cloud, APIs) into centralized repositories. Ensure data quality, consistency, and lineage tracking.

• Automation & Orchestration -

  Build automated ETL pipelines using Azure ADF, Step Functions, or Airflow. Implement CI/CD for ETL deployments.

• Monitoring & Optimization -

  Set up monitoring for ETL jobs and troubleshoot performance issues. Optimize ETL processes for large-scale datasets and real-time streaming.

Required Skills

• Azure Data Sources

  : Azure Data Lake Storage (ADLS), Blob Storage, Azure SQL Database, Synapse Analytics. External Sources: APIs, on-prem databases, flat files (CSV, Parquet, JSON).

• Tools

  : Azure Data Factory (ADF) for orchestration, Databricks connectors.

• Apache Spark

  : Strong knowledge of Spark (PySpark, Spark SQL) for distributed processing.

• Data Cleaning & Normalization

  : Handling nulls, duplicates, schema evolution.

• Performance Optimization

  : Partitioning, caching, broadcast joins.

• Delta Lake

  : Implementing ACID transactions, time travel, and schema enforcement.

• Azure Data Factory (ADF):

  Building pipelines to orchestrate Databricks notebooks.

• Azure Key Vault

  : Secure credential management.

• Azure Monitor & Logging

  : For ETL job monitoring and alerting.

• Networking & Security

  : VNET integration, private endpoints.

Ideal Client Profile

• 2- 6 years in data engineering/ETL roles, with at least 2+ years in Azure cloud ETL leadership. Azure certifications (e.g., Azure Analytics Specialty, Solutions Architect) preferred.