Azure Data Engineer

Role Overview

We are seeking an experienced PySpark Developer with strong expertise in Azure Databricks (ADB) and Azure Data Factory (ADF). The candidate will be responsible for building, optimizing, and managing large-scale data processing solutions to enable seamless data integration, transformation, and analytics.

Key Responsibilities

• Design, develop, and deploy PySpark applications and workflows on Azure Databricks for data transformation, cleansing, and aggregation.

• Build and manage data pipelines using Azure Data Factory (ADF) to support ETL/ELT processes across diverse data sources.

• Collaborate with Data Engineers and Data Scientists to process structured and unstructured datasets and derive actionable insights.

• Optimize PySpark jobs and pipelines for scalability, performance, and reliability.

• Ensure data quality, integrity, and compliance throughout all stages of data processing.

• Conduct financial risk assessments and implement mitigation strategies within data pipelines.

• Troubleshoot, debug, and resolve issues in data workflows.

• Implement data security best practices and maintain compliance with regulatory standards.

• Prepare and maintain technical documentation including data flows, architecture, and specifications.

Required Qualifications

• Bachelors degree in Computer Science, Engineering, or a related field (Masters preferred).

• Proven experience as a PySpark Developer or similar role with in-depth knowledge of Apache Spark.

• Hands-on expertise with Azure Databricks (ADB) and Azure Data Factory (ADF).

• Strong proficiency in Python and SQL.

• Experience in designing and optimizing ETL/ELT pipelines.

• Familiarity with Microsoft Azure or other cloud platforms.

• Strong problem-solving and critical thinking skills.

• Excellent communication and collaboration abilities.

• Experience in Financial, Risk, Compliance, or Banking domains is a plus.

Preferred Qualifications

• Certification in Azure Data Engineering or equivalent.

• Knowledge of big data tools such as Hadoop, Hive, Kafka.