Lead Data Engineer

Minimum Requirements:

• 9+ years of experience overall in IT
• Current - 4+ years of experience on Azure Cloud as Data Engineer
• Current - 4+ years of hands-on experience on Databricks/Azure Databricks
• Proficient in Python/PySpark
• Proficient in SQL/TSQL
• Proficient in Data Warehousing Concepts (ETL/ELT, Data Vault Modelling, Dimensional Modelling, SCD, CDC)
• Logistics domain experience

Primary Skills:

Azure Cloud, Databricks, Azure Data Factory, Azure Synapse Analytics, SQL/TSQL, PySpark, Python

Technical Expertise:

• ETL/ELT Pipelines: Design, implementation, and optimization.
• Azure ADF and Synapse: Proficiency in creating pipelines, developing notebooks, and performing data transformations.
• Databricks: Hands-on experience with creating notebooks and performing data transformations.
• Familiarity with Dataflows, Data Wrangling, and integration with REST APIs.
• Experience with Parquet and Delta file processing, along with optimization techniques
• Familiarity with Azure DevOps and CI/CD

Concepts and Methodologies:

• Data Warehousing: Comprehensive understanding of concepts and implementation
• Data Modelling Techniques: Familiarity with Data Vault (if not, then Dimensional Modelling Technique)
• SCD and CDC: Strong grasp of Slowly Changing Dimensions (SCD) and Change Data Capture (CDC) methodologies
• Agile Methodology

Programming Skills:

Python/PySpark/SparkSQL Strong proficiency in any of these SQL/T-SQL

A must-have:

• Demonstrates strong association with team dynamics and the ability to maintain harmony in a collaborative environment.
• Proven ability to be a team player who contributes to shared goals while supporting peers.

Good to have:

• Basic knowledge on Delta Tables, Delta Live Tables (DLT), Partitioning and Optimization.
• Knowledge of Unity Catalog and Job Clusters is an added advantage.
• Familiarity with tools like Azure Monitor, Log Analytics, and Spark UI is an added advantage.
• Good to have knowledge with real-time data ingestion and streaming pipelines.
• Experience with framework driven projects (Metadata framework, Audit control framework)
• Relational and No-SQL Databases

Certifications:

• AZ 900 Azure fundamentals
• DP 203 – Azure Data Engineer
• Databricks Associate Data Engineer