Lead Data Engineer

Job Description

Minimum Requirements: 9+ years of experience overall in IT Current - 4+ years of experience on Azure Cloud as Data Engineer Current - 4+ years of hands-on experience on Databricks/Azure Databricks Proficient in Python/PySpark Proficient in SQL/TSQL Proficient in Data Warehousing Concepts (ETL/ELT, Data Vault Modelling, Dimensional Modelling, SCD, CDC) Logistics domain experience Primary Skills: Azure Cloud, Databricks, Azure Data Factory, Azure Synapse Analytics, SQL/TSQL, PySpark, Python Technical Expertise: ETL/ELT Pipelines: Design, implementation, and optimization. Azure ADF and Synapse: Proficiency in creating pipelines, developing notebooks, and performing data transformations. Databricks: Hands-on experience with creating notebooks and performing data transformations. Familiarity with Dataflows, Data Wrangling, and integration with REST APIs. Experience with Parquet and Delta file processing, along with optimization techniques Familiarity with Azure DevOps and CI/CD Concepts and Methodologies: Data Warehousing: Comprehensive understanding of concepts and implementation Data Modelling Techniques: Familiarity with Data Vault (if not, then Dimensional Modelling Technique) SCD and CDC: Strong grasp of Slowly Changing Dimensions (SCD) and Change Data Capture (CDC) methodologies Agile Methodology Programming Skills: Python/PySpark/SparkSQL Strong proficiency in any of these SQL/T-SQL A must-have: Demonstrates strong association with team dynamics and the ability to maintain harmony in a collaborative environment. Proven ability to be a team player who contributes to shared goals while supporting peers. Good to have: Basic knowledge on Delta Tables, Delta Live Tables (DLT), Partitioning and Optimization. Knowledge of Unity Catalog and Job Clusters is an added advantage. Familiarity with tools like Azure Monitor, Log Analytics, and Spark UI is an added advantage. Good to have knowledge with real-time data ingestion and streaming pipelines. Experience with framework driven projects (Metadata framework, Audit control framework) Relational and No-SQL Databases Certifications: AZ 900 Azure fundamentals DP 203 – Azure Data Engineer Databricks Associate Data Engineer