Lead Python Data Engineer

Lead Python Data Engineer - Databricks (Unity Catalog, Delta, CI/CD) 

Job Summary

We are seeking a highly skilled Data Engineer with a strong background in Databricks and a passion for data engineering. The ideal candidate will have extensive experience in designing and implementing data solutions using various technologies, including PySpark, Spark SQL, and Power BI. This role requires a proactive individual who can work independently and collaboratively in a fast-paced environment. 

Required Technical Skills:

• 7+ years of hands-on experience in software development, with a strong emphasis on Python programming. 
• Deep expertise in PySpark (Spark SQL, DataFrames, and Datasets) for building and optimizing scalable data pipelines. (PySpark) 
• Solid experience with Azure Databricks (Unity Catalog), including working with Delta tables, workflows, and scripting in Python. 
• Understanding of data governance, data management, security, and compliance in cloud environments. 
• Familiar with Azure cloud services, especially Azure Data Lake, Azure Data Factory, 
• Experience with Power BI, including DAX and Power Query. 
• Modeling data for Data Warehousing and Visualization projects through Semantic Layers. 
• Skilled in writing complex SQL queries and building views involving joins across multiple tables. 
• Proficient in tools like DevOps (project management), and version control and CI/CD pipelines. 

Nice to have Skills:

• Experience Azure AI, Azure Copilot Studio and Azure Fabric. 
• Experience building workflows with Power Automate. 
• EventHub or Kafka for streaming data ingestion. 
• Proficient in designing and implementing real-time data streaming solutions. 
• Experience across the Software Development Lifecycle (SDLC), with a strong grasp of Agile Scrum methodologies. 

Soft Skills & Collaboration:

• Strong collaboration mindset with enthusiasm for mobbing sessions and cross-team coordination. 
• Excellent problem-solving and code review skills. 
• Strong communication skills with the ability to understand business requirements and translate them into technical solutions. 
• Proactive and responsive during business hours for any ad-hoc calls or meetings. 
• Demonstrated ability to prioritize effectively, think creatively, and adapt quickly to new technologies and feedback. 
•