Lead Azure Data Engineer

Job Summary:

Lead Data Engineer to architect and drive the development of scalable, secure, and high-performance data solutions

Key Responsibilities:

• Lead the design and architecture of end-to-end data engineering solutions across cloud platforms. 
• Develop and oversee robust data pipelines and

  ETL

  workflows using

  Python

  and

  Apache

  Spark. 
• Architect and implement scalable Delta Lake solutions for structured and semi-structured data. 
• Orchestrate complex workflows using Databricks Workflows or Azure Data Factory. 
• Translate business rules, retention metadata, and data governance policies into reusable, modular, and scalable pipeline components. 
• Ensure adherence to data privacy, security, and compliance standards (e.g., GDPR, HIPAA, etc.). 
• Mentor and guide junior data engineers, fostering best practices in coding, testing, and deployment. 
• Collaborate with cross-functional teams including data architects, analysts, and business stakeholders to align data solutions with business goals. 
• Drive performance optimization, cost-efficiency, and innovation in data engineering practices. 

Required Skills & Qualifications:

• 8+ years of experience in data engineering, with at least 2 years in a lead or architect role. 
• Expert-level proficiency in Python, Apache Spark, and Delta Lake. 
• Strong experience with Databricks Workflows and/or Azure Data Factory. 
• Deep understanding of data governance, metadata management, and business rule integration. 
• Proven track record in implementing data privacy, security, and regulatory compliance in insurance or financial domains. 
• Strong leadership, communication, and stakeholder management skills. 
• Experience with cloud platforms such as Azure, AWS, or GCP. 

Preferred Qualifications:

• Experience with CI/CD pipelines and DevOps practices in data engineering. 
• Familiarity with data cataloging and data quality tools. 
• Certifications in Azure Data Engineering or related technologies. 
• Exposure to enterprise data architecture and modern data stack tools.