Lead Data Engineer (Azure)

We are seeking an experienced and forward-thinking Lead Data Engineer to spearhead the development of scalable, secure, and high-performance data solutions. You must possess in-depth technical knowledge in Python, Apache Spark, Delta Lake, and orchestration tools like Databricks Workflows or Azure Data Factory. Your expertise should also include a solid understanding of data governance, metadata management, and regulatory compliance within the insurance and financial services sectors. Proficiency in developing Python applications, Spark-based workflows, utilizing Delta Lake, and orchestrating jobs with Databricks Workflows or Azure Data Factory is essential. Furthermore, you should be able to incorporate retention metadata, business rules, and data governance policies into reusable pipelines and have a strong grasp of data privacy, security, and regulatory requirements in insurance and finance. As the Lead Data Engineer, your responsibilities will include designing and architecting end-to-end data engineering solutions across cloud platforms, creating and managing robust data pipelines and ETL workflows using Python and Apache Spark, and implementing scalable Delta Lake solutions for structured and semi-structured data. You will also be tasked with orchestrating complex workflows using Databricks Workflows or Azure Data Factory, translating business rules and data governance policies into modular pipeline components, ensuring compliance with data privacy and security standards, and mentoring junior data engineers to promote coding best practices, testing, and deployment efficiency. Collaboration with cross-functional teams such as data architects, analysts, and business stakeholders to align data solutions with business objectives is crucial, as is driving performance optimization, cost-efficiency, and innovation in data engineering practices. Key Qualifications: - Minimum of 8 years of experience in data engineering, with at least 2 years in a lead or architect position. - Expertise in Python, Apache Spark, and Delta Lake at an advanced level. - Strong familiarity with Databricks Workflows and/or Azure Data Factory. - Deep comprehension of data governance, metadata management, and integration of business rules. - Proven track record of implementing data privacy, security, and regulatory compliance in insurance or financial domains. - Strong leadership, communication, and stakeholder management skills. - Experience working with cloud platforms such as Azure, AWS, or GCP. Preferred Qualifications: - Background in CI/CD pipelines and DevOps practices within data engineering. - Knowledge of data cataloging and data quality tools. - Certifications in Azure Data Engineering or related technologies. - Exposure to enterprise data architecture and modern data stack tools.,