Azure Data Engineering Lead

You are an accomplished and dynamic Data Engineering Lead responsible for driving the design, development, and delivery of cutting-edge data solutions. Your role involves overseeing the design and development of robust data pipelines and systems, managing and mentoring a team of 5 to 7 engineers, and playing a critical role in architecting innovative solutions tailored to client needs. You will lead by example, fostering a culture of accountability, ownership, and continuous improvement while delivering impactful, scalable data solutions in a fast-paced consulting environment. Key Responsibilities: - Architecture & Delivery: - Lead the end-to-end architecture of data pipelines leveraging Azure Databricks, Delta Lake, and Apache Spark. - Design and implement the medallion architecture (Bronze, Silver, Gold) to structure scalable data lakes. - Oversee data ingestion from diverse sources, transformation pipelines, and delivery of analytics-ready datasets. - Define best practices for real-time and batch data processing patterns. - Technical Leadership: - Mentor and guide a team of data engineers, perform code reviews, and enforce engineering best practices. - Collaborate with solution architects, DevOps, data scientists, and business stakeholders. - Set and maintain coding standards, data governance policies, and deployment strategies. - Platform Engineering: - Architect CI/CD pipelines for data projects using Azure DevOps and Git. - Optimize Spark jobs, Delta Lake performance, and storage cost management. - Implement observability, data quality checks, and job orchestration across the stack. - Governance & Security: - Implement RBAC, Key Vault, managed identities, and other security controls. - Ensure compliance with data privacy regulations (GDPR, HIPAA, etc.). - Promote data cataloging and lineage through Azure Purview or equivalent tools. Qualifications Required: - 7 to 10 years of experience in data engineering with hands-on expertise in data pipeline development, architecture, and system optimization. - 4+ years of hands-on experience with Azure Databricks, Apache Spark, and Delta Lake.,