Azure Data Lead

Role Overview: As a Lead Data Engineer specializing in Databricks & Master Data Management, you will play a crucial role in designing, developing, and optimizing the next-generation data platform. Your responsibilities will include leading a team of data engineers, providing technical guidance, mentorship, and ensuring the scalability and high-performance of data solutions. Key Responsibilities: - **Technical Leadership:** - Lead the design, development, and implementation of scalable and reliable data pipelines using Databricks, Spark, and other relevant technologies. - Define and enforce data engineering best practices, coding standards, and architectural patterns. - Provide technical guidance and mentorship to junior and mid-level data engineers. - Conduct code reviews to ensure the quality, performance, and maintainability of data solutions. - **Databricks Expertise:** - Architect and implement data solutions on the Databricks platform, including Databricks Lakehouse, Delta Lake, and Unity Catalog. - Optimize Spark workloads for performance and cost efficiency on Databricks. - Develop and manage Databricks notebooks, jobs, and workflows. - Proficiently use Databricks features such as Delta Live Tables (DLT), Photon, and SQL Analytics. - **Master Data Management:** - Lead the technical design and implementation of the MDM solution for critical domains (e.g., Customer, Product, Vendor). - Define and implement data quality rules, entity resolution, matching, and survivorship logic to create and maintain Golden Records. - Partner with Data Governance and Data Stewardship teams to ensure organizational policies, standards, and data definitions for master data assets. - Ensure seamless provisioning of high-quality master data from the MDM/Lakehouse platform to downstream systems (ERP, CRM, BI). - **Pipeline Development & Operations:** - Develop, test, and deploy robust ETL/ELT pipelines for data ingestion, transformation, and loading from various sources. - Implement monitoring, alerting, and logging for data pipelines to ensure operational excellence. - Troubleshoot and resolve complex data-related issues. - **Collaboration & Communication:** - Work closely with cross-functional teams such as product managers, data scientists, and software engineers. - Communicate complex technical concepts clearly to both technical and non-technical stakeholders. - Stay updated with industry trends and emerging technologies in data engineering and Databricks. Qualifications Required: - Extensive hands-on experience with Databricks platform, including Databricks Workspace, Spark on Databricks, Delta Lake, and Unity Catalog. - Strong proficiency in optimizing Spark jobs and understanding Spark architecture. - Experience with Databricks features like Delta Live Tables (DLT), Photon, and Databricks SQL Analytics. - Deep understanding of data warehousing concepts, dimensional modeling, and data lake architectures. - Familiarity with data governance and cataloging tools (e.g., Purview, Profisee).,