Azure Data Lead

Role Overview: As a Lead Data Engineer specializing in Databricks, you will be responsible for designing, developing, and optimizing the next-generation data platform. Your role will involve leading a team of data engineers, providing technical guidance, mentorship, and ensuring scalability and high performance of data solutions. Key Responsibilities: - Lead the design, development, and implementation of scalable and reliable data pipelines using Databricks, Spark, and other relevant technologies. - Define and enforce data engineering best practices, coding standards, and architectural patterns. - Provide technical guidance and mentorship to junior and mid-level data engineers, conduct code reviews, and ensure quality, performance, and maintainability of data solutions. - Architect and implement data solutions on the Databricks platform, including Databricks Lakehouse, Delta Lake, and Unity Catalog. - Optimize Spark workloads for performance and cost efficiency on Databricks. - Develop and manage Databricks notebooks, jobs, and workflows, and proficiently use Databricks features such as Delta Live Tables (DLT), Photon, and SQL Analytics. - Develop, test, and deploy robust ETL/ELT pipelines for data ingestion, transformation, and loading from various sources. - Implement monitoring, alerting, and logging for data pipelines to ensure operational excellence. - Troubleshoot and resolve complex data-related issues. - Collaborate closely with cross-functional teams, including product managers, data scientists, and software engineers. - Communicate complex technical concepts to both technical and non-technical stakeholders. - Stay updated with industry trends and emerging technologies in data engineering and Databricks. Qualifications Required: - Extensive hands-on experience with the Databricks platform, including Databricks Workspace, Spark on Databricks, Delta Lake, and Unity Catalog. - Strong proficiency in optimizing Spark jobs and understanding Spark architecture. - Experience with Databricks features like Delta Live Tables (DLT), Photon, and Databricks SQL Analytics. - Deep understanding of data warehousing concepts, dimensional modeling, and data lake architectures.,