Databricks Data Engineering Lead

The Databricks Data Engineering Lead role requires a highly skilled individual who will architect and lead the implementation of scalable, high-performance data pipelines and platforms using the Databricks Lakehouse ecosystem. As a Data Engineering Lead, you will be responsible for managing a team of data engineers, establishing best practices, and collaborating with cross-functional stakeholders to unlock advanced analytics, AI/ML, and real-time decision-making capabilities. Your key responsibilities will include leading the design and development of modern data pipelines, data lakes, and lakehouse architectures using Databricks and Apache Spark. You will manage and mentor a team of data engineers, providing technical leadership and fostering a culture of excellence. Additionally, you will architect scalable ETL/ELT workflows to process structured and unstructured data from various sources (cloud, on-prem, streaming), build and maintain Delta Lake tables, and optimize performance for analytics, machine learning, and BI use cases. Collaboration with data scientists, analysts, and business teams to deliver high-quality, trusted, and timely data products is crucial. Ensuring best practices in data quality, governance, lineage, and security, including the use of Unity Catalog and access controls, will also be part of your responsibilities. Integration of Databricks with cloud platforms (AWS, Azure, or GCP) and data tools (Snowflake, Kafka, Tableau, Power BI, etc.) and implementation of CI/CD pipelines for data workflows using tools such as GitHub, Azure DevOps, or Jenkins are essential tasks. It is important to stay current with Databricks innovations and provide recommendations on platform strategy and architecture improvements. Qualifications for this role include a Bachelors or Masters degree in Computer Science, Data Engineering, or a related field. You should have at least 7+ years of experience in data engineering, including 3+ years working with Databricks and Apache Spark. Proven leadership experience in managing and mentoring data engineering teams is required. Proficiency in PySpark, SQL, and experience with Delta Lake, Databricks Workflows, and MLflow are necessary skills. A strong understanding of data modeling, distributed computing, and performance tuning is essential. Familiarity with one or more major cloud platforms (Azure, AWS, GCP) and cloud-native services, experience implementing data governance and security in large-scale environments, and familiarity with real-time data processing using Structured Streaming or Kafka are also expected. Knowledge of data privacy, security frameworks, compliance standards (e.g., PCIDSS, GDPR), exposure to machine learning pipelines, notebooks, and ML Ops practices are additional qualifications required. A Databricks Certified Data Engineer or equivalent certification is preferred.,