Lead Data Engineer

As a Lead Data Engineer specializing in Databricks, your role involves designing, developing, and optimizing scalable, efficient, and secure data solutions. Your primary responsibilities include: - Leading the design, development, and optimization of data solutions using Databricks. - Collaborating with cross-functional teams to gather and analyze data requirements, translating them into robust data architectures and solutions. - Developing and maintaining ETL pipelines, leveraging Databricks and integrating with Azure Data Factory as needed. - Implementing machine learning models and advanced analytics solutions, incorporating Generative AI to drive innovation. - Ensuring data quality, governance, and security practices are adhered to, maintaining the integrity and reliability of data solutions. - Providing technical leadership and mentorship to junior engineers, fostering an environment of learning and growth. - Staying updated on the latest trends and advancements in data engineering, Databricks, Generative AI, and Azure Data Factory to continually enhance team capabilities. In order to excel in this role, you are required to have: - A Bachelor's or Master's degree in Computer Science, Information Technology, or a related field. - 7+ to 10 years of experience in data engineering, with a focus on Databricks. - Proven expertise in building and optimizing data solutions using Databricks and integrating with Azure Data Factory/AWS Glue. - Proficiency in SQL and programming languages such as Python or Scala. - Strong understanding of data modeling, ETL processes, and Data Warehousing/Data Lakehouse concepts. - Familiarity with cloud platforms, particularly Azure, and containerization technologies such as Docker. - Excellent analytical, problem-solving, and communication skills. - Demonstrated leadership ability with experience mentoring and guiding junior team members. Additionally, the following skills would be advantageous for this role: - Experience with Generative AI technologies and their applications. - Familiarity with other cloud platforms, such as AWS or GCP. - Knowledge of data governance frameworks and tools. This role offers you the opportunity to lead and innovate in the realm of data engineering, leveraging cutting-edge technologies and fostering a culture of continuous learning and growth within the team.,