Lead Data Engineer

Data Engineering Lead

About the Role

We are searching for a strategic, results-driven Data Engineering Lead to join our Data Science software engineering team. The ideal candidate will possess a robust technical background in data engineering, extensive hands-on experience with modern data architectures, and a proven ability to mentor and lead high-performing engineering teams. As a thought leader, you will design and implement scalable data solutions that empower our data scientists and drive impactful insights across the organization.

Key Responsibilities

• Lead, mentor, and grow a team of data engineers, fostering technical excellence and professional development.
• Architect, develop, and maintain advanced ELT data pipelines using state-of-the-art tools and best practices.
• Collaborate closely with data scientists to understand analytical requirements and translate them into scalable, reliable data solutions.
• Oversee the management and optimization of distributed, column-oriented databases such as Vertica and Snowflake to ensure high performance and data integrity.
• Drive the implementation and continuous improvement of data workflow orchestration with tools like Apache Airflow.
• Conduct root cause analysis and lead troubleshooting efforts to resolve complex data issues, ensuring availability and quality of critical datasets.
• Champion best practices for data governance, security, and compliance throughout all engineering processes.
• Document data processes, workflows, and architectural decisions for transparency and knowledge sharing.
• Promote innovation by evaluating emerging technologies and integrating them into existing data platforms as appropriate.

Required Qualifications

• Bachelors degree in computer science, Engineering, or a related field.
• 8+ years of progressive experience in data engineering, with at least 2 years in a technical leadership role.
• Expertise in Python and SQL for large-scale data manipulation and transformation.
• Extensive experience with cloud-based data platforms and services, particularly AWS.
• Deep understanding of data warehousing, dimensional modelling, and analytics frameworks.
• Hands-on experience with orchestration and workflow automation tools such as Apache Airflow and Jenkins.
• Proficiency in distributed, column-oriented databases (e.g., Vertica, Snowflake) and familiarity with CI/CD practices using Git or similar tools.
• Demonstrated ability to solve complex analytical problems and deliver high-quality, reliable data solutions at scale.
• Outstanding communication, collaboration, and organizational skills, with a track record of cross-functional partnership.

Preferred Qualifications

• Master’s degree in computer science, Engineering, or a related discipline.
• Experience implementing data governance, security, and compliance at scale.
• Knowledge of Apache Spark for distributed data processing and Apache Kafka for real-time data streaming.
• Background in supporting data science workflows and familiarity with machine learning data needs.

Join us to lead our data engineering efforts and enable the next generation of data science innovation.