Function as part of an Agile engineering team in the role of Lead Data Engineer, responsible for orchestrating and upholding the highest levels of Engineering and Enterprise Data Warehouse Concepts and practices throughout the Software Development Life Cycle, collaborating closely with development, operations, and architecture teams.
Define delivery phases and perform project, task and individual management duties for team, setting activities, sub-activities, and milestones, while ensuring that development and delivery standards are rigorously enforced.
Manage and enhance our Continuous Integration, Delivery, and Deployment processes, teams, and users to ensure high-quality and efficient delivery.
Collaborate closely with Engineering Managers and Business Analysts to produce accurate delivery estimates and oversee the transition from analysis through design to deliver
Design and implement scalable data solutions using AWS services
Design and implement data models for warehouse solutions
Create and maintain Infrastructure as Code using CloudFormation
Implement comprehensive unit testing strategies
Work closely with cross-functional teams to gather requirements
Mentor junior engineers on AWS best practices and data engineering concepts
Participate in code reviews and technical discussions
Document technical solutions and maintain knowledge base
Deploy and troubleshoot solutions as needed, adhering to best practices throughout the development lifecycle to identify and resolve issues prior to deployment.
Plan, design, and develop cloud-based applications, analytical services, and platforms.
Ensure adherence to company security guidelines and requirements.Qualifications

What we are looking for

An enthusiastic team member who fosters collaborative relationships, advocate for technical excellence, and adapt to evolving requirements within a dynamic team environment.
Demonstrate excellent communication, organizational skills, and a proactive approach to meeting deadlines and achieving project goals.
Proven experience as a Lead Data Engineer with expertise in Python and Spark.
Experience in CI/CD pipelines using GitHub and GitHub Actions to streamline development workflows.
Develop and maintain ETL pipelines using AWS Glue, Lambda, and EventBridge.
Manage data storage solutions across S3 and Redshift.
Implement security best practices using AWS IAM.
Experience with cloud computing services, especially AWS ECS, RDS, EFS, S3, Lambda, to build scalable and resilient applications.
Experience with database management including SQL (e.g., MySQL) and NoSQL (e.g., MongoDB), ensuring efficient data storage and retrieval.
Ensure code quality through SonarCloud integration
Create and maintain Infrastructure as Code using CloudFormation
In-depth knowledge in Test Driven Development (TDD) for Data Pipelines and Python
Excellent container orchestration skills using Docker, ensuring efficient application deployment and management.
Strong understanding of Agile methodologies and implementations (Kanban, XP, Scrum).
Excellent problem-solving and troubleshooting skills.
Process-oriented with great documentation skills.
Bachelor s degree in computer science or a related field, or equivalent work experience preferred

More Jobs at Taylor & Francis

Regional Director - eBooks Sales (South & West)

hyderabad, chennai, bengaluru

8.0 - 13.0 yrs

INR 40 - 50 Lacs

Lead Data Engineer

bengaluru

3.0 - 7.0 yrs

INR 11 - 15 Lacs

Senior Talent Acquisition Partner

bengaluru

3.0 - 8.0 yrs

INR 12 - 16 Lacs

Business Development Manager

chennai, bengaluru

8.0 - 13.0 yrs

INR 15 - 20 Lacs

Sales Manager

bengaluru

10.0 - 15.0 yrs

INR 14 - 16 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.