Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.
Job Summary
For this role, we are seeking a Senior Data Engineer for our Client's ETL Support Team to act as a technical pillar during our Client's transition to an AWS-native data environment. You will be a key escalation point, responsible for handling the most complex data pipeline issues. This role may move beyond simple support; you might be requested to harden and improve our Client's systems by driving performance tuning, cost optimization, and automation initiatives.
Work Location:
Noida, Sec 62
Working days per week:
5
Rotational Shift :
Candidates should be flexible working on a Roster basis including Weekends but maximum 5 days a week
Responsibilities:
- Automate data workflows such as data ingestion, aggregation, and ETL processing.
- Logging and instrumentation of pipelines and services.
- Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures.
- Advanced Support: Lead the monitoring (~2,300 jobs) and support of all scheduled and on-demand ETL jobs, acting as the primary escalation point for the team.
- Expert Troubleshooting: Own the most complex troubleshooting and Root Cause Analysis (RCA) efforts, ensuring detailed documentation and implementing long-term, structural fixes.
- Optimization: Proactively perform performance tuning to improve system efficiency and reliability. Actively identify and implement cost optimization strategies for AWS Glue usage.
- Automation & Development: Design and develop new automation solutions (using Glue, Lambda, and Step Functions) to reduce the need for manual support. Lead development for business enhancement requirements.
- Data Management: Oversee and execute complex data integrity tasks, including reloading CDC tables and managing data synchronization between external databases (like Teradata) and AWS Glue.
- Integration: Implement AWS Lambda functions to trigger external events, such as SAS & RMJ, as needed.
- Decision & Execution: Make critical, autonomous decisions under pressure to resolve incidents and execute complex tasks, ensuring all SLAs are met. Escalate to leadership only when necessary.
- Mentorship: Mentor junior engineers, lead knowledge transfer sessions with internal resources, and oversee the creation of high-quality technical documentation and runbooks.
Qualifications:
- Technical Expertise: In-depth, expert-level knowledge and extensive hands-on experience with AWS S3, Glue, Lambda, Step Functions and other related services.
- Demonstrated expertise with a minimum of 5-8 years of relevant experience.
- Deep understanding of Data engineering, ETL principles and data warehouse design.
- Optimization Experience: Demonstrable experience in system performance tuning and cloud cost optimization, specifically with AWS Glue.
- Automation: Proven ability to design and develop automation solutions from the ground up.
- Problem-Solving: Expert-level troubleshooting and RCA skills.
- Leadership: Ability to work independently, make critical decisions, and provide technical guidance.
- Availability: Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis.
- Database Skills: Strong experience with Change Data Capture (CDC), managing database synchronization, and working with Teradata.
