Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.
Job Summary
For this role, we are seeking a Senior Data Engineer for our Client's ETL Support Team to act as a technical pillar during our Client's transition to an AWS-native data environment. You will be a key escalation point, responsible for handling the most complex data pipeline issues. This role may move beyond simple support; you might be requested to harden and improve our Client's systems by driving performance tuning, cost optimization, and automation initiatives.
Work Location: Noida, Sec 62
Working days per week: 5
Rotational Shift : Candidates should be flexible working on a Roster basis including Weekends but maximum 5 days a week
-
Automate data workflows such as data ingestion, aggregation, and ETL processing.
-
Logging and instrumentation of pipelines and services.
-
Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures.
-
Advanced Support: Lead the monitoring (~2,300 jobs) and support of all scheduled and on-demand ETL jobs, acting as the primary escalation point for the team.
-
Expert Troubleshooting: Own the most complex troubleshooting and Root Cause Analysis (RCA) efforts, ensuring detailed documentation and implementing long-term, structural fixes.
-
Optimization: Proactively perform performance tuning to improve system efficiency and reliability. Actively identify and implement cost optimization strategies for AWS Glue usage.
-
Automation & Development: Design and develop new automation solutions (using Glue, Lambda, and Step Functions) to reduce the need for manual support. Lead development for business enhancement requirements.
-
Data Management: Oversee and execute complex data integrity tasks, including reloading CDC tables and managing data synchronization between external databases (like Teradata) and AWS Glue.
-
Integration: Implement AWS Lambda functions to trigger external events, such as SAS & RMJ, as needed.
-
Decision & Execution: Make critical, autonomous decisions under pressure to resolve incidents and execute complex tasks, ensuring all SLAs are met. Escalate to leadership only when necessary.
-
Mentorship: Mentor junior engineers, lead knowledge transfer sessions with internal resources, and oversee the creation of high-quality technical documentation and runbooks.
-
Technical Expertise: In-depth, expert-level knowledge and extensive hands-on experience with AWS S3, Glue, Lambda, Step Functions and other related services.
-
Demonstrated expertise with a minimum of 5-8 years of relevant experience.
-
Deep understanding of Data engineering, ETL principles and data warehouse design.
-
Optimization Experience: Demonstrable experience in system performance tuning and cloud cost optimization, specifically with AWS Glue.
-
Automation: Proven ability to design and develop automation solutions from the ground up.
-
Problem-Solving: Expert-level troubleshooting and RCA skills.
-
Leadership: Ability to work independently, make critical decisions, and provide technical guidance.
-
Availability: Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis.
-
Database Skills: Strong experience with Change Data Capture (CDC), managing database synchronization, and working with Teradata.
