Lead Data Engineer

Lead Data Engineer

Main Responsibilities

Support & Enhancement Leadership

• Act as the primary technical lead for support and enhancement of assigned products.
• Ensure incident resolution, problem management, and enhancement delivery within agreed upon

  SLAs

  .
• Perform

  root cause analysis (RCA)

  and provide technical solutions to recurring issues.
• Design end-to-end data engineering solutions that are scalable and modular.
• Work in

  Agile

  implementations and lead all data engineering initiatives.

Technical Ownership

• Provide technical direction and architectural guidance for improvements, optimizations, and issue resolutions.
• Drive best practices in code performance tuning,

  ETL

  processing, and cloud-native data management.
• Lead the modernization of legacy data pipelines and applications by leveraging

  AWS services

  (

  Glue, Lambda, Redshift, S3, EMR, Athena

  , etc.).
• Utilize

  PySpark, SQL, and Python

  to manage big data processing pipelines efficiently.

Client Engagement

• Maintain high visibility with client stakeholders and act as a trusted technical advisor.
• Proactively identify and suggest areas for improvement or innovation to enhance business outcomes.
• Participate in daily stand-ups, retrospectives, and client presentations, communicating complex technical concepts clearly.

Team & Delivery Management

• Lead a cross-functional team of engineers, providing mentorship and driving upskilling.
• Monitor team performance, support capacity planning, and ensure timely, high-quality deliveries.
• Enforce adherence to governance, documentation, and change management practices.

Process & Quality Assurance

• Implement and ensure compliance with engineering best practices, including

  CI/CD

  , version control, and automated testing.
• Define support procedures and documentation standards.
• Identify risk areas and dependencies, and propose mitigation strategies.

Required Skills & Qualifications

• Extensive experience in

  Data Engineering

  or Application Support and Development.
• Deep hands-on expertise in the

  AWS ecosystem

  (

  Glue, Lambda, Redshift, S3, Athena, EMR, CloudWatch

  ).
• Proficiency in

  PySpark, SQL, and Python

  , with experience in handling big data pipelines.
• Strong application debugging skills across batch and near real-time systems.
• Good knowledge of incident lifecycle management, RCA, and performance optimization.
• Proven leadership experience leading engineering teams, preferably in support/enhancement environments.
• Excellent communication and client-facing skills.
• Experience with tools like

  JIRA, Confluence, Git, and Jenkins

  .

Good to Have

• Experience in

  on-prem to AWS migration

  projects.
• Familiarity with legacy technologies and their interaction with modern cloud stacks.
• Good knowledge of designing

  Hive tables

  with partitioning for performance.