Lead Data Engineer (AWS)

Role & responsibilities

• Data Ingestion Framework:
• Design & Development: Architect, develop, and maintain an end-to-end data ingestion framework that efficiently extracts, transforms, and loads data from diverse sources.
• Framework Optimization: Use AWS services such as AWS Glue, Lambda, EMR, ECS , EC2 and Step Functions to build highly scalable, resilient, and automated data pipelines.
• Data Quality & Validation: Validation Processes: Develop and implement automated data quality checks, validation routines, and error-handling mechanisms to ensure the accuracy and integrity of incoming data.
• Monitoring & Reporting: Establish comprehensive monitoring, logging, and alerting systems to proactively identify and resolve data quality issues.
• API Development: Design & Implementation: Architect and develop secure, high-performance APIs to enable seamless integration of data services with external applications and internal systems.
• Documentation & Best Practices: Create thorough API documentation and establish standards for API security, versioning, and performance optimization.
• Collaboration & Agile Practices:
• Cross-Functional Communication: Work closely with business stakeholders, data scientists, and operations teams to understand requirements and translate them into technical solutions.
• Agile Development: Participate in sprint planning, code reviews, and agile ceremonies, while contributing to continuous improvement initiatives and CI/CD pipeline development (using tools like GitLab).

Preferred candidate profile

• Bachelors or Masters degree in Computer Science, Data Engineering, or a related field.
• Experience with additional AWS services such as Kinesis, Firehose, and SQS.
• Familiarity with data lakehouse architectures and modern data quality frameworks.
• Prior experience in a role that required proactive data quality management and API-driven integrations in complex, multi-cluster environments.