Lead Data Engineer

Job Title:

Location:

Experience:

Notice Period:

Budget:

Primary Skills

• Core:

  AWS, Python

Job Overview

Senior/Lead Data Engineer

Key Responsibilities

• Data Ingestion Framework:

• Architect and maintain ETL pipelines using

  AWS Glue, Lambda, EMR, ECS, EC2, Step Functions

  .
• Optimize data ingestion for scalability, resiliency, and automation.

• Data Quality & Validation:

• Build automated validation and error-handling mechanisms.
• Implement monitoring, logging, and alerting for data quality assurance.

• API Development:

• Design and develop

  secure RESTful APIs

  for internal/external data integration.
• Maintain API documentation and ensure adherence to best practices.

• Collaboration & Agile:

• Partner with business stakeholders, data scientists, and operations.
• Contribute to

  CI/CD pipelines

  (preferably GitLab) and Agile processes.

Required Qualifications

• Experience:

  Minimum 5+ years in data engineering / analytical platform development.

• Programming:

  Strong in

  Python / PySpark / SQL

  for ETL and large-scale data handling.

• AWS Expertise:

  Hands-on with

  AWS Glue, Lambda, Step Functions, S3

  , and

  data platforms

  (EMR, Apache Spark, Iceberg).

• Database Exposure:

  DynamoDB, Aurora, Postgres, Redshift.

• API Development:

  Proven REST API design & integration skills.

• DevOps:

  Experience with

  GitLab CI/CD

  , Agile methodology.

Preferred Qualifications

• Bachelor’s/Master’s in Computer Science, Data Engineering, or related field.
• Familiarity with

  AWS Kinesis, Firehose, SQS

  , and

  data lakehouse architectures

  .
• Experience with

  data quality frameworks

  and

  API-driven integrations

  in multi-cluster setups.
• Commitment to

  Information Security policies and best practices

  .