Senior Data Engineer

Lead Data Engineer

Skills Mandatory- S3, Glue, Lambda, Step Function, Snowflake

Skills - Primary - PySpark, Snowflake.

Skills - Good to have - Cassandra

Total Experience - 7+ years ; Relevant Experience - 5+ years

Work Location : Cochin/TVM

Job Purpose

Responsible for consulting for the client to understand their data needs and delivering data solution to the client.

Job Description / Duties & Responsibilities

Data Ingestion Framework:

oDesign & Development: Design, develop, and maintain an end-to-end data ingestion framework that efficiently extracts, transforms, and loads data from diverse sources.

oFramework Optimization: Use AWS services such as AWS Glue, Lambda, EC2 and Step Functions to build highly scalable, resilient, and automated data pipelines.

Data Quality & Validation:

oValidation Processes: Develop and implement automated data quality checks, validation routines, and error-handling mechanisms to ensure the accuracy and integrity of incoming data.

oMonitoring & Reporting: Establish comprehensive monitoring, logging, and alerting systems to proactively identify and resolve data quality issues.

Collaboration & Agile Practices:

oCross-Functional Communication: Work closely with business stakeholders, and application development teams to understand requirements and translate them into technical solutions.

oAgile Development: Participate in sprint planning, code reviews, and agile ceremonies, while contributing to continuous improvement initiatives and CI/CD pipeline development (using tools like GitLab).

Job Specification / Skills and Competencies:

Experience & Technical Skills:

oProfessional Background: At least 5 years of relevant experience in data engineering with a strong emphasis on analytical platform development.

oProgramming Skills: Proficiency in PySpark, Snowflake for developing ETL processes and handling large-scale data manipulation.

Job Description

oAWS Expertise: Extensive experience using AWS services including AWS Glue, Lambda, Step Functions, and S3 to build and manage data ingestion frameworks.

oData Platforms: Familiarity with big data systems (e.g., AWS EMR, Apache Spark, Apache Iceberg).

oCI/CD & Agile: Hands-on experience with CI/CD pipelines (preferably with GitLab) and Agile development methodologies.