Lead Data Engineer AWS

As a Senior Data Engineer at our company, you will play a crucial role in leading the development of a scalable data ingestion framework and ensuring high data quality and validation. Your responsibilities will include: - Architecting, developing, and maintaining end-to-end data ingestion framework for extracting, transforming, and loading data from diverse sources - Utilizing AWS services such as Glue, Lambda, EMR, ECS, EC2, and Step Functions to build scalable, resilient automated data pipelines - Developing and implementing automated data quality checks, validation routines, and error-handling mechanisms - Establishing comprehensive monitoring, logging, and alerting systems for data quality issues - Architecting and developing secure, high-performance APIs for data services integration - Creating thorough API documentation and establishing standards for security, versioning, and performance - Working closely with business stakeholders, data scientists, and operations teams to understand requirements - Participating in sprint planning, code reviews, and agile ceremonies - Contributing to CI/CD pipeline development using GitLab Qualifications and Skills required for this role include: - 5+ years experience in data engineering with a focus on analytical platform development - Proficiency in Python and/or PySpark - Strong SQL skills for ETL processes and large-scale data manipulation - Extensive experience with AWS services like Glue, Lambda, Step Functions, and S3 - Familiarity with big data systems such as AWS EMR, Apache Spark, and Apache Iceberg - Database experience with DynamoDB, Aurora, Postgres, or Redshift - Designing and implementing RESTful APIs - Hands-on experience with CI/CD pipelines, preferably GitLab - Agile development methodology experience - Strong problem-solving abilities, attention to detail, communication, and interpersonal skills - Ability to work independently and collaboratively - Capacity to quickly learn and adapt to new technologies - Total experience of 10+ years It is preferable to have a Bachelors/Masters in Computer Science, Data Engineering, or a related field. Additionally, experience with AWS services like Kinesis, Firehose, and SQS, familiarity with data lakehouse architectures, modern data quality frameworks, and proactive data quality management in multi-cluster environments would be beneficial. Adherence to Information Security Management policies and procedures is essential in this role.,