Job
Description
Project Overview: This role is part of Project Acuity , specifically within the PASD Data Platform workstream. The PASD Data Platform is a centralized web application designed for internal PASD users across the Recruitment Business to support marketing and operational use cases. Building a database at the patient level will provide significant benefits to PASD s future reporting capabilities and enhance engagement with external stakeholders. Timings: Minimum of 6 hours overlap with US timings (2 PM to 11 PM IST) Role Scope & Deliverables: As a Sr. Data Engineer on the AWS Cloud team, you will be responsible for designing and developing data ingestion pipelines that integrate data from disparate sources into the cloud. You will lead the delivery of data products , leveraging Cloud Native strategies and best practices, drawing from 15+ years of IT experience. Must-Have Skills: 15+ years of experience in designing and delivering Distributed Systems capable of handling petabytes of data. 10+ years of experience in Data Lake development with data ingestion from various sources such as relational databases, flat files, APIs, and streaming data. Expertise in designing and developing Data Platforms and ingesting data into the AWS Cloud . Strong hands-on expertise in AWS core services , including: AWS IAM, VPC, EC2, EKS/ECS, S3, RDS, DMS, Lambda, CloudWatch, CloudFormation, CloudTrail. Proficiency in Python and PySpark for efficient data processing (preferably Python). Experience in architecting and implementing ETL pipelines using AWS Glue with well-defined extraction, transformation, and loading procedures. 15+ years of experience with Infrastructure-as-Code (IaC) tools like Terraform. 10+ years of experience in developing CI/CD pipelines (GitHub Actions, Jenkins). Strong expertise in Event-Driven Distributed Systems in the Cloud using Serverless Architecture . Experience working with Infrastructure teams for AWS service provisioning (databases, network design, IAM roles, AWS clusters). 2-3 years of experience with DocumentDB . Ability to design, orchestrate, and schedule jobs using Airflow . Experience with AWS AI Services , including AWS Entity Resolution and AWS Comprehend . Ability to run custom LLMs using Amazon SageMaker . Experience using Large Language Models (LLMs) for Data Classification and PII Data Identification . Nice-to-Have Skills: 10+ years of experience developing Data Audit, Compliance, and Retention standards for Data Governance, along with automation of governance processes . Experience in data modeling with NoSQL Databases such as DocumentDB . Experience with column-oriented data file formats like Apache Parquet and Apache Iceberg for analytical datasets. Expertise in developing Retrieval-Augmented Generation (RAG) and Agentic Workflows to provide context to LLMs based on proprietary enterprise data. Ability to develop re-ranking strategies using Index and Vector stores to improve LLM output quality .
