AWS Data Engineer – Data Quality & Reliability Engineering

Senior Data Engineer with a deep focus on data quality, validation frameworks, and reliability engineering. This role will be instrumental in ensuring the accuracy, integrity, and trustworthiness of data assets across our cloud-native infrastructure. The ideal candidate combines expert-level Python programming with practical experience in data pipeline engineering, API integration, and managing cloud-native workloads on AWS and Kubernetes.

• Design, develop, and deploy automated data validation and quality frameworks using Python.
• Build scalable and fault-tolerant data pipelines that support quality checks across data ingestion, transformation, and delivery.
• Integrate with REST APIs to validate and enrich datasets across distributed systems.
• Deploy and manage validation workflows using AWS services (EKS, EMR, EC2) and Kubernetes clusters.
• Collaborate with data engineers, analysts, and DevOps to embed quality checks into CI/CD and ETL pipelines.
• Develop monitoring and alerting systems for real-time detection of data anomalies and inconsistencies.
• Write clean, modular, and reusable Python code for automated testing, validation, and reporting.
• Lead root cause analysis for data quality incidents and design long-term solutions.
• Maintain detailed technical documentation of data validation strategies, test cases, and architecture.
• Promote data quality best practices and evangelize a culture of data reliability within the engineering teams.

Required Skills:

• Experience with data quality platforms such as Great Expectations, Collibra Data Quality, or similar tools.
• Proficiency in Docker and container lifecycle management.
• Familiarity with serverless compute environments (e.g., AWS Lambda, Azure Functions), Python, PySpark
• Relevant certifications in AWS, Kubernetes, or data quality technologies.
• Prior experience working in big data ecosystems and real-time data environments.