Data Engineer

Pipeline Design & Development
Design, build, and maintain data ingestion and transformation pipelines using MWAA (Apache Airflow) and AWS Redshift.Develop Airflow DAGs to orchestrate data extraction from diverse sources (e.g., Oracle EBS, APIs, flat files, or S3).Implement transformation logic using SQL, Python, or Redshift stored procedures to curate data for analytical consumption.Create and manage staging, curated, and semantic data layers in Redshift aligned with the Medallion architecture principles.Integrate data ingestion pipelines with AWS services (e.g., S3, Glue, Lambda, Secrets Manager

Performance, Optimization & Maintenance
Tune Redshift queries and schema for high performance and cost efficiency.Implement Airflow monitoring, logging, and alerting using CloudWatch or similar tools.Maintain and version-control Airflow DAGs and Redshift scripts in Git.Continuously improve data pipeline reliability, error handling, and observability.Establish data validation checks and ensure pipeline SLAs are met.Integration & SecurityConfigure and manage secure data access between MWAA, S3, and Redshift using IAM roles and Secrets Manager.Collaborate with data architects and DevOps teams to implement CI/CD automation for pipeline deployment.Work closely with business analysts and data consumers to ensure datasets meet functional and performance requirements.Testing & ValidationPerform unit, integration, and regression testing on data pipelines and transformations.Validate data quality, schema consistency, and referential integrity between source and Redshift layers.Participate in User Acceptance Testing (UAT) and resolve issues collaboratively with stakeholders.Document data flows, dependencies, and operational runbooks for pipeline management.Required Skills & Experience58 years of experience in data engineering or related roles.Proven experience with Apache Airflow / MWAA for building and orchestrating data workflows.Strong proficiency in Amazon Redshift including schema design, SQL optimization, and performance tuning.Solid understanding of AWS services: S3, Glue, Lambda, CloudWatch, Secrets Manager, and IAM.Experience with Python (for Airflow DAGs and data transformations).Strong SQL development skills for ELT logic in Redshift.Familiarity with Medallion architecture (Raw, Curated, Gold) or other layered data design patterns.Hands-on experience managing data ingestion from enterprise systems (e.g., Oracle EBS, SAP, Salesforce) is a plus.Understanding of data quality frameworks, testing automation, and operational monitoring.

Kindly share your updates resume to Sheena.James@mastek.com

We will connect you Shortly .