Senior Data Engineer Apache Airflow Expert

We are looking for an experienced Apache Airflow Expert to design and manage complex data workflows in our AWS-based data platform. The ideal candidate will have deep expertise in Airflow orchestration, PySpark ETL development, dbt transformations, and Snowflake data pipelines, with a strong background in data engineering and AWS cloud technologies.

Key Responsibilities:

1. Design, build, and maintain complex Airflow DAGs for orchestrating ETL/ELT workflows across AWS, PySpark, dbt, and Snowflake.


2. Integrate and optimize data pipelines using AWS services (S3, Glue, Lambda, Redshift) and ensure reliability, scalability, and cost efficiency.


3. Develop and manage PySpark-based data processing workflows for batch and streaming data.


4. Implement and orchestrate dbt transformations within Airflow for modular and version-controlled SQL workflows in Snowflake.


5. Ensure data quality, observability, and lineage tracking with robust monitoring, logging, and alerting systems.


6. Collaborate with data, analytics, and DevOps teams to streamline CI/CD, automate deployments, and enforce data governance standards.

Required Skills & Experience:

• 6 to 8 years of experience in data engineering with hands-on expertise in Apache Airflow (core and MWAA).


• Strong Python and PySpark skills for data processing and Airflow customization (custom operators, sensors, hooks).


• Proven experience with dbt (data build tool) for SQL transformation pipelines and Snowflake for data warehousing.


• Deep understanding of AWS services: S3, Glue, Lambda, ECS/EKS, Redshift, and IAM.


• Experience with Docker, Terraform, and CI/CD pipelines (GitHub Actions, Jenkins, or similar).