Job
Description
Role Overview We are looking for a Senior ETL Engineer with deep expertise in Apache Airflow to design, build, and manage complex data workflows and pipelines across cloud platforms. The ideal candidate will bring strong experience in Python, SQL, and cloud-native tools (AWS/GCP) to deliver scalable and reliable data infrastructure, supporting analytics, reporting, and operational systems. Key Responsibilities Design, implement, and optimize scalable ETL/ELT workflows using Apache Airflow DAGs . Build and maintain data pipelines with Python and SQL , integrating multiple data sources. Develop robust solutions for pipeline orchestration, failure recovery, retries , and notifications. Leverage AWS or GCP services (e.g., S3, Lambda, BigQuery, Cloud Functions, IAM). Integrate with internal and external data systems via secure REST APIs and Webhooks . Monitor Airflow performance, manage DAG scheduling, and resolve operational issues. Implement observability features like logging, metrics, alerts, and pipeline health checks. Collaborate with analytics, data science, and engineering teams to support data needs. Drive automation and reusability across pipeline frameworks and templates. Ensure data quality, governance, compliance , and lineage across ETL processes. Required Skills 5+ years of experience in ETL/Data Engineering with hands-on Airflow expertise. Strong programming skills in Python , with solid experience writing production scripts. Proficiency in SQL for data manipulation, joins, and performance tuning. Deep knowledge of Apache Airflow scheduling, sensors, operators, XComs, and hooks. Experience working on cloud platforms like AWS or GCP in data-heavy environments. Comfort with REST APIs , authentication protocols, and data integration techniques. Knowledge of CI/CD tools, Git, and containerization (Docker, Kubernetes). Nice to Have Familiarity with dbt, Snowflake, Redshift, BigQuery, or other modern data platforms. Experience with Terraform or infrastructure-as-code for Airflow deployment. Understanding of data privacy, regulatory compliance (GDPR, HIPAA), and metadata tracking.
