Python Developer (Airflow & ETL Specialist)

Experience: 5+ Years

Location: Gurgaon / Remote (as applicable)

About the Role

We are looking for an experienced Python Developer with strong hands-on experience in Apache Airflow and ETL pipeline development. The ideal candidate will be responsible for designing, building, and maintaining data workflows, ensuring scalability, performance, and reliability of data processing systems. This role requires a deep understanding of data integration, orchestration, and automation within modern cloud or on-prem data ecosystems.

Key Responsibilities

• Design, develop, and maintain ETL workflows and data pipelines using Python and Apache Airflow.
• Integrate data from multiple sources (databases, APIs, files, etc.) and ensure data quality, consistency, and integrity.
• Develop reusable Python scripts, operators, and DAGs in Airflow for scheduling and orchestration.
• Optimize ETL processes for performance and scalability across large datasets.
• Monitor, troubleshoot, and improve existing Airflow jobs and pipelines.
• Collaborate with data engineers, analysts, and product teams to understand data requirements and translate them into robust data flows.
• Work with SQL and NoSQL databases for data extraction and transformation.
• Implement logging, alerting, and error handling for ETL processes.
• Ensure adherence to best practices for code quality, testing, and version control (Git).

Required Skills & Experience

• 5+ years of experience in Python-based data engineering and ETL development.
• Strong proficiency in Apache Airflow for workflow orchestration and DAG management.
• Solid experience with data transformation and integration processes (ETL/ELT).
• Strong SQL skills and understanding of relational database concepts (PostgreSQL, MySQL, etc.).
• Experience with cloud data platforms (AWS, GCP, or Azure) and cloud storage solutions (S3, GCS, Azure Blob).
• Familiarity with APIs, JSON, and data serialization formats like Parquet or Avro.
• Experience with CI/CD tools and version control (Git).
• Strong problem-solving skills and ability to work in fast-paced, agile environments.

Good to Have

• Exposure to containerization (Docker, Kubernetes).
• Experience with data warehouses (Snowflake, Redshift, BigQuery).
• Knowledge of data modeling and data governance principles.
• Familiarity with monitoring tools like Prometheus, Grafana, or Airflow UI metrics.