Airflow Developer

We are looking for an experienced Airflow Developer with 4-5 years of expertise to create, implement, and manage efficient, scalable, and dependable workflows utilizing Apache Airflow. The perfect candidate should possess a solid comprehension of data pipelines, ETL procedures, and orchestration strategies. Your role will involve collaborating with diverse teams to ensure smooth integration and enhancement of workflows within our data ecosystem. Your responsibilities will include designing and crafting workflows, Directed Acyclic Graphs (DAGs), and custom operators employing Apache Airflow. You will partner with data engineers, analysts, and stakeholders to grasp requirements and transform them into scalable workflows. Monitoring, troubleshooting, and optimizing Airflow pipelines for reliability and performance will be crucial. Integration of Airflow with different data sources, storage systems, and APIs will also be part of your duties. Moreover, you will be expected to follow best practices for code quality, version control, and deployment processes, as well as produce comprehensive documentation for workflows, configurations, and procedures. Ensuring the scalability, security, and maintainability of data pipelines by implementing suitable solutions will be essential. Additionally, you will be responsible for managing and updating the Airflow environment, including upgrades and plugins. The ideal candidate should have 4 to 5 years of experience in constructing and supervising workflows using Apache Airflow. Proficiency in Python programming for Airflow DAGs and custom operator development is required. A strong understanding of ETL/ELT processes, data pipeline orchestration, SQL, and databases (e.g., PostgreSQL, MySQL, or NoSQL) is essential. Familiarity with cloud platforms (AWS, GCP, or Azure) and their data services, as well as knowledge of CI/CD pipelines and version control systems (e.g., Git), is preferred. Strong problem-solving abilities and attention to detail are highly valued. Desirable qualifications include experience with containerization tools like Docker, orchestration systems like Kubernetes, big data technologies such as Spark, Hadoop, or Kafka, and data warehousing solutions like Snowflake, Redshift, or BigQuery. Knowledge of monitoring and alerting tools (e.g., Prometheus, Grafana) would be an advantage.,