Data Engineer

immediate joiners only

Senior Data Engineer (Airflow & PostgreSQL Specialist)

Role Overview

Senior Data Engineer

Key Responsibilities

1. Advanced Airflow Orchestration (Core)

• Architect and manage complex

  Airflow DAGs

  , moving beyond basic scheduling to implement dynamic task generation and custom operators.
• Implement high-level dependency management, including

  Cross-DAG dependencies

  , sensors, and sophisticated retry/alerting logic.
• Optimize the Airflow environment (KubernetesExecutor/CeleryExecutor) for reliability and resource efficiency.

2. PostgreSQL Expert-Level Development

• Lead the design of

  PostgreSQL schemas

  , focusing on performance for both OLTP and OLAP patterns.
• Deep-dive into

  Query Optimization

  : Analyze EXPLAIN plans, implement advanced indexing strategies, and manage table partitioning.
• Ensure data integrity through advanced SQL techniques, stored procedures, and rigorous validation logic.

3. Scalable ETL/ELT & Big Data

• Develop robust pipelines using

  Python, Spark, and SQL

  to move data across Cloud Storage, Kafka, and Snowflake/Redshift.
• Design and maintain

  Data Lakes

  and

  Warehouses

  that support real-time and batch processing.

4. Data Governance & Reliability

• Implement

  Data Quality (DQ) frameworks

  (e.g., Great Expectations) to ensure accuracy and completeness.
• Establish data lineage, metadata management, and compliance standards (GDPR/FedRAMP).
• Monitor system health and implement automated recovery strategies for mission-critical pipelines.

Required Technical Skills

• Databases:

  Deep expertise in

  PostgreSQL

  (Performance tuning, Partitioning, Vacuuming). Proficiency in NoSQL (MongoDB/Cassandra) is a plus.

• Orchestration:

  Expert-level

  Apache Airflow

  (DAG design, Monitoring, Scaling).

• Languages:

  Advanced

  Python

  and

  SQL

  (Mandatory). Familiarity with Java or Scala.

• Big Data:

  Hands-on experience with

  Apache Spark

  and

  Kafka

  .

• Cloud:

  AWS (S3, Redshift, Glue) or Azure (ADLS, Synapse).

• DevOps:

  Strong understanding of

  Docker, Kubernetes

  , and CI/CD pipelines.

Qualifications

• Bachelors or Masters in Computer Science or a related technical field.

• 8+ years

  of experience in Data Engineering, with at least 3 years focused on high-scale Airflow/Postgres environments.
• Proven track record of architecting data systems from the ground up.