Data Engineer

Role & responsibilities

• Design, develop, and optimize data pipelines using PySpark and AWS services.
• Implement and manage data workflows, ETL processes, and schema validations.
• Ensure data quality, integrity, and consistency by applying validation frameworks (e.g., Deequ, Great Expectations).
• Work with data in different zones (raw, staging, curated) and implement partitioning, schema evolution, and governance best practices.
• Collaborate with analysts, data scientists, and cross-functional teams to deliver reliable and consumable datasets.
• Support data lineage, metadata management, and compliance requirements.
• Strong hands-on experience with PySpark & Python for ETL, data transformation, and automation.
• Proficiency in SQL (joins, window functions, aggregations).
• Experience with AWS data stack (S3, Glue, Athena, Redshift, EMR, or similar).
• Knowledge of data quality frameworks (Deequ, Great Expectations) and data governance principles.
• Good understanding of data modeling, partitioning strategies, and schema enforcement.

Preferred candidate profile

• Exposure to workflow orchestration tools (Airflow, Step Functions, or similar).
• Experience with BI/reporting integrations (QuickSight, Superset, or Metabase).
• Familiarity with real-time data ingestion (Kafka, Kinesis, MSK).