Senior Data Engineer

We are seeking a highly skilled Senior Data Engineer with deep expertise in AWS data services, data wrangling using Python & PySpark, and a solid understanding of data governance, lineage, and quality frameworks. The ideal candidate will have a proven track record of delivering end-to-end data pipelines for logistics, supply chain, enterprise finance, or B2B analytics use cases.

Role & responsibilities

• Design, build, and optimize ETL pipelines using AWS Glue 3.0+ and PySpark.
• Implement scalable and secure data lakes using Amazon S3, following bronze/silver/gold zoning.
• Write performant SQL using AWS Athena (Presto) with CTEs, window functions, and aggregations.
• Take full ownership from ingestion transformation validation metadata documentation dashboard-ready output.
• Build pipelines that are not just performant, but audit-ready and metadata-rich from the first version.
• Integrate classification tags and ownership metadata into all columns using AWS Glue Catalog tagging conventions.
• Ensure no pipeline moves to QA or BI team without validation logs and field-level metadata completed.
• Develop job orchestration workflows using AWS Step Functions integrated with EventBridge or CloudWatch.
• Manage schemas and metadata using AWS Glue Data Catalog.
• Take full ownership from ingestion transformation validation metadata documentation dashboard-ready output.
• Ensure no pipeline moves to QA or BI team without validation logs and field-level metadata completed.
• Enforce data quality using Great Expectations, with checks for null %, ranges, and referential rules.
• Ensure data lineage with OpenMetadata or Amundsen and add metadata classifications (e.g., PII, KPIs).
• Collaborate with data scientists on ML pipelines, handling JSON/Parquet I/O and feature engineering.
• Must understand how to prepare flattened, filterable datasets for BI tools like Sigma, Power BI, or Tableau.
• Interpret business metrics such as forecasted revenue, margin trends, occupancy/utilization, and volatility.
• Work with consultants, QA, and business teams to finalize KPIs and logic.
• Build pipelines that are not just performant, but audit-ready and metadata-rich from the first version.
• Integrate classification tags and ownership metadata into all columns using AWS Glue Catalog tagging conventions.

Preferred candidate profile

• Strong hands-on experience with AWS: Glue, S3, Athena, Step Functions, EventBridge, CloudWatch, Glue Data Catalog.
• Programming skills in Python 3.x, PySpark, and SQL (Athena/Presto).
• Proficient with Pandas and NumPy for data wrangling, feature extraction, and time series slicing.
• Strong command over data governance tools like Great Expectations, OpenMetadata / Amundsen.
• Familiarity with tagging sensitive metadata (PII, KPIs, model inputs).
• Capable of creating audit logs for QA and rejected data.
• Experience in feature engineering rolling averages, deltas, and time-window tagging.

BI-readiness with Sigma, with exposure to Power BI / Tableau (nice to have).