Senior Data Engineer

Role & responsibilities

As a Senior Data Engineer, you will be responsible for designing, building, and optimizing data pipelines and lakehouse architectures on AWS. You will ensure data availability, quality, lineage, and governance across analytical and operational platforms. Your expertise will enable scalable, secure, and cost-effective data solutions that power advanced analytics and business intelligence.

Responsibilities :

• Implement and manage S3 (raw, staging, curated zones), Glue Catalog, Lake Formation, and Iceberg/Hudi/Delta Lake for schema evolution and versioning.
  
• Develop PySpark jobs on Glue/EMR, enforce schema validation, partitioning, and scalable transformations.
  
• Build workflows using Step Functions, EventBridge, or Airflow (MWAA), with CI/CD deployments via CodePipeline & CodeBuild.
  
• Apply schema contracts, validations (Glue Schema Registry, Deequ, Great Expectations), and maintain lineage/metadata using Glue Catalog or third-party tools (Atlan, OpenMetadata, Collibra).
  
• Enable Athena and Redshift Spectrum queries, manage operational stores (DynamoDB/Aurora), and integrate with OpenSearch for observability.
  
• Design efficient partitioning/bucketing strategies, adopt columnar formats (Parquet/ORC), and implement spot instance usage/bookmarking.
  
• Enforce IAM-based access policies, apply KMS encryption, private endpoints, and GDPR/PII data masking.
  
• Prepare Gold-layer KPIs for dashboards, forecasting, and customer insights with QuickSight, Superset, or Metabase.
  
• Partner with analysts, data scientists, and DevOps to enable seamless data consumption and delivery.

Preferred candidate profile

• Hands-on expertise with AWS data stack (S3, Glue, Lake Formation, Athena, Redshift, EMR, Lambda).
  
• Strong programming skills in PySpark & Python for ETL, scripting, and automation.
  
• Proficiency in SQL (CTEs, window functions, complex aggregations).
  
• Experience in data governance, quality frameworks (Deequ, Great Expectations).
  
• Knowledge of data modeling, partitioning strategies, and schema enforcement.
  
• Familiarity with BI integration (QuickSight, Superset, Metabase).

Benefits

• This role offers the flexibility of working remotely in India.