Data Engineer

Key Responsibilities

• Migration Strategy & Execution
  • Design and implement data ingestion pipelines to extract data from Oracle into GCS/Iceberg.
  • Migrate and modernize existing Oracle schemas, partitions, and materialized views into Iceberg tables.
  • Define CDC (Change Data Capture) strategies using custom ETL.
• Data Lakehouse Architecture
  • Configure and optimize Trino clusters (coordinator/worker, Helm charts, autoscaling).
  • Design partitioning, compaction, and clustering strategies for Iceberg tables.
  • Implement schema evolution, time-travel, and versioning capabilities.
• Performance & Cost Optimization
  • Benchmark Trino query performance vs Oracle workloads.
  • Tune Trino/Iceberg for large-scale analytical queries, minimizing query latency and storage costs.
• Data Quality, Metadata & Governance
  • Integrate Iceberg datasets with metadata/catalog services (Postgre/Hive Metastore, or Glue).
  • Ensure compliance with governance, observability, and lineage requirements.
  • Define and enforce standards for unit testing, regression testing, and data validation.
• Collaboration & Delivery
  • Support existing reporting workloads (regulatory reporting, DWH) during and after migration.
  • Document architecture, migration steps, and provide knowledge transfer.

• 5 + years of experience
• Prior experience migrating financial/regulatory datasets.
• Experience with Regulatory Reporting or similar enterprise workloads.
• Familiarity with large-scale performance benchmarking and cost modelling.

• Core Expertise:
  • Strong hands-on experience with Trino/Presto, Apache Iceberg, and Oracle SQL/PLSQL.
  • Proven experience with data lakehouse migrations at scale (50 TB+).
  • Proficiency in Parquet formats.
• Programming & Tools:
  • Solid coding skills in Java, Scala, or Python for ETL/ELT pipeline development.
  • Experience with orchestration (Spark).
  • Familiarity with CDC tools, JDBC connectors, or custom ingestion frameworks.
• Cloud & DevOps:
  • Strong background in GCP (preferred) or AWS/Azure cloud ecosystems.
  • Experience with Kubernetes, Docker, Helm charts for deploying Trino workers.
  • Knowledge of CI/CD pipelines and observability tools.
• Soft Skills:
• Strong problem-solving mindset with ability to manage dependencies and shifting scopes.
• Clear documentation and stakeholder communication skills.
• Ability to work in tight delivery timelines with global teams.