Data Engineer

• Build scalable

  data ingestion pipelines

  for relational systems and unstructured/semi-structured sources.
• Design and optimize lakehouse workloads using

  Apache Iceberg

  .
• Optimize

  table layouts, partitioning strategies, compaction, and schema evolution

  for Iceberg-based datasets.
• Implement best practices for

  versioning, time travel, incremental processing, and ACID guarantees

  .
• Develop and tune

  Apache Spark

  (batch & streaming) for performant, scalable data processing.
• Work with

  AWS

  services (Glue, EMR, Lambda, Step Functions, S3) focusing on

  cost optimization

  and

  performance tuning

  .
• Implement and manage

  Kafka

  and

  Kafka streaming

  pipelines for large-scale real-time processing.
• Build and orchestrate data workflows using

  DBT

  and

  Airflow

  .
• Leverage

  AI/LLM automation

  to accelerate ETL/ELT development, validation, and debugging.
• Implement observability monitoring, logging, and alerting to ensure platform stability and reliability.
• Participate in code reviews to ensure code quality and share best practices with the team.

Required Skills

• Strong SQL, Python, and Spark experience
• Deep experience ingesting and processing structured & unstructured data
• Strong knowledge of

  Apache Iceberg

  and lakehouse architectures
• Hands-on AWS experience with tuning and cost optimization
• Expertise with

  Kafka, Kafka Streaming, DBT, and Airflow

• Solid understanding of distributed systems and data engineering best practices