Big Data Developer - Hadoop / HDFS

Senior Data Engineer – Vendor Experience: 6–12 Years

Role Summary

We are seeking Senior Data Engineer resources to work on the migration of applications from our legacy Cloudera environment to the new Kubernetes-based data platform. The role requires strong hands-on development skills in data engineering, with the ability to deliver high-quality pipelines under guidance from internal leads.

Key Responsibilities

Develop and optimize data pipelines using Spark 3.5 and Python/Scala.

Migrate existing Hive, Spark, and Control-M jobs to Airflow and DBT-based workflows.

Integrate data pipelines with messaging systems (Kafka, Solace) and object stores (S3, MinIO).

Troubleshoot and optimize distributed jobs running in Kubernetes environments.

Collaborate closely with internal leads and architects to implement best practices.

Design and implement migration/acceleration framework to automate end to end migration.

Continuous enhancements to the frameworks to ensure the stability, scalability and support for diverse use cases and scenarios.

Work with various data applications to enable and support the migration process.

Deliver assigned migration tasks within agreed timelines.

Required Skills

6–9 years of hands-on data engineering experience.

Strong expertise in Apache Spark (batch + streaming) and Hive.

Proficiency in Python, Scala, or Java.

Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred).

Experience working with Kafka, Solace, and object stores (S3, MinIO).

Exposure to Docker/Kubernetes for deployment.

Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi