Data Engineer PySpark

IDESLABS PRIVATE LIMITED

4 - 6 years

5 - 8 Lacs

hyderabad

Posted:3 weeks ago| Platform:

Apply

Skills Required

data engineer airflow s3 pyspark java pr data pipelines scala kafka code quality apache sql apache spark design hive data quality data lakehouse docker dbt spark data processing python kubernetes

Work Mode

Work from Office

Job Type

Full Time

Job Description

We are looking for a skilled Data Engineer with expertise in Apache Spark and Hive to join our team at IDESLABS PRIVATE LIMITED. The ideal candidate will have 4-6 years of experience.

Roles and Responsibility

Design, develop, and implement data pipelines using Apache Spark and Hive.
Collaborate with cross-functional teams to identify and prioritize project requirements.
Develop and maintain large-scale data processing systems using Python, Scala, or Java.
Work with Kafka, Solace, and object stores such as S3 and MinIO.
Ensure data quality and integrity by implementing robust testing and validation frameworks.
Participate in code reviews and contribute to improving overall code quality.

Job Requirements

Strong expertise in Apache Spark (batch + streaming) and Hive.
Proficiency in Python, Scala, or Java.
Knowledge of orchestration tools like Airflow or Control-M and SQL transformation frameworks like DBT.
Experience working with Docker/Kubernetes for deployment.
Hands-on experience with data Lakehouse formats such as Iceberg, Delta Lake, or Hudi.
Ability to work in a fast-paced environment and meet deadlines.

More Jobs at IDESLABS PRIVATE LIMITED

Bank Fast Platform Engineer

Bengaluru

3.0 - 7.0 yrs

INR 5 - 9 Lacs

Risk Business Analyst

Bengaluru

5.0 - 10.0 yrs

INR 8 - 13 Lacs

Salesforce Commerce Cloud Professional

Hyderabad

3.0 - 5.0 yrs

INR 4 - 6 Lacs

Python Scripting with Terraform Professional

Hyderabad

3.0 - 5.0 yrs

INR 4 - 6 Lacs

Nodejs / AWS Developer

Hyderabad

3.0 - 5.0 yrs

INR 4 - 6 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Data Engineer Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.