Spark & Delta Lake Professional

IDESLABS PRIVATE LIMITED

7 - 12 years

6 - 10 Lacs

hyderabad

Posted:3 days ago| Platform:

Apply

Skills Required

scala java git spark python ab initio schema hive continuous integration ci/cd data warehousing dbms apache pig spark core data modeling flume hadoop etl big data hbase datastage oozie data quality troubleshooting sqoop informatica etl process

Work Mode

Work from Office

Job Type

Full Time

Job Description

Looking for a skilled professional with expertise in Spark and Delta Lake to join our team. The ideal candidate will have 7-12 years of experience.

Roles and Responsibility

Design and implement data pipelines using Spark and Delta Lake as the data storage layer.
Develop and optimize Spark applications using Python, Scala, or Java.
Work with large datasets to ensure data quality and integrity.
Implement data partitioning and schema evolution techniques using Delta Lake.
Collaborate with cross-functional teams to integrate Spark applications with ETL processes.
Troubleshoot and resolve issues related to Spark application performance and data quality.

Job Requirements

Strong understanding of Spark core concepts including RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming.
Experience with Spark optimization techniques to improve application performance.
Deep knowledge of Delta Lake features such as time travel, schema evolution, and data partitioning.
Proficiency in Python, Scala, or Java for Spark development and integrating with ETL processes.
Knowledge of data ingestion techniques from various sources (flat files, CSV, API, database).
Understanding of data warehouse concepts and data modeling techniques.
Expertise in Git for code management.
Familiarity with CI/CD pipelines and containerization technologies.
Nice to have experience using data integration tools like DataStage/Prophecy/Informatica/Ab Initio.

More Jobs at IDESLABS PRIVATE LIMITED

Bank Fast Platform Engineer

Bengaluru

3.0 - 7.0 yrs

INR 5 - 9 Lacs

Risk Business Analyst

Bengaluru

5.0 - 10.0 yrs

INR 8 - 13 Lacs

Salesforce Commerce Cloud Professional

Hyderabad

3.0 - 5.0 yrs

INR 4 - 6 Lacs

Python Scripting with Terraform Professional

Hyderabad

3.0 - 5.0 yrs

INR 4 - 6 Lacs

Nodejs / AWS Developer

Hyderabad

3.0 - 5.0 yrs

INR 4 - 6 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Java Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.