Big Data Engineer Lead

Brillio

6 - 10 years

20 - 35 Lacs

bengaluru

Posted:-1 days ago| Platform:

Apply

Skills Required

pyspark hive iceberg hadoop python sql

Work Mode

Hybrid

Job Type

Full Time

Job Description

Key Skills - Hadoop, PySpark, Python, SQL, Apache Iceberg (niche) , Hive

Requirements

Solid knowledge of Hadoop ecosystem such as Hive, Iceberg, Spark SQL and Proficiency in Python, Unix ,SQL
Experience with Apache Kafka, Apache Flink, and other relevant streaming technologies.
Strong hands-on experience with Hive and SQL for querying and data transformation
Proficiency in Python for data manipulation and automation
Expertise in Apache Spark (batch and streaming)
Deep understanding of Hadoop ecosystem (HDFS, YARN, MapReduce)
Experience working with Kafka for streaming data pipelines
Experience with workflow orchestration tools (Airflow, Oozie, etc.)
Knowledge of cloud-based big data platforms (AWS EMR, GCP Dataproc, Azure HDInsight)
Fam
iliarity with CI/CD pipelines and version control (Git)

Responsibilities

Design, develop, and maintain scalable data pipelines and ETL processes.
Work with large datasets using Hadoop ecosystem tools (Hive, Spark).
Build and optimize real-time and batch data processing solutions using Kafka and Spark Streaming.
Write efficient, high-performance SQL queries to extract, transform, and load data.
Develop reusable data frameworks and utilities in Python.
Collaborate with data scientists, analysts, and product teams to deliver reliable data solutions.
Monitor, troubleshoot, and optimize big data workflows for performance and cost efficiency.

More Jobs at Brillio

React Js Frontend Developer

Pune, Bengaluru

4 - 8 yrs

INR 14 - 20 Lacs

Java Full Stack Developer

Pune, Bengaluru

3 - 8 yrs

INR 14 - 20 Lacs

Senior Software Development Lead - R01536938

Gurgaon, Haryana, India

Experience: Not specified

Salary: Not disclosed

Angular Lead

Hyderabad, Pune, Bengaluru

6 - 10 yrs

INR 14 - 20 Lacs

Network Perimeter Browsing - R01542500

Bengaluru, Karnataka, India

Experience: Not specified

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.