Big Data Engineer

Brillio

5 - 8 years

18 - 32 Lacs

bengaluru

Posted:2 hours ago| Platform:

Apply

Skills Required

hive hadoop iceberg hdfs sql

Work Mode

Hybrid

Job Type

Full Time

Job Description

Skills - Hadoop , Hive , HDFS , ETL Processes , CI/CD

Data Engineer

Job Description:

As a Data Engineer on our team, you will work on our Hadoop-based data warehouse, contributing to scalable and reliable big data solutions for analytics and business insights. This is a hands-on role focused on building, optimizing, and maintaining large data pipelines and warehouse infrastructure.

Key Responsibilities

Design, develop, and maintain robust data pipelines in Hadoop and related ecosystems, ensuring data reliability, scalability, and performance.
Implement data ETL processes for batch and streaming analytics requirements.
Optimize and troubleshoot distributed systems for ingestion, storage, and processing.
Collaborate with data engineers, analysts, and platform engineers to align solutions with business needs.
Ensure data security, integrity, and compliance throughout the infrastructure.
Maintain documentation and contribute to architecture reviews.
Participate in incident response and operational excellence initiatives for the data warehouse.
Continuously learn mindset and apply new Hadoop ecosystem tools and data technologies.

Required Skills and Experience

Proficiency in Hadoop ecosystems such as Spark, HDFS, Hive, Iceberg, Spark SQL.
Extensive experience with Apache Kafka, Apache Flink, and other relevant streaming technologies.
Proven ability to design and implement automated data pipelines and materialized views.
Proficiency in Python, Unix or similar languages.
Good understanding of SQL oracle, SQL server or similar languages.
Ops & CI/CD: Monitoring (Prometheus/Grafana), logging, pipelines (Jenkins/GitHub Actions).
Core Engineering: Data structures/algorithms, testing (JUnit/pytest), Git, clean code.
5+ years of directly applicable experience
BS in Computer Science, Engineering, or equivalent experience.

More Jobs at Brillio

React Js Frontend Developer

Pune, Bengaluru

4 - 8 yrs

INR 14 - 20 Lacs

Java Full Stack Developer

Pune, Bengaluru

3 - 8 yrs

INR 14 - 20 Lacs

Senior Software Development Lead - R01536938

Gurgaon, Haryana, India

Experience: Not specified

Salary: Not disclosed

Angular Lead

Hyderabad, Pune, Bengaluru

6 - 10 yrs

INR 14 - 20 Lacs

Network Perimeter Browsing - R01542500

Bengaluru, Karnataka, India

Experience: Not specified

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start Job-Specific Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.