Big Data Engineer - Hadoop/Spark

Role Overview

As a Big Data Engineer, you will be responsible for building and maintaining scalable data pipelines and architectures that support data analytics, reporting, and real-time processing. You will collaborate with data scientists, business analysts, and software engineers to design robust solutions that drive data-driven decisions.

Mandatory Skill Set

• Extensive hands-on experience with the Hadoop ecosystem (HDFS, MapReduce, YARN).
• Strong expertise in Apache Spark for large-scale data processing and analytics.
• Proficiency in Hive writing complex HiveQL queries and managing Hive Metastore.
• Experience in developing and maintaining Apache Kafka-based streaming solutions.
• In-depth knowledge of HDFS storage management and optimization.
• Proven programming skills in Python or Scala for data pipeline development and automation.
• Strong expertise in writing complex SQL queries for data transformation and retrieval.
• Experience in workflow orchestration using Apache Airflow or Oozie to manage dependencies, schedule jobs, and monitor workflows.
  

Key Expectations

• Proven experience implementing Big Data solutions in production environments theoretical knowledge is not sufficient.
• Ability to design and develop reliable, high-performance data pipelines to process large volumes of data.
• Strong debugging, optimization, and troubleshooting skills for distributed data applications.
• Good understanding of data modeling, schema design, and ETL best practices.
• Collaborative mindset to work effectively with cross-functional teams and deliver high-quality solutions.
  

Important Note

We require candidates who have demonstrable, hands-on project experience in Big Data Engineering knowledge alone is not acceptable. Candidates with only academic or certification-level exposure will not be considered.

What We Offer

• Opportunity to work on advanced Big Data technologies and real-time data processing solutions.
• Exposure to challenging data engineering projects across different domains.
• Collaborative and innovative work environment focused on learning and growth.
• Qualified and experienced candidates are encouraged to apply through the platform.
• Take this opportunity to advance your career by contributing to cutting-edge Big Data projects.
  

(ref:hirist.tech)