BigData and Hadoop Ecosystems

BigData and Hadoop Ecosystems Engineer

Roles and Responsibilities:

• Big Data Platform Management:

  Install, configure, and maintain components of the

  Hadoop ecosystem

  (e.g., HDFS, YARN, Hive, Spark, Kafka, HBase) to ensure optimal performance, scalability, and high availability.

• Data Pipeline Development:

  Design, develop, and implement robust and efficient data pipelines for ingestion, processing, and transformation of large datasets using tools like

  Apache Spark, Hive, or Kafka

  .

• Performance Tuning:

  Monitor the performance of big data clusters and applications. Identify bottlenecks and implement optimization strategies for Spark jobs, Hive queries, and other big data processes.

• Data Lake/Warehouse Design:

  Contribute to the design and implementation of data lake and data warehouse solutions leveraging Hadoop-based technologies.

• ETL/ELT Processes:

  Develop and manage complex ETL/ELT processes to integrate data from various sources into the big data ecosystem.

• Troubleshooting:

  Perform in-depth troubleshooting, debugging, and resolution for complex issues within the Hadoop ecosystem, including cluster stability, data processing failures, and performance degradation.

• Security & Governance:

  Implement and maintain security best practices for big data platforms, including access control, encryption, and data governance policies.

• Automation:

  Develop scripts and automation routines for cluster management, deployment, monitoring, and routine operational tasks within the big data environment.

• Collaboration:

  Work closely with data scientists, data analysts, application developers, and infrastructure teams to support data-driven initiatives.

Preferred Candidate Profile:

• Hadoop Ecosystem Expertise:

  Strong hands-on experience with core components of the

  Hadoop ecosystem

  (HDFS, YARN) and related technologies like

  Apache Spark, Hive, Kafka, HBase, or Presto

  .

• Programming/Scripting:

  Proficient in programming languages commonly used in big data, such as

  Python, Scala, or Java

  . Strong scripting skills for automation.

• SQL Proficiency:

  Excellent proficiency in

  SQL

  for data manipulation and querying in big data environments (e.g., HiveQL, Spark SQL).

• Cloud Big Data (Plus):

  Familiarity with cloud-based big data services (e.g., AWS EMR, Azure HDInsight, Google Cloud Dataproc) is a plus.

• Distributed Systems:

  Understanding of distributed computing principles and challenges in managing large-scale data systems.

• Problem-Solving:

  Excellent analytical and problem-solving skills with a methodical approach to complex big data challenges.

• Communication:

  Strong verbal and written communication skills to articulate technical concepts and collaborate effectively with diverse teams.

• Education:

  Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related technical field.