Big Data Engineer

As a Big Data Engineer focusing on the Hadoop ecosystem in an IT services and enterprise data engineering sector, your primary role will involve designing, deploying, and maintaining Hadoop ecosystem clusters (HDFS, YARN) to ensure stability, scalability, and security for production workloads. You will be responsible for developing and optimizing batch ETL pipelines using tools like Spark, Hive, and Sqoop to ingest, transform, and store large datasets. Additionally, building and operating streaming data pipelines using Kafka to ensure low-latency delivery and fault tolerance will be a key part of your responsibilities. Performance tuning, capacity planning, and cluster troubleshooting will also fall under your purview, including resource contention, job optimization, and storage format tuning (Parquet/ORC/Avro). Implementing data reliability, lineage, and governance best practices, as well as automating routine operations with Oozie/Airflow and scripting, will be essential. Collaboration with Data Engineering, Data Science, and DevOps teams to productionize models, integrate CI/CD, and enable observability and alerting will also be a key aspect of your role. Must-Have Skills & Qualifications: - Apache Hadoop - HDFS - YARN - Apache Spark - Apache Hive - Apache Kafka - Sqoop - Oozie Preferred Skills: - AWS EMR or cloud-managed Hadoop services - Apache HBase - Apache Airflow To qualify for this role, you must hold a Bachelor's degree in Computer Science, IT, or have equivalent practical experience, with a demonstrated track record of delivering production Big Data solutions in Hadoop ecosystems. In this on-site role based in India, you will have the opportunity for hands-on ownership of large-scale data platforms and work across analytics and ML data pipelines. You will gain exposure to modern data engineering practices, performance tuning, and platform automation in a collaborative engineering culture that emphasizes operational excellence and career growth. (Note: Benefits & Culture Highlights section is omitted as it is not present in the provided job description) As a Big Data Engineer focusing on the Hadoop ecosystem in an IT services and enterprise data engineering sector, your primary role will involve designing, deploying, and maintaining Hadoop ecosystem clusters (HDFS, YARN) to ensure stability, scalability, and security for production workloads. You will be responsible for developing and optimizing batch ETL pipelines using tools like Spark, Hive, and Sqoop to ingest, transform, and store large datasets. Additionally, building and operating streaming data pipelines using Kafka to ensure low-latency delivery and fault tolerance will be a key part of your responsibilities. Performance tuning, capacity planning, and cluster troubleshooting will also fall under your purview, including resource contention, job optimization, and storage format tuning (Parquet/ORC/Avro). Implementing data reliability, lineage, and governance best practices, as well as automating routine operations with Oozie/Airflow and scripting, will be essential. Collaboration with Data Engineering, Data Science, and DevOps teams to productionize models, integrate CI/CD, and enable observability and alerting will also be a key aspect of your role. Must-Have Skills & Qualifications: - Apache Hadoop - HDFS - YARN - Apache Spark - Apache Hive - Apache Kafka - Sqoop - Oozie Preferred Skills: - AWS EMR or cloud-managed Hadoop services - Apache HBase - Apache Airflow To qualify for this role, you must hold a Bachelor's degree in Computer Science, IT, or have equivalent practical experience, with a demonstrated track record of delivering production Big Data solutions in Hadoop ecosystems. In this on-site role based in India, you will have the opportunity for hands-on ownership of large-scale data platforms and work across analytics and ML data pipelines. You will gain exposure to modern data engineering practices, performance tuning, and platform automation in a collaborative engineering culture that emphasizes operational excellence and career growth. (Note: Benefits & Culture Highlights section is omitted as it is not present in the provided job description)