Job
Description
Job Title: Data Engineer (Spark/Scala/Cloudera) Location: Chennai/Pune Job Type : Full time Experience Level: 6- 9 years Job Summary: We are seeking a skilled and motivated Data Engineer to join our data engineering team. The ideal candidate will have deep experience with Apache Spark, Scala, and Cloudera Hadoop ecosystem. You will be responsible for building scalable data pipelines, optimizing data processing workflows, and ensuring the reliability and performance of our big data platform. Key Responsibilities: Design, build, and maintain scalable and efficient ETL/ELT pipelines using Spark and Scala. Work with large-scale datasets on the Cloudera Data Platform (CDP). Collaborate with data scientists, analysts, and other stakeholders to ensure data availability and quality. Optimize Spark jobs for performance and resource utilization. Implement and maintain data governance, security, and compliance standards. Monitor and troubleshoot data pipeline failures and ensure high data reliability. Participate in code reviews, testing, and deployment activities. Document architecture, processes, and best practices. Required Skills and Qualifications: Bachelor's or Master's degree in Computer Science, Engineering, or related field. 6+ years of experience in big data engineering roles. 2 + Years of Hands on experience into Scala Proficient in Apache Spark (Core/DataFrame/SQL/RDD APIs). Strong programming skills in Scala. Hands-on experience with the Cloudera Hadoop ecosystem (e.g., HDFS, Hive, Impala, HBase, Oozie). Familiarity with distributed computing and data partitioning concepts. Strong understanding of data structures, algorithms, and software engineering principles. Experience with CI/CD pipelines and version control systems (e.g., Git). Familiarity with cloud platforms (AWS, Azure, or GCP) is a plus. Preferred Qualifications: Experience with Cloudera Manager and Cloudera Navigator. Exposure to Kafka, NiFi, or Airflow. Familiarity with data lake, data warehouse, and lakehouse architectures. Preferred candidate profile
