Big Data Engineer (Spark & Scala)

Role Overview: As a Big Data Engineer specializing in Spark & Scala, you will be responsible for developing and optimizing big data processing pipelines using Apache Spark and Scala. Your role will involve designing and implementing ETL workflows for large-scale batch and real-time data processing. Additionally, you will be expected to optimize Spark performance through partitioning, caching, and memory/shuffle tuning. Collaboration with cross-functional teams and adherence to best practices in coding, testing, and deployment are essential aspects of this role. Key Responsibilities: - Develop and optimize big data processing pipelines using Apache Spark (Core, SQL, Streaming) and Scala. - Design and implement ETL workflows for large-scale batch and real-time data processing. - Optimize Spark performance through partitioning, caching, and memory/shuffle tuning. - Work with big data ecosystems like Hadoop, Hive, HDFS, and cloud platforms (AWS/Azure/GCP). - Collaborate with cross-functional teams and follow best practices in coding, testing, and deployment. Qualifications Required: - Bachelor's degree in a relevant field. - 6 to 10 years of experience in a similar role. - Strong expertise in Apache Spark and Scala. - Familiarity with big data ecosystems like Hadoop, Hive, HDFS, and cloud platforms. - Self-confidence and patience. (Note: The additional details of the company were not provided in the job description.),