Data Engineer- Spark with Scala- Immediate joiners

Job Description

Job Description: Big Data Engineer- Immediate joiners only Location: Bangalore Experience Level:6+ years We are seeking a highly skilled Big Data Engineer with expertise in distributed systems, the Hadoop ecosystem, and advanced programming techniques. The ideal candidate will possess strong technical knowledge, thought leadership, and a passion for creating scalable, high-performance systems. Key Responsibilities: Design, develop, and optimize distributed systems and Big Data solutions. Implement and maintain batch and streaming pipelines using Scala and Spark. Leverage experience in the Hadoop ecosystem, including technologies like Hive, Oozie, and Kafka. Build and maintain CI/CD pipelines to ensure efficient code deployment and integration. Apply design patterns, optimization techniques, and locking principles to enhance system performance and reliability. Scale systems and optimize performance through effective caching mechanisms. Demonstrate strong computer science fundamentals, logical reasoning, and problem-solving abilities. Collaborate with cross-functional teams to influence and drive innovative solutions. Provide technical leadership, mentorship, and presentations to guide team members and stakeholders. Required Qualifications: Proven experience working with distributed systems and Big Data technologies. Proficiency in Scala, Spark, and related tools. In-depth understanding of design patterns and system optimisation principles. Hands-on experience with batch and streaming data pipelines. Familiarity with CI/CD pipelines and modern software development practices. Strong understanding of system scaling and caching techniques. Excellent communication and presentation skills, with the ability to inspire and influence others. Preferred Qualifications: Scala, spark(preferably scala should be an experienced spark exposure Good understanding of Spark Architecture including Spark Core, Spark SQL, data Frames, Spark Streaming, Driver Node, Worker Node, stages, Executors and Tasks. Should have experienced with Hadoop ecosystem-HDFS, hive (must), Oozie, sqoop etc. Should have prior experience working with big data ingestion from Various sources handling TB of data. Excellent Knowledge of SQL and coding skills. Experience with scripting languages such as Shell. Effective working in a team as well as working independently. Experience of working in Agile delivery. We would love to hear from you if you are passionate about working on complex systems and leading the charge in cutting-edge Big Data solutions.