Scala Data Engr/Sr Data Pipeline Python, Spark Streaming German MNC

Job Description

Full time with top German MNC for location Bangalore - Experience on SCALA is a must Job Overview: To work on development, monitoring and maintenance of Data pipelines across clusters. Primary responsibilities: Develop, Monitor and Maintain data pipeline for various plants. Create and maintain optimal data pipeline architecture. Assemble large, complex data sets that meet functional / non-functional business requirements. Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability. Work with stakeholders including the Data officers and stewards to assist with data-related technical issues and support their data infrastructure needs. Work on incidents highlighted by the data officers. Incident diagnosis, routing, evaluation & resolution. Analyze the root cause of incidents. Create incident closure report. Qualifications Qualifications Bachelors degree in Computer Science, Electronics & Communication Engineering, a related technical field, or equivalent practical experience. 6-8 years of experience in Spark, Scala software development. Experience in large-scale software development. Excellent software engineering skills (i.e., data structures, algorithms, software design). Excellent problem-solving, investigative, and troubleshooting skills. Experience in Kafka is mandatory Additional Information Skills Self-starter and empowered professional with strong execution and project management capabilities Ability to collaborate effectively, well developed inter personal relationships with all levels in the organization and outside contacts. Outstanding written and verbal communication skills. High Collaboration & a perseverance to drive performance & change Additional information Key Competencies- Distributed computing systems Experience with CI/CD tools such as Jenkins or Github Actions Experience with Python programming Working knowledge of Docker & Kubernetes Experience in developing data pipelines using spark & scala. Experience in debugging pipeline issues. Experience in writing python and shell scripts. In-Depth Knowledge of SQL and Other Database Solutions Having a strong understanding of Apache Hadoop-based analytics Hands on experience on InteliJ, Github /Bitbucket, HUE.