Pyspark Architect

• Enterprise Data Hub EDH is based on the Cloudera Data Platform CDP
• As an EDH Data Engineer you contribute to the ingestion of different data sources originating from our IT application landscape and networks
• You will work within an agile team squad following agile practices and interact with IT and business colleagues architects functional analysts project managers data scientists to contribute to the roadmap of our Enterprise Data Hub
• The position of Data Engineer is crucial in our Data tribe
• You are part of the Data Engineering Chapter which is responsible for ensuring efficient and qualitative ways of working for data ingestion
• Together with your colleagues you define and implement these processes and adapt them to remain up to date with the technological evolutions in our data landscape
• As a Data Engineer you grow continuously as you are in contact with several colleagues and stakeholders in and outside our Data tribe building a network learning new technologies and growing your interpersonal competences
• From pure developing over analysing and modelling towards solution design or a combination of these Data Engineers can evolve in the roles they are taking over time keeping it a highly motivating environment
• Artificial Intelligence is making its entrance in the activities of a Data Engineer
• You will be able to use prompting and contribute to the introduction of AI agents in our engineering process

Key Responsibilities:

• Build robust performing and high scalable flexible data pipelines with a focus on time to market with quality
• Responsibilities
• Act as an active team member to ensure high code quality unit testing regression tests delivered in time and within budget
• Document the delivered code solution
• Participate to the implementation of the releases following the change release management processes
• Provide support to the operation team in case of major incidents for which engineering knowledge is required
• Participate to effort estimations
• Provide solutions bug fixes for problem mgt

Technical Requirements:

• You have experience with most of these technologies Pyspark AWS Databricks Spark HDFS Python

Additional Responsibilities:

• Good knowledge on software configuration management systems
• Strong business acumen strategy and cross industry thought leadership
• Awareness of latest technologies and Industry trends
• Logical thinking and problem solving skills along with an ability to collaborate
• Two or three industry domain knowledge
• Understanding of the financial processes for various types of projects and the various pricing models available
• Client Interfacing skills
• Knowledge of SDLC and agile methodologies
• Project and Team management

Preferred Skills: