Expert Data Engineer

Position Overview:

We are seeking a highly skilled Data Engineer to join the Cloud Data Hub (CDH) Team, collaborating closely with teams in Munich and BMW TechWorks India. The ideal candidate will have expertise in Python, AWS, Terraform, and Git, with a passion for building scalable, reliable, and efficient data pipelines and solutions. As a Data Engineer, you will play a crucial role in designing, developing, and optimizing the data infrastructure that underpins BMWs transformation into a data-driven organization.

About the project

The Cloud Data Hub (CDH) is a cloud-based, centralized data lake developed by BMW, serving as the organization's central data landing zone. Designed to democratize data usage across all departments, the CDH consolidates data into a single source of truth, enabling providing and consuming entities to leverage data effectively and efficiently. It plays a pivotal role in BMW's transformation into a truly data-driven organization, supporting data acquisition, integration, processing, and analysis across its value chain.

Key Responsibilities

• Design and implement scalable, reliable, and efficient data pipelines and ETL/ELT processes using Python and AWS.
• Build and maintain cloud-native data infrastructure on AWS, leveraging services such as Lambda, S3, and Glue.
• Develop and manage infrastructure as code (IaC) using Terraform to ensure repeatable, automated deployment of cloud resources.
• Utilize Git for version control and collaborate with the team through code reviews and CI/CD pipelines.
• Apply Test-Driven Development (TDD) principles to ensure robust, maintainable, and high-quality data pipelines.
• Monitor and troubleshoot data pipelines and infrastructure, ensuring high availability and reliability.
• Ensure compliance with data governance, security policies, and BMW’s best practices.
• Document technical processes, data models, and architecture to ensure transparency and ease of maintenance.

Qualifications

• Expert-level proficiency in building and managing data pipelines with Python.
• Strong experience in AWS cloud services, including Lambda, S3, Glue, and other data-focused services.
• Advanced skills in Terraform for provisioning and managing infrastructure-as-code on AWS.
• Proficiency in SQL for querying and modeling large-scale datasets.
• Hands-on experience with Git for version control and managing collaborative workflows.
• Familiarity with ETL/ELT processes and tools for data transformation.
• Strong understanding of data architecture, data modeling, and data lifecycle management.
• Excellent problem-solving and debugging skills.
• Strong communication and collaboration skills to work effectively in a global, distributed team environment