Senior Principal Data Engineer & Architect

Role Summary:

We are looking for a Senior Principal Data Engineer & Architect a hands-on technical leader capable of designing, building, and optimising the core customer data platform across ingestion, modelling, automation, and ML operations.

This role is central to the mission of the team.

You will act as the technical backbone, ensuring high data reliability, architectural coherence, and scalable ML delivery across 12 global markets. You will lead the consolidation of fragmented pipelines, define the Gold Layer, modernise legacy codebases, automate monitoring, and build production-grade data and ML systems that our analysts and regional teams can trust.

This is a unicorn” role that spans data engineering, data architecture, MLOps, automation, and applied data science — ideal for someone who thrives at the intersection of engineering excellence and analytical innovation.

Key Responsibilities

1. Platform Engineering & Automation

• Modernise and refactor legacy pipelines (AWS Glue, Python, PySpark, Databricks) into modular, automated, and high-performance jobs.
• Build zero-touch automated monitoring and alerting systems (Great Expectations, Soda.io, or custom frameworks).
• Implement CI/CD pipelines (GitHub Actions/Terraform) ensuring production-safe deployments.
• Introduce serverless and cost-optimised patterns to reduce operating expenses.

2. Data Architecture & Modelling

• Own the design of the Customer Gold Layer — the single source of truth for customer, loyalty, and marketing data.
• Lead identity resolution logic (tokens, loyalty IDs, cards).
• Define data, table schemas, and implement standardised business logic (e.g., margin, loyalty KPIs).
• Migrate legacy SQL/SP logic into scalable, modular dbt projects.
• Ensure architectural consistency across all global markets.

3. Data Quality & Observability

• Build automated data quality tests, anomaly detection, and lineage tracking.
• Reduce the “risk surface” by enforcing PII protection, schema governance, and controlled access layers.

4. MLOps & Applied Machine Learning

• Productionise ML models using MLflow/Databricks.
• Build feature stores, model registries, scoring pipelines, and scalable batch/real time serving.
• Work with data scientists to operationalise experimentation and model refinement.
• Ensure reproducibility, monitoring of model drift, and robust promotion-to production processes.

5. Cross-Functional Technical Leadership

• Serve as the technical authority across CDI pods (Stability, Engineering, Reporting & DS).
• Partner with regional data teams to harmonise data definitions and architectural patterns.
• Guide Mumbai Hub analysts by providing technical guardrails, certified data, and reusable assets.
• Review code, coach engineers, and raise delivery standards across Technology.
• Act as the escalation point for complex data/platform incidents.

6. Strategic Contribution

• Shape the long-term data architecture roadmap for customer data at bp.
• Identify automation and consolidation opportunities to reduce operating cost and manual effort.
• Support large-scale transformations, including SCI proliferation, margin unification, and platform modernisation.
• Ensure global alignment with Technology standards and security principles.

Required Skills & Experience

Engineering Excellence

• Expert-level Python and SQL across both analytics and production engineering contexts.
• Deep experience with AWS services (Glue, Lambda, Step Functions, S3, Athena) and/or Databricks/PySpark.

Architecture & Modelling

• Proven experience designing complex data models (dimensional modelling, entity resolution, canonical layers).
• Strong ability to rationalise dozens of pipelines into a unified, governed architecture.

Automation & Infrastructure

• Hands-on experience with Terraform, Azure Dev Ops, CI/CD, containerisation (Docker).
• Experience implementing monitoring/observability frameworks for data pipelines.

MLOps & Data Science

• Competency in the full ML lifecycle: feature engineering, model training, registry management, deployment, and monitoring.
• Experience with MLflow, Databricks ML, or equivalent.

Data Quality & Governance

• Experience implementing automated DQ tests, schema validation, and data contracts.
• Strong understanding of PII handling, security boundaries, and compliance.

Leadership & Collaboration

• Ability to work across organisational boundaries and influence senior stakeholders.
• Comfortable mentoring engineers and partnering with analysts and data scientists.
• Strong communication skills with the ability to translate complexity into clear actions.

Ideal Profile

• Someone who is hands-on, pragmatic, and comfortable owning complex technical decisions end to end.
• A builder who enjoys modernising messy legacy systems.
• A systems thinker who designs for scale across 12+ markets.
• A collaborative leader who lifts the engineering maturity of everyone around them.
• Comfortable being the “go-to” person for the hardest technical problems.