Senior Data Engineer

At The Institute of Clever Stuff (ICS), we don’t just solve problems—we revolutionise results. Our mission is to empower a new generation of Future Makers today, to revolutionise results and create a better tomorrow. Our vision is to pioneer a better future together. We are a consulting firm with a difference, powered by AI, driving world-leading results from data and change.

We partner with visionary organisations to solve their toughest challenges, drive transformation, and deliver high-impact results. We combine a diverse network of data professionals, designers, software developers, and rebel consultants alongside our virtual AI consultant, fortu.ai, who combine human ingenuity with fortu.ai’s AI-powered intelligence to deliver smarter, faster and more effective results.

Meet fortu.ai

• Used by some of the world’s leading organisations as a business question pipeline generator, ROI tracker, and innovation engine all in one.
• Trained on 400+ accelerators and 8 years of solving complex problems with global organisations.
• With fortu.ai, we’re disrupting a $300+ billion industry, turning traditional consulting on its head.

Context of work:

• The client is a global energy company undergoing a significant transformation to support the energy transition. We work within their Customers & Products (C&P) division, serving both B2C and B2B customers across key markets such as the UK, US, Germany, Spain, and Poland.
• This business unit includes mobility (fuel and EV), convenience retail, and loyalty.

Scope of the work:

Client project to deliver:

• Data Pipeline Development: Building new pipelines for data models using AWS Glue and PySpark. Leading on end-to-end data pipeline creation and execution
• Data Pipeline Management: Deploying new features into core data models that require re-deployment of the pipeline through staging environments (dev, pre-prod, prod). Supporting regular refreshes of the data.
• Data Model Performance: Leading on finding opportunities to optimise and automate data ingestion, data refreshes, and data validation steps for the data models.
• Data Modelling: Supporting the team in building new data models and solutions, working closely with data scientists.
• Data Quality Assurance: Establish processes to monitor data pipelines for data loss, corruption, or duplication and take corrective action.

Requirements:

• Capable and confident in data engineering concepts: designing data models, building data warehouses, automating data pipelines, and managing large datasets.
• Strong background in data modelling, creating relational data models, data warehousing and ETL processes.
• The ability to design, build and manage efficient and reliable data pipelines.
• Strong coding best practices, including version control.
• Experience working in agile sprint-based delivery environments.
• Experience working with customer and transactional data.
• Experience collaborating with a mixed team of permanent client colleagues and other partners and vendors – working with: Data Scientists, Data Engineers, Data Analysts, Software Engineers, UI/UX Designers and internal Subject Matter Experts.
• Experience delivering to a large enterprise of stakeholders.

Core Technologies: SQL, Python, PySpark/Spark SQL, AWS (Redshift, Athena, Glue, Lambda, RDS), AWS Serverless Data Lake Framework (SDLF), SQL client software (e.g. Dbeaver), Bazel (automated testing), Git.

Nice-to-have Technologies: Databricks, Amazon SageMaker, Jupyter Notebook, MLOps, ML model development, and ML engineering would be advantageous.