Lead - Digital Experimentation

brilliant opportunity

demonstrated technical and commercial expertise

While the emphasis of this role is on the enablement of our data, product, marketing and technology teams to run their own experiments, this person would also often be responsible for designing, executing, and analyzing more complex and ambitious AB/multivariate tests across digital platforms to identify optimal user experiences by comparing different variations of a webpage, app feature, or marketing campaign, ultimately providing data-driven insights to improve key business metrics and inform product development decisions.

Role & responsibilities

Key responsibilities:

• Manage the Amplitude experimentation platform to enable other users e.g. setting up standard reporting templates, metrics, audiences, etc. for use by other analysts and engineers
• Identify training and skill development needs amongst Amplitude user community and, based on these needs, organise and/or run training, set up and run user groups, organise best practice clinics and drop-in sessions, etc.
• Ensure that innovative experimentation methodologies and best practices that would add value to The Economist Group are identified, tested and the outcomes widely shared across the business
• Ensure that the results of experimentation portfolio are well curated and widely shared across the business
• Alongside being hands-on designing and evaluating our more complex experiments, act as an advisor and consultant to the wider analyst and stakeholder community in ensuring that all our experiments are well-design, well-executed and rigorously evaluated
• Building excitement and enthusiasm about the capabilities we have and about the power and potential of experimentation
• Building deep technical expertise in our experimentation platforms [particularly Amplitude Experiment]

Preferred candidate profile

Experimentation Design & Execution

• Strong ability to design, scope, and implement controlled experiments (A/B, multivariate, Feature Flag experiments [holdout tests]).
• Expertise in defining hypotheses, test metrics, and success criteria aligned with business objectives by choosing the right statistical test for the experiment, knowledge of methodologies for identifying issues or bias when analysing experiment results.
• Knowledge of randomization techniques, sampling strategies, and statistical significance and the ability to explain these concepts to stakeholders in simpler terms.
• Experience with setting up and monitoring experiment guardrails (e.g., avoiding revenue cannibalization or bias).

Technical & Analytical Proficiency

• Hands-on experience with experimentation platforms (e.g., Amplitude, Optimizely, Adobe Target, Google Optimize, VWO or any such similar platforms).
• Familiarity with statistical methods (t-tests, Bayesian inference [multi-arm bandits etc], regression, uplift modeling, sequential testing).
• Strong SQL skills for querying large datasets and joining multiple sources.
• Proficiency in Python or R for deeper statistical analysis and automation of experimentation workflows.
• Ability to build dashboards and visualization of experiment results (within Amplitude and/or Tableau).

Optimization & Scaling

• Experience running simultaneous experiments across multiple user journeys, platforms, and geographies.
• Ability to prioritize experiments based on business impact and resource availability.
• Comfort with test orchestration in environments with feature flags and continuous deployment.
• Strong understanding of personalization and advanced targeting strategies.

Data Quality & Implementation

• Hands-on ability to validate experiment instrumentation, ensuring events and conversions are tracked correctly.
• Knowledge of web and mobile tagging frameworks (e.g. Tealium, GTM, Segment) for clean experiment setup.
• Familiarity with debugging tools (e.g., Chrome DevTools, Charles Proxy, Firebase DebugView) to troubleshoot test setups.

Collaboration & Communication

• Skilled in translating complex statistical findings into business-friendly insights.
• Ability to coach cross-functional teams (product managers, marketers, engineers, analysts) on best practices in experimentation.
• Hands-on experience creating playbooks, templates, and knowledge-sharing resources for scaling experimentation culture.

Key Stakeholders

• Product teams (web and apps)
• Marketing Teams on conversion rate optimisation
• Customer Experience and In Life (Retention) marketing
• Data Research and Insights (DRI) Team
• Technology Teams (Front end teams, Data engineers etc)