Research:Privacy-conscious AB testing at Wikimedia Foundation

Duration:  2016-10 – 2017-03
Web, A/B testing, Privacy
VisualEditor - Icon - Check.svg
This page documents a completed research project.

Every time we need to segment wikimedia project users we end up having to write code on varnish and discuss whether we are segmenting per IP, per device or per “user”. When we want to launch a feature that might be disruptive, we normally follow the path of enabling it on a small wiki and extrapolating those results to a bigger wiki, which does not always work.

The goal of this document is to explore the design of a basic A/B testing framework with strong privacy constrain such we do not end up having to modify varnish code every time we want to launch a specific feature. Our initial objective is to provide a simple framework to obtain data statistically valid in the absence of logged in users or sessions. Thus, in the initial design we are only considering readers using the Wikimedia sites anonymously. We do not consider editors.


Design Document to be delivered by March 2017

A/B testing design document


  1. Designing and Deploying Online Field Experiments, 2014:
  2. PlanOut:
  3. Overlapping Experiment Infrastructure: More, Better, Faster Experimentation:
  4. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web: