These requirements apply to PEF as well as any other experiment with (1) an eligibility check based on registration date (for example experiments only targeting new editors or editors with accounts older than N days) and (2) a limited and consistent duration of the treatment for each participant in the experiment.


  start of the experiment (deployment completed)
  activation buffer period
  beginning of eligibility period
  end of eligibility period
  eligibility period
  registration time
  duration of treatment
  treatment period
  end of treatment
  end of the experiment
experiment the duration of the experiment for the experimenter, spanning the whole process of deploying the code to production, collecting data and disabling the code
eligibility the set of conditions that determine the population of participants, i.e. whether a user can participate at all in the experiment
bucket the actual condition to which a participant is assigned to if in the eligible population (i.e. an experimental treatment, a control group or nothing if groups are a sample of the eligible population)
treatment the actual experimental condition any individual participant is exposed to during the test

Experiment activation workflow

  • the code gets deployed at  , we allow a buffer of time   before the experiment is activated and anyone can participate in the experiment;
  •   is the period that determines if a user is eligible as a participant in the experiment. If the user registration date falls within this period the user is eligible, otherwise nothing happens
  • if the user is eligible, than a bucketing function determines which condition the user is assigned to. Conditions can be two (in case of a split test) or multiple (in case of multiple treatments).
  • The duration of the treatment   is calculated from the time at which the user becomes eligible as a participant (the registration date) and ensures that every participant is exposed to the treatment for the same duration regardless of when she became eligible. This will ensure, in particular, that users who registered towards the end of the eligibility period will still get exposed to a sufficiently long treatment to produce usable data.
  • The overall duration of the experiment   will need to extend at least   days after the eligibility period is over.