Growth/Activation

These requirements apply to PEF as well as any other experiment with (1) an eligibility check based on registration date (for example experiments only targeting new editors or editors with accounts older than N days) and (2) a limited and consistent duration of the treatment for each participant in the experiment.

Definitions

Milestones

$t_{0}$	start of the experiment (deployment completed)
$[t_{0},t_{1}]$	activation buffer period
$t_{1}$	beginning of eligibility period
$t_{2}$	end of eligibility period
$[t_{1},t_{2}]$	eligibility period
$t_{R}$	registration time
$N$	duration of treatment
$[t_{R},t_{R}+N]$	treatment period
$t_{R}+N$	end of treatment
$t_{3}=t_{2}+N$	end of the experiment

Terminology

experiment	the duration of the experiment for the experimenter, spanning the whole process of deploying the code to production, collecting data and disabling the code
eligibility	the set of conditions that determine the population of participants, i.e. whether a user can participate at all in the experiment
bucket	the actual condition to which a participant is assigned to if in the eligible population (i.e. an experimental treatment, a control group or nothing if groups are a sample of the eligible population)
treatment	the actual experimental condition any individual participant is exposed to during the test

Experiment activation workflow

• the code gets deployed at $t_{0}$, we allow a buffer of time $[t_{0},t_{t1}]$ before the experiment is activated and anyone can participate in the experiment;
• $[t_{1},t_{t2}]$ is the period that determines if a user is eligible as a participant in the experiment. If the user registration date falls within this period the user is eligible, otherwise nothing happens
• if the user is eligible, than a bucketing function determines which condition the user is assigned to. Conditions can be two (in case of a split test) or multiple (in case of multiple treatments).
• The duration of the treatment $N$ is calculated from the time at which the user becomes eligible as a participant (the registration date) and ensures that every participant is exposed to the treatment for the same duration regardless of when she became eligible. This will ensure, in particular, that users who registered towards the end of the eligibility period will still get exposed to a sufficiently long treatment to produce usable data.
• The overall duration of the experiment $[t_{0},t_{3}]$ will need to extend at least $N$ days after the eligibility period is over.