Research:Notifications/Experiment 2

Goals

1. To understand the effect of revert notifications on the behavior of new editors on Wikipedia.
2. To vet the decision to turn on or off these notifications by default.

Overview

This experiment aims to test the effects of revert notifications. During the experimental period, we'll randomly assign newly registered users into two "buckets" corresponding to the two conditions (see Conditions below).

We'll use behavioral measurements to determine which users were more active (survival rate, edits, labor hours), productive (revert rate), communicative (talk & user talk edits) and burdensome (revert rate, block rate). We'll also use the preference change log to track users' preference changes within each condition.

Conditions

1. Echo-Current: Current defaults (control condition)
2. Echo+Reverts: Current defaults + revert notification (web and email)

Experimental period and sample size

Duration: 1 week. Assuming 4K new registrations per day, and a 25% rate of 24 hours new editors in the main namespace, we should have 14K total users per condition and about 3.5K users with a mainspace edit in 24h per condition.

Implementation

Users will be assigned to one of the two conditions using deterministic bucketing against their user ID. Users in Echo-Current will have default settings. For users in Echo+Reverts, notifications of type reverted will be enabled (both for web and email). At the end of the test bucketing will be removed, and new users will get the default settings. Users in the Echo+Reverts bucket will keep receiving revert notifications unless they opt out.

Hypotheses

The presence of revert notification will cause newcomers to have more negative interactions with Wikipedians

1. This will reduce newcomer retention.
2. This will increase Wikipedian work load.