Research:Onboarding new Wikipedians/OB1

This page in a nutshell: Shortly after launching the onboarding experiment, we collected data for 10 days, comparing the activity of users who chose 'getting started' tasks with those who completed edits after returning to their internal referrer (pre-registration). We found a significant 6.2% difference in the rate at which gettingstarted users completed a main namespace edit in 24 hours, compared to those who directly returned to an editable article. We did not see a statistically significant difference in the rate at which any cohort reached 5 edits in 24 hours.

Overview

Screenshot

This iteration started on 2013-01-15 and ended on 2013-01-25T00:00:00Z and logged the editing activity of users entering one of the two following subfunnels from the GettingStarted landing page:

returnto: users clicking on the "Return To" button and attempting to edit the page they came from when they registered an account
gettingstarted: users clicking on one or more links in the task section of the GettingStarted page

Funnel analysis

Cohort analysis

We defined the following cohorts in order to measure their respective productivity:

Cohort definition

OB1 cohorts
ID	Unique users	Description	% of group
e3_ob1	35,526	Users being served tasks via GettingStarted	100%
e3_ob1_noclick	29,624	GettingStarted users neither clicking on a gettingstarted link nor the returnto button	83.4%
e3_ob1_returnto_click	4,712	GettingStarted users clicking on the returnto link	13.3%
e3_ob1_gettingstarted_click	1,316	GettingStarted users clicking on one or more gettingstarted links	3.7%
e3_ob1_returnto_page-impression	5,904	GettingStarted users landing on an editable page in the returnTo funnel	16.6%
e3_ob1_gettingstarted_page-impression	3,757	GettingStarted users landing on a page in the gettingstarted funnel	10.6%

Notes:

The overlap between returnto and gettingstarted clickers is very small (N=126) and so is the overlap between users landing on an editable page via the returnto funnel and the gettingstarted funnel (N=40): the two groups are therefore clearly disjoint. The discrepancy between clicks and page-impressions is likely due to users using right-clicks to open multiple tabs, which we're currently not logging due to limitations of EventLogging.
Editable pages are those flagged with isEditable=true in Schema:GettingStarted.

1 edit in 24 hours threshold

We measured the proportion of users in each cohort completing their first main-namespace edit within 24 hours since registration.

OB1 1+ ns0-edit 24h threshold
ID	Unique users	Editing users	%
e3_ob1	35,526	8,124	23.1%
e3_ob1_noclick	29,624	6,848	23.1%
e3_ob1_returnto_click	4,712	1,041	22.1%
e3_ob1_gettingstarted_click	1,316	325	24.7%
e3_ob1_returnto_page-impression	5,904	1,173	19.9%
e3_ob1_gettingstarted_page-impression	3,757	980	26.1%

We compared the proportion of threshold-hitting users in the e3_ob1_returnto_page-impression and e3_ob1_gettingstarted_page-impression cohorts and found a significant 6.2% difference between these cohorts (a chi-square test rejects the null hypothesis that the true proportion of threshold-hitting accounts is the same in both groups [X² = 50.87, N = 9,661, p < .001]). The difference in the proportion of threshold-hitters between the e3_ob1_gettingstarted_page-impression and the e3_ob1_noclick cohorts (3.0%) is also significant [X² = 16.19, N = 33,381, p < .001]).

5 edits in 24 hours threshold

We measured the proportion of users in each cohort completing their first 5 main-namespace edit within 24 hours since registration.

OB1 5+ ns0-edit 24h threshold
ID	Unique users	Editing users	%	% from 1+ editing users
e3_ob1	35,526	1,157	3.2%	14.24%
e3_ob1_noclick	29,624	972	3.3%	14.19%
e3_ob1_returnto_click	4,712	145	3.1%	13.93%
e3_ob1_gettingstarted_click	1,316	40	3.0%	12.31%
e3_ob1_returnto_page-impression	5,904	180	3.0%	15.35%
e3_ob1_gettingstarted_page-impression	3,757	119	3.2%	12.14%

The proportion of threshold-hitters varies insignificantly between 3.0%-3.3% across the cohorts.