Research:Onboarding new Wikipedians/OB5

This page in a nutshell: We observed that the test cohort is significantly more successful when looking at the 1+ article edits metric, among others, but still did not produce a statistically significant increase in active editors (5+) compared to control. This compares favorably with previous versions of the onboarding experiment series.

In this iteration of onboarding experiments, we will compare a control group to one delivered a new landing GettingStarted page, as well as a new navigation bar delivered to those who accept GettingStarted tasks, which lets them skip to a new random task, go back to the GettingStarted landing page, or generate a short guided tour showing them how to complete the task.

The landing page in this experiment
Task article with navigation bar

Overview

Cohorts

Key events in the gettingstarted funnels, test cohort

Control - delivered no GettingStarted
Test - delivered GettingStarted, which breaks down in the following funnels:
- gettingstarted-copyediting - editors who chose the copyediting task
- gettingstarted-clarify - editors who chose the clarification task
- gettingstarted-addlinks - editors who chose the add wikilinks task
- returnto - editors who opted to click the "No thanks, return to the page I was reading" link to their internal referrer

Metrics

1+ NS0 edits in 24 hrs
5+ NS0 edits in 24 hrs
Revert rate

Research questions

How do the conversion rates of gettingstarted users exposed to this version of the landing page compared to the control group? Specifically...
- Does the navigation bar's functions for returning to Special:GettingStarted or skipping to a new random article enhance conversion rates?
- Does the guided tour generated by "Show help" enhance conversion rates and editor productivity?
- How do these relative conversion rates compare to the results OB2 and OB4?
- How do the conversion rates among task types (copyediting, clarification, add links) compare to each other?
What is the quality of edits made in each task funnel, as measured by revert rate and hand-coding?
Is there a difference (and what kind) in the activity of editors who authenticated an email address and those who did not, in both control and test? Note: Email is confound in this experiment, as it is likely that users who do authenticate an email address have naturally higher levels of engagement, in addition to receiving gettingstarted notifications.

Data

We ran the test for one week and logged activity for users in the two groups via the Schema:GettingStartedNavbar log. We introduced a fix on May 20 due to a bug that affected logging of users clicking on the ReturnTo button in the control condition.

OB5 timeline
Schema revid	start	end	log version	notes
5467309	2013-05-16T21:57:59Z	2013-05-20T23:37:37Z	1	Missing users in the returnto funnel
5496876	2013-05-20T23:38:26Z	2013-05-23T20:31:09Z	2	Fixed logging

Cohort analysis

We defined the following cohorts in order to measure their respective productivity. We included in the sample all new registered users on the English Wikipedia (excluding mobile registrations) during the two rounds of the split test (log version: 1, 2). Cohorts in the GettingStarted funnel were obtained by checking for the first click upon account creation. Cohorts in the ReturnTo funnel were obtained by checking for impressions on an editable article linked via the ReturnTo button.

OB5 cohorts
Cohort ID	Unique users	Description	% of total bucketed in group
All users
e3_ob5a	12,201	Users in the control group	100%
e3_ob5b	12,375	Users in the test group	100%
GettingStarted funnel
e3_ob5b_gettingstarted-any_page-impression	4,047	Unique users in the test group who clicked on one of the three GettingStarted buttons upon successful account creation.	32.7%
e3_ob5b_gettingstarted-addlinks_page-impression	1,446	Users in the test group who clicked "Add links" upon successful account creation.	11.7%
e3_ob5b_gettingstarted-clarify_page-impression	981	Users in the test group who clicked "Clarify" upon successful account creation.	7.9%
e3_ob5b_gettingstarted-copyedit_page-impression	1,661	Users in the test group who clicked "Copyedit" upon successful account creation.	13.4%
ReturnTo funnel
e3_ob5a_returnto_page-impression	693**	Users in the control group who returned to an editable page via the ReturnTo button upon successful account creation.	5.7%
e3_ob5b_returnto_page-impression	1,076	Users in the test group who returned to an editable page via the ReturnTo button upon successful account creation.	8.7%

Notes

Due to a bug that affected the logging of events in the control condition in v1, the e3_ob5a_returnto_page-impression cohort is incomplete and its size cannot be used for historical comparison.

Live account rate

We measured the proportion of "live accounts" or users in each cohort clicking at least once on the edit button on an article in the main namespace within 24 hours of registration (measurements taken 24 hours after the last valid registration).

OB5 ns0 24h live accounts (test vs control)
ID	Unique users	Live accounts	%
e3_ob5a	12,201	4,142	33.9%
e3_ob5b	12,375	4,801	38.8%

We found a significant 4.9% difference in the proportion of live accounts between test and control [X² = 62.16, N = 24,576, p < .01]. The live account rates for the other cohorts are the following:

OB5 ns0 24h live accounts (subcohorts)
ID	Unique users	Live accounts	%
GettingStarted funnel
e3_ob5b_gettingstarted-any_page-impression	4,047	1,731	42.8%
e3_ob5b_gettingstarted-addlinks_page-impression	1,446	521	36.0%
e3_ob5b_gettingstarted-clarify_page-impression	981	494	50.3%
e3_ob5b_gettingstarted-copyedit_page-impression	1,661	728	43.8%
ReturnTo funnel
e3_ob5a_returnto_page-impression	693	320	46.2%
e3_ob5b_returnto_page-impression	1076	450	41.8%

1 edit in 24 hours

We measured the proportion of users in each cohort completing their first main-namespace edit within 24 hours of registration (measurements taken 24 hours after the last valid registration).

OB5 1+ ns0-edit 24h threshold (test vs control)
ID	Unique users	Editing users	%
e3_ob5a	12,201	2,831	23.2%
e3_ob5b	12,375	3,081	24.9%

We compared the proportion of threshold-hitting users in each group and found a significant 1.7% difference between test and control [X² = 9.55, N = 24,576, p < .01]. The threshold-hitter rates for the other cohorts are the following:

OB5 1+ ns0-edit 24h threshold (subcohorts)
ID	Unique users	Editing users	%
GettingStarted funnel
e3_ob5b_gettingstarted-any_page-impression	4,047	928	22.9%
e3_ob5b_gettingstarted-addlinks_page-impression	1,446	230	15.9%
e3_ob5b_gettingstarted-clarify_page-impression	981	294	30.0%
e3_ob5b_gettingstarted-copyedit_page-impression	1,661	409	24.6%
ReturnTo funnel
e3_ob5a_returnto_page-impression	693	201	29.0%
e3_ob5b_returnto_page-impression	1,076	301	28.0%

5+ edits in 24 hours

We also measured the proportion of users in each cohort completing 5 or more main-namespace edits within 24 hours of registration (measurements taken 24 hours after the last valid registration). The proportion of 5+ threshold hitters varies insignificantly between the two conditions.

OB5 5+ ns0-edit 24h threshold (test vs control)
ID	Unique users	Editing users	%
e3_ob5a	12,201	468	3.8%
e3_ob5b	12,375	489	3.9%

The 5+ threshold-hitter rates for the other cohorts are the following:

OB5 5+ ns0-edit 24h threshold (subcohorts)
ID	Unique users	Editing users	%
GettingStarted funnel
e3_ob5b_gettingstarted-any_page-impression	4,047	142	3.5%
e3_ob5b_gettingstarted-addlinks_page-impression	1,446	39	2.7%
e3_ob5b_gettingstarted-clarify_page-impression	981	40	4.0%
e3_ob5b_gettingstarted-copyedit_page-impression	1661	65	3.9%
ReturnTo funnel
e3_ob5a_returnto_page-impression	693	29	4.2%
e3_ob5b_returnto_page-impression	1,076	45	4.2%

Revert rate

OB5 24-hour mean revert rate (test vs control)
ID	Unique users	Reverts	%
e3_ob5a	12,201	998	8.1%
e3_ob5b	12,375	1,046	8.4%

Funnel analysis

Guided Tour

The following is the breakdown of users in the e3_ob5b_gettingstarted-any_page-impression group taking the Guided Tour embedded in the navigation bar within 24 hours of the end of the split test.

GuidedTour funnel
ID	Total users	Step 1	Step 2	Step 3	Step 4	Step 5
e3_ob5b_gettingstarted-any_page-impression	4,047 (100%)	366 (9.0% / 9.0%)	322 (88.0% / 7.9%)	274 (85.0% / 6.8%)	173 (63.1% / 4.3%)	54 (31.2% / 1.3%)

Conclusions

Though we have not yet made a gain in the rate of active editors hitting the 5+ edits threshold, we have so far observed the largest gain in new editors hitting the 1+ edits threshold of any of our controlled tests of Getting Started. Funnel analysis and further analysis of subcohorts (such as by task type and those who opted in to the guided tour) is merited, to understand more detail about which features may be driving this effect.