Research:Onboarding new Wikipedians
The purpose of this growth project is to build an onboarding experience for Wikipedians, starting in English, that shows new users how to get started with making productive contributions to the project. See more about what this means in the context of Wikipedia in our design document.
Research questions
edit- How many participants in an onboarding experience contribute to the encyclopedia?
- Which onboarding experience is most effective at helping newly-registered people edit?
Overall metrics
editThe primary metrics we are interested in are:
- Increased the number of newly-registered editors who complete one edit
- Increased the number of newly-registered editors who complete five edits
The reason for these metrics is because they are directly relatable to the active editors measurements of Wikipedia overall.
Event logging
editWe've used the following schemas to log events related to this experiment, in roughly chronological order:
Changelog
edit| Shortcut | Changes | Start | End | Schema rev id | Version |
|---|---|---|---|---|---|
| OB0 | Launch! [screenshot] | 2012-12-13 | 2012-12-20 | - | - |
| OB0 | Minor design updates [screenshot] | 2012-12-20 | 2013-12-22 | - | - |
| OB0 | No deploy, but fully randomized the list of tasks, creating much higher refresh rate | 2012-12-22 | 2013-01-10 | 4876947 | - |
| OB1 | Added returnTo funnel logging | 2013-01-10 | 2013-01-25T00:00:00Z | 4993508 | - |
| OB2 | 50/50 split test vs control (no GettingStarted) | 2013-01-25T00:00:00Z | 2013-01-31T22:45:00Z | 4993508 | - |
| OB3 | 50/50 split test: GettingStarted + GuidedTour vs GettingStarted only | 2013-02-08T01:16:56Z | 2013-02-14 | 5219269, 5222838 | ob3 |
| OB3 | 50/50 split test: GettingStarted + GuidedTour vs GettingStarted only (stronger enforcement of control treatment) | 2013-02-14 | 2013-02-21T20:59:39Z | 5243394, 5222838 | ob3-split |
| OB3 | GuidedTour at 100% for all users | 2013-02-21T20:59:39Z | 2013-02-22T20:17:16Z | 5243394, 5222838 | ob3-all |
| OB3 | 50/50 split test: GettingStarted + GuidedTour vs GettingStarted only (fixed bug with urlencoded page titles) | 2013-02-22T20:17:16Z | 2013-02-25T20:56:01Z | 5243394, 5222838 | ob3-split |
| OB3 | 50/50 split test: GettingStarted + GuidedTour vs GettingStarted only (reinstate test after merge disabled it) | 2013-02-28 | 2013-03-08T00:20:00Z | 5285779, 5222838 | ob3-split-retest-split |
| OB4 | Launch of new landing page | 2013-03-08T00:20:00 | 2013-03-14 | 5285779 | ob3-split-retest-split |
| OB4 | 50/50 split test: GettingStarted v.2 vs control (no GettingStarted) | 2013-03-14 | 2013-03-21 | 5320430 | 1 (version replaces experimentId) |
| OB4 | GettingStarted v.2 at 100% | 2013-03-21 | 2013-04-18 | 5320430 | 2 |
| OB4 | GettingStarted v.2 at 100%, version bump to note removal of BLPs | 2013-04-18 | 2013-04-30 | 5320430 | 3 |
| OB4.5 | GettingStarted v.2 at 100%, version bump to note launch of notifications | 2013-04-30T20:00:00Z | 2013-05-16 | 5320430 | 4 |
| OB5 | Split test of GettingStarted with new landing page and new nav bar | 2013-05-16 | 2013-05-20 | 5467309 | 1 |
| OB5 | Split test of GettingStarted with new landing page and new nav bar, update logging | 2013-05-20 | 2013-05-23T20:31:14Z | 5496876 | 2 |
| OB5 | End split test, deliver GettingStarted new landing page, nav bar, and Echo to all users | 2013-05-23T20:31:14Z | 2013-06-26T09:33:23Z | 5496876 | 3 |
| OB5 | Improve appearence of task toolbar on low width screens and print, and log whether the toolbar was shown with the page-* impression. | 2013-06-20T20:50:39Z | 2013-08-08 | 5588671 | 4 |
| OB6 | A/B test of the next design iteration of the GettingStarted workflow | 2013-09-08 | 2013-09-15 | 5658668 | 1 |
| Rollout | Continued rollout of onboarding across languages | 2014-02-06 | ∞ | 7355552 | N/A |
![[screenshot]](/wiki/File:Screen_Shot_2012.12.13_of_Special_page_GettingStarted.png){kind=link}
![[screenshot]](/wiki/File:Screen_Shot_20130103_of_Special_GettingStarted.png){kind=link}
Analysis
editPrimary metrics:
- How many users in the funnel complete 1+ NS0 edits in 24 hours
- How many users in the funnel complete 5+ NS0 edits in 24 hours
- Conversion rates in each funnel (GettingStarted, control, returnTo and other comparable funnels)
- As a proportion of total impressions
- As a proportion of the previous step in the funnel
- How many edits made through the funnel are reverted?
We also have conducted further qualitative analysis about editing activity by users during this experiment. Individual experiments also may measure the above metrics for subcohorts within each experiment, to compare the effect of different user interface modifications within an experiment.
Overview of results
editResults refer to the whole experimental groups from split tests only. All reported effects are significant (p < .05), full reports are available under each test page.
24 hours live account rate
edit| Test | Effect size |
|---|---|
| OB2: GettingStarted v.1 vs default welcome page | 4.3
|
| OB3: GettingStarted v.1 + GuidedTour vs GettingStarted v.1 only | 2.2(net effect of guiders)
|
| OB4: GettingStarted v.2 vs default welcome page | 3
|
| OB5: GettingStarted v.3 vs default welcome page | 4.4
|
1+ edit in 24 hours
edit| Test | Effect size |
|---|---|
| OB2: GettingStarted v.1 vs default welcome page | 1.9
|
| OB3: GettingStarted v.1 + GuidedTour vs GettingStarted v.1 only | 1.1(net effect of guiders)
|
| OB4: GettingStarted v.2 vs default welcome page | 1.5
|
| OB5: GettingStarted v.3 vs default welcome page | 1.7
|