Research:Mobile editor engagement/Tutorial test

The tutorial A/B test is a controlled experiment designed to test if displaying a short tutorial about wiki markup to newly registered users editing on mobile devices affects their edit completion rate. The test was initially enabled for Beta users only. The actual controlled experiment was rolled out in November for users of the stable version of the mobile site.

Timeline

Screenshot of the tutorial displayed to users in the test condition.

2013-11-01 00:00:00: Test starts
2013-11-15 00:00:00: Test ends

Experimental design

We selected a sample of users newly registered via a mobile device on the English Wikipedia for a period of at least two weeks. Only users registering via the edit CTA were considered eligible for this test.
We randomly assigned eligible users in the sample to one of the following two conditions, based on their user id

Tutorial (test): when clicking for the first time on the edit button (whether on a section or on the entire page), the user sees a tutorial explaining how to edit wiki markup.
No tutorial (control): users clicking on the edit button are directly sent to the edit screen. Note that this is not the default UX for new users registering via the edit CTA (as they are normally redirected to the article they came from, not the edit screen).

Users remained bucketed in the experimental condition for the duration of the test (initially set to 2 weeks). At the end of the test, they reverted to default condition.

Research questions

Does displaying a tutorial increase the edit completion rate of new mobile registered users (proportion of users registering via the edit CTA funnel and successfully editing the page they came from)?
Does the tutorial affect the overall activation rate of these users?
Are there differences in the quality of contributions made by users in each condition, as measured via unreverted edits?

Data sources

We used for this test a combination of data from the MediaWiki database and EventLogging logs (Schema:MobileWebEditing).

Cohort analysis

A total of 3,434 users mobile users registered on the English Wikipedia via the Edit CTA participated in this test.

Tutorial test: experimental groups
Cohort ID	Unique users	Description
mobile_tutorial_test_control	1,780	Users in the control group
mobile_tutorial_test_test	1,654	Users in the test group

The following is the proportion of users in each group attempting and completing an edit on the referral page. We only considered editable ns0 pages and conservatively included edit events up to 1 hour after registration (i.e. not limited to the account registration referral page).

We found a significant 8.5% difference in the proportion of users attempting to edit the referral page between control and test [X² = 27.95, N = 3,434, p < .001].
Test was the winning condition for edit completion, with a significant, 8.4% difference in the proportion of users successfully completing an edit of the referral page compared to users in the control group [X² = 28.70, N = 3,434, p < .001].

Tutorial test: Edit CTA funnel
Cohort ID	Unique users	Description	% of total bucketed in group
Users attempting to edit the referral article
mobile_tutorial_test_control	1,265	Users in the control group	71.1%
mobile_tutorial_test_test	1,034	Users in the test group	62.5%
Users completing an edit on the referral article
mobile_tutorial_test_control	465	Users in the control group	26.1%
mobile_tutorial_test_test	572	Users in the test group	34.6%

We also considered the overall main-namespace editing behavior of these users in their first 24 hours since registration.

When looking at the overall number of users completing a main-namespace edit in their first day (whether on mobile or desktop), test significantly outperformed control with a 3.5% difference in proportion compared to control. [X² = 4.18, N = 3,434, p < .05].
The difference between test and control was found only marginally significant when we considered mobile edits only and excluded edits these users completed on desktop [X² = 3.47, N = 3,434, p ~ .05].

Tutorial test: 1+ ns0 edits in 24 hours
Cohort ID	Unique users	Description	% of total bucketed in group
Users with 1+ ns0 edits on mobile or desktop
mobile_tutorial_test_control	687	Users in the control group	38.6%
mobile_tutorial_test_test	696	Users in the test group	42.1%
Users with 1+ ns0 edits on mobile only
mobile_tutorial_test_control	651	Users in the control group	36.6%
mobile_tutorial_test_test	657	Users in the test group	39.7%

Data quality

See a list of issues and proposed changes in the instrumentation.

Discussion

In response to the research questions we know: Does displaying a tutorial increase the edit completion rate of new mobile registered users (proportion of users registering via the edit CTA funnel and successfully editing the page they came from)? Yes we saw a statistically significant (8.5%) advantage to showing the edit guider/tutorial in getting users to complete an edit on that page.

Does the tutorial affect the overall activation rate of these users? Yes we saw a statistically significant (3.5%) advantage to showing the edit guider/tutorial in getting users to activate (1+ ns0 edits in 24hrs).

This became a marginally statistically significant (3.5%) advantage when we looked only at completed mobile edits. However, this was not the research question and is more a product of amount of data. It does not change the fact that activation rate for the test group was higher than the control group.

Are there differences in the quality of contributions made by users in each condition, as measured via unreverted edits? There are not statistical differences in the quality of edits. [data needs to be entered]

As a result of this we conclude that the edit guider is a good on boarding strategy for new users. This guider is similar to the strategy employed by the desktop Growth team's OB6 test. From this we can conclude that some combination of reminding the user what they were doing/the context they are in, warning the user about wikitext, and hand holding the user through the workflow; is a strategy for on boarding first time users that is useful on both desktop and mobile for new Wikipedia editors.

This edit guider will be the default experience for new editors going through the edit CTA and a modified version of the guider is being developed for users logging in through the left nav menu, this feature will also be subject to an A/B test.