This page in a nutshell: The goal of this experiment is to measure the effect of guided tours in driving edit conversions, within the context of the Onboarding New Wikipedians project. Among the users who viewed a GettingStarted article and were given a guided tour, we found a statistically significant 3.9% difference in the rate of first edits made, and an even larger 13.6% increase in the rate at which new users attempted to edit.
This iteration started on 2013-02-28T21:50:18Z[1] and ended on 2013-03-08T00:20:00.[2] We randomly split a sample of new users registered on the English Wikipedia into a control group (OB3a) and a test group (OB3b). Users in the control group were served a list of copyediting tasks generated by Extension:GettingStarted. Users in the test group saw the same tasks but in addition they were shown guiders when visiting any of the articles in the list. We showed the returnto button to these users but the button doesn't trigger a guided tour and it behaves exactly as in the control group.
We logged the editing activity of users entering one of the two following subfunnels from the GettingStarted landing page in both the control and the test group:
returnto
users clicking on the "Return To" button and attempting to edit the page they came from when they registered an account
gettingstarted
users clicking on one or more links in the task section of the GettingStarted page
Users in the control group neither returning to an editable page via the ReturnTo button nor landing on a GettingStarted article upon successful account creation.
Users in the test group neither returning to an editable page via the ReturnTo button nor landing on a GettingStarted article upon successful account creation.
Fig.1 Proportion of new registered users in the OB3 test accepting a task and clicking the edit button at least once in the main namespace within 24 hours of registration.
We measured the proportion of "live accounts" or users in each cohort clicking at least once on the edit button on an article in the main namespace within 24 hours of registration (measurement taken 24h after the last valid registration).
We compared the proportion of live accounts in the two experimental groups and found a significant 2.2% increase in the test group compared to control [X² = 16.80, N = 31,713, p < .001]. The effect is much more pronounced if we only consider the subgroup of users accepting a task (i.e. landing on an article linked from the task) in the two conditions, where the difference between test and control is 13.6% [X² = 65.3, N = 3,524, p < .001] (see fig.1).
Fig.2 Proportion of 1+ 24h main-namespace editors among new registered users in the OB3 test who accepted a task (test vs control).
We measured the proportion of users in each cohort completing their first main-namespace edit within 24 hours of registration (measurement taken 24h after the last valid registration).
We compared the proportion of threshold-hitting users in each group and found a significant 1.1% difference between test and control [X² = 5.42, N = 31,713, p < .05]. There was a larger, significant difference of 3.9% between test and control for users accepting a task [X² = 6.53, N = 3,524, p < .05] (see fig.2).
We also measured the proportion of users in each cohort completing 5 or more main-namespace edits within 24 hours of registration (measurement taken 24h after the last valid registration). The proportion of 5+ threshold hitters varies insignificantly between the two conditions (both when comparing the experimental groups with each other or the subgroup of users accepting a task).
Users in the test group being served the 'gettingstarted' guided tour
Buckets
All eligible users in the test are accurately bucketed into one of the two groups (no eligible user missing a bucketId) and the sum of users with a valid bucketId is identical to the sample of eligible users (no spurious bucketIds corresponding to non-eligible users).
GettingStarted page-impression vs GuidedTour impression
697 eligible unique users generated a GuidedTour impression out of a total 726 users in the test cohort (the 29 user difference being due to users who tested guided tour without being part of the OB3 experiment). Out of these 697 users, 87 were in the control group but generated guided tour impressions erroneously. These 87 users in the control group who landed on an editable page in the gettingstarted funnel are a potential issue as they should not have been served a guided tour and should not generate any GuidedTour event.
We will be correcting for this error where 87 users in the control group were served the guided tour by ensuring that only first time visitors can be served the tour. When that fix was deployed 2/14 (48596), we ran the test for a full week through 2/21.
Also note that for the week-long test of 2/14-2/21, bug 45251 means that a number of users were logged as being in the test group, but may have not been delivered the test tour. This led us to re-enable the test from Friday 2/22-Thursday 3/07.
