Research:Onboarding new Wikipedians/OB4.5
Overview
editCohorts
editAfter we enabled the Getting Started landing page in OB4 as default for 100% of new signups, the Extension:Echo, a new notification system was deployed on English Wikipedia. Following launch, all new editors received both the Getting Started landing page with task suggestions, as well as the following notifications:
- A generic welcome message
- On email confirmation, a message via Web and email directing users to Special:GettingStarted, with slightly different language depending on whether the user had made a namespace zero edit or not.
The cohort of these users starts from launch of Echo at 8pm UTC on April 30, 2013[1], to the launch of the "red popup" gadget[2]. Users after the gadget launch still receive notifications, but their experience of interacting with Web notifications is significantly more annoying and confusing, so we will discount these users from the initial cohort.)
In the present analysis we considered all new desktop users registered on the English Wikipedia between 2012-04-30T20:00:00Z
and 2012-05-07T00:37:00Z
.
Notes
editNotifications generated by GettingStarted are special insofar as they are triggered (both on-site and by mail) only for users with an authenticated email address. This introduces a potential confound in the analysis, as we are not measuring the effect of notifications on the total population of new users registered in the above period, but on the subpopulation of users with an authenticated email address (who may have a higher level of engagement than occasional users who register without specifying an email address or without verifying it). We can correct for this confound by comparing this cohort only to the previous OB4 users who authenticated an email address.
Even without the above confound, minor differences are potentially due to seasonal fluctuations or other unknown differences between two week-long cohorts, but it may still be worthwhile to see if there are drastically better or worse activity levels in the cohort that got notifications. In the future we will run a proper A/B test measuring the impact of any notifications.
Research questions
edit- What is the overall activity level of the cohort who received notifications?
- How does this cohort's activity level compare to that of OB4, which received the same landing page but no notifications?
Metrics
edit- The rate at which this cohort hit 1+ NS0 edits in 24hrs
- The rate at which this cohort hit 5+ NS0 edits in 24hrs
Cohort analysis
editCohort ID | Unique users | Description |
---|---|---|
e3_ob45 | 20,655 | All new users registered in the 1st week since the launch of Echo notifications and landing on GettingStarted v.2. |
e3_ob45_echo_gettingstarted-start-editing | 4,703 | Users with an authenticated email address registered in the 1st week since the launch of Echo notifications who received a gettingstarted "start editing" notification by mail within the same 1-week period. |
e3_ob45_echo_gettingstarted-continue-editing | 231 | Users with an authenticated email address registered in the 1st week since the launch of Echo notifications who completed their first edit within the same 1-week period, received a gettingstarted "continue editing" notification and visited once more Special:GettingStarted within the same period. |
Live account rate
editWe measured the proportion of "live accounts" or users in each cohort clicking at least once on the edit button on an article in the main namespace within 24 hours of registration (measurement taken 24h after the last valid registration).
ID | Unique users | Live accounts | % |
---|---|---|---|
e3_ob45 | 20,655 | 7,699 | 37.3% |
e3_ob45_echo_gettingstarted-start-editing | 4,703 | 972 | 20.7% |
e3_ob45_echo_gettingstarted-continue-editing | 231 | 204 | 88.3% |
1 edit in 24 hours
editWe measured the proportion of users in each cohort completing their first main-namespace edit within 24 hours of registration (measurement taken 24h after the last valid registration).
ID | Unique users | Editing users | % |
---|---|---|---|
e3_ob45 | 20,655 | 5,161 | 25.0% |
e3_ob45_echo_gettingstarted-start-editing | 4,703 | 354 | 7.5% |
e3_ob45_echo_gettingstarted-continue-editing | 231 | 223 | 96.5% |
Note
- The near-100% threshold hitter rate for the e3_ob45_echo_gettingstarted-continue-editing cohort is expected by the very definition of this cohort.
5+ edits in 24 hours
editWe also measured the proportion of users in each cohort completing 5 or more main-namespace edit within 24 hours of registration (measurement taken 24h after the last valid registration).
ID | Unique users | Editing users | % |
---|---|---|---|
e3_ob45 | 20,655 | 864 | 4.2% |
e3_ob45_echo_gettingstarted-start-editing | 4,703 | 71 | 1.5% |
e3_ob45_echo_gettingstarted-continue-editing | 231 | 47 | 20.3% |
Conclusions
editIn this cohort analysis (i.e. not a controlled A/B test, but an observational analysis of a group of editors who all joined during the same week), we found that the entire cohort exposed to the new GettingStarted notifications had a first time edit rate of 25% -- slightly higher than previous cohorts. We also found that users who did not yet edit by the time they confirmed their email address, but who received a notification inviting them to do so, came back and edited at a rate of 7.5%. While users who received a notification inviting them to continue editing if they are already had were very successful (20.3% of them reached 5 NS0 edits in 24hrs after registration), this is likely not due to the notifications alone. In the future, we may run a controlled test to further determine the precise impact of notifications on new editors.
Notes
edit- ↑ The first
gettingstarted-start-edit
notification was delivered at2013-04-30T19:58:59Z
- ↑ This community gadget was turned on by default at 00:37 UTC on May 7, 2013