Research:Community portal redesign/Opentask

The purpose of this experiment is to measure the effects of modifications to the open task list on the Community portal on English Wikipedia.

BackgroundEdit

The Community portal attracts thousands of visitors each day, the majority of whom are anonymous and new editors. A relatively high percentage of these users click on articles in the open task list, which is refreshed hourly by a bot with articles from various maintenance categories.

While we know how many users are clicking on these articles, we don't have a good sense of how many of them are attempting and/or completing an edit. Qualitative coding of a day's worth of data suggests that few to no edits emerge from these visits. Measuring the number of task attempts and completions in a discrete period of time will give us a good baseline for other task recommendation focused experiments, as well as provide insight for iteratively improving the usability of this and other task recommendation lists.

Extended content

Week 1Edit

Click resultsEdit

Total clicks
1206
Task # of total clicks # of unique clicks # of returned clicks
Wikify 49 92% 8%
Copyedit 60 82% 18%
Remove original research 37 95% 2%
De-orphan 183 88% 12%
Verify 41 90% 10%
Merge 61 90% 10%
Split 91 93% 7%
Expand 166 90% 10%
Make neutral 36 94% 6%
Update 78 92% 8%
Discuss AfD 178 93% 7%
Clean up 55 87% 13%
Add image 7 100% 0%
Translate 104 95% 5%
Create 60 88% 12%
Total 1206

Week 2Edit

Click resultsEdit

Total clicks
2125

(50% increase from week 1)

Task # of total clicks # of unique clicks # of returned clicks
Wikify 435 90% 10%
Copyedit 345 85% 15%
Remove original research 186 86% 14%
De-orphan 148 89% 11%
Verify 101 92% 8%
Merge 74 89% 11%
Split 80 90% 10%
Expand 94 93% 7%
Make neutral 101 90% 10%
Update 86 95% 5%
Discuss AfD 72 90% 10%
Clean up 128 96% 4%
Add image 71 99% 1%
Translate 92 92% 8%
Create 183 85% 15%
Total 2191
Summary
Increasing the whitespace and visibility of the list and removing excess links and formatting doubled the number of clicks on open tasks.

Week 3Edit

Click resultsEdit

Total clicks
Task # of total clicks # of unique clicks # of returned clicks
Wikify 156 94% 6%
Copyedit 187 82% 18%
Remove original research 75 91% 9%
De-orphan 79 90% 10%
Verify 78 87% 13%
Merge 53 92% 8%
Split 64 97% 3%
Expand 80 84% 16%
Make neutral 110 90% 10%
Update 94 93% 7%
Discuss AfD 61 93% 7%
Clean up 100 97% 3%
Add image 90 90 100%
Translate 79 86% 14%
Create 82 89% 11%
Total 1388
Summary
Moving the Collaborations section to the top of the page reduced clicks on opentask articles, though there were still more clicks generated than in the original page layout.

Measuring the funnelEdit

 
Funnel of task acceptance, attempt, and completion

Funnels are ways of measuring user conversion through multiple steps and tasks. We're interested in how many unique users who view the landing page go on to view and attempt one of the tasks. Additionally, we're interested in seeing what kinds of tasks most users are attempting, and whether they successfully save their edits to those task articles.

Event Definition What's tracked Properties
Opentask click User clicks on any article in the open task section of the Community portal anonymized token, timestamp, click target task type (e.g., "copyedit," "npov", etc.)
Edit click User clicks on the edit button of the task page anonymized token, timestamp, click target task type
Edit completion User closes the editing interface either through a successful save or cancel action anonymized token, timestamp, click target save, task type
CP return User returns to the CP after passing through any stage of the funnel anonymized token, timestamp, click referrer none

ResultsEdit

 

Pre-redesign (November 3-14)

The above chart shows unique users converted along each of the following steps: 1) clicking any link on the Community portal, 2) clicking on an opentask link (accepting a task), 3) clicking the edit button on the article, 4) saving an edit to the article. Except for the first column, each column is the percentage of users converted from the previous step (the number of users who completed that step divided by the number of users who completed the step to the left).*

*Absolute numbers: 3239 users accepted a task, 136 clicked edit, 27 saved edit.

It appears that the major drop-off point in the funnel is between accepting a task (clicking on one of the articles in the opentask list) and finding/clicking the edit button. This suggests that users are interested in trying out a task, but either don't find the edit button or can't figure out how they can improve the article.

After the redesign (November 15-26)

*Absolute numbers: 1193 users accepted a task, 56 attempted an edit, and 19 saved. This represents a small increase (from 4% to 5%) for edit attempts, and a larger increase (from 0.1% to 2%) of successful saves per click-through to task.

Conversion by taskEdit

 

The above image shows absolute numbers of unique users selecting, attempting, and completing the different kinds of tasks in the opentask list. Copyediting and wikifying appear to be the two most successful task types, measured by both numbers of users selecting articles in those task types and successfully editing the articles.

Clicks and conversionsEdit

The following is a heatmap of clicks on the different sections of the opentask list, and a table of unique users who passed through different stages of the funnel and the percentages that completed each stage (except task, edit article, save edit) in a 10-day period. The top two task types, wikifying and copyediting, received a fourth of all the clicks on the list. However, only copyediting was able to convert a significant number of users.

Heatmap of clicks

 

Complete table of unique users in the funnel, by task type
Task type # of unique users who accepted at least once # of users who clicked on the edit button % converted # of users who saved an edit % converted
Improve prose 136 8 6% 0 0
Copyedit 412 41 10% 15 37%
Wikify 447 14 3% 4 29%
Discuss AfD 144 0 0 0 0
Translate 214 10 5% 0 0
Merge 167 6 4% 1 17%
Split 108 5 5% 0 0
Verify 245 8 3% 0 0
Update 159 4 3% 0 0
Cleanup 279 5 2% 1 20%
Orphan 207 7 3% 0 0
Expand 136 4 3% 2 50%
Add image 187 13 7% 2 15%
NPOV 130 2 2% 0 0
Original research 252 9 4% 2 22%
Total 3239 136 n/a 27 n/a