Research:Article feedback/Reader survey

Article Feedback v5	Data & Metrics	Stage 1: Design (December 2011 - March 2012)	Stage 2: Placement (March 2012 - April 2012)	Stage 3: Impact on engagement (April 2012 - May 2012)
WP:AFT5 (Talk) Feature requirements Dashboards Feedback Feedback moderation	Overview Article samples Feature data Clicktracking data	Volume analysis Quality assessment Reader survey / Team survey Usability testing	Volume analysis Quality assessment	Conversions and newcomer quality Final tests Quality assessment Research report (2012 Q4) Moderation tools usability study

Overview

The results of the reader survey

The Wikimedia Foundation, in collaboration with editors of the English Wikipedia, is developing a tool to enable readers to contribute productively to building the encyclopedia. To that end, we started development of a new version of the Article Feedback Tool (known as AFTv5) in October 2011. Once the AFTv5 feedback forms were deployed in late December 2011, we began gathering data on how our editors, readers and team members felt about each design. This report is covers the survey responses we received from readers.

From 25 January to 13 February 2012, the call to action - a panel that appears after the feedback form has been used, normally encouraging readers to try editing - was replaced with a link to a survey that invited readers to give feedback on Version 5 itself. We gathered both quantitative and qualitative data in this fashion, using questions that are included in the appendix of this report, for each version of the form. In total, 1,472 people have filled out the survey, with an average of 491 for each of the three versions of the form. Our intention was to discover which version readers preferred before making a final decision on what would be used when the tool is finally deployed.

Key findings

Of the readers who chose to fill out the survey, an average of 64 percent approved of at least one version of the form, while 13 percent disapproved - the remaining 23 percent were not sure about the tool's usefulness. Using only percentages, Option 1 and Option 3 were tied, with 66 percent approval for each. However, the net likes (the percentage who liked it, minus the percentage who did not) puts Option 1 in the lead, from a reader perspective with 55 percent net approval compared to 52 percent for Option 3. In addition, although Options 1 and 3 were tied for raw "likes", the survey for Option 1 gathered far more feedback, with 38.63 percent of the feedback coming from Option 1 compared to 33.83 percent for Option 3. Overall, Option 1 is narrowly considered the best design by our readers, with both raw and net data bringing it into the lead if one considers the substantially larger number of readers commenting on this design compared to the others.

Option 1

"make it more eye-catching on your website. otherwise it is perfect."

"no free text - prefer options. its more quick"

"You aced it, as far as I can tell, it's nice and easy to use. "

66% like

11% don't like

55% net likes

The first form we asked readers to comment on, “Option 1”, asked readers “Did you find what you were looking for?”, with “yes” and “no” check boxes, and contained a non-mandatory free text field; readers were only required to either the check boxes or the free text field, not both. This design was deployed in December on 0.3 percent of all Wikipedia articles, and the sample size was doubled on 4 January to 0.6 percent. 578 readers chose to respond to the survey for Option 1 - 38.63 percent of the total number of respondents - and, overall, 66 percent of them liked the design. Another 11 percent did not like Option 1, while 22 percent were unsure. This represents both the highest percentage of "likes" and the lowest percentage of "dislikes", while gathering more pieces of feedback than either of the other surveys. Using a "net likes" metric (the total number of likes, minus the total number of dislikes), 55 percent of readers approve of the form - the highest of any option.

Option 2

"It's simple and understandable."

"It should be small. The bigger the size of the popup, the bigger the event seems."

"it's already quite good"

59% like

13% don't like

46% net likes

The second form we tested, “Option 2”, clearly indicated what sort of feedback we were looking for by subdividing the form into “suggestion” “praise”, “problem” and “question”. A reader is required to select one of those tabs (“suggestion” is checked by default) and then enter text into the free text box. Entering text was mandatory for this form, simply because selecting a category and then hitting “post your feedback” does not provide us with any useful data, unlike the check-boxes for the other two forms. 396 readers chose to respond to our survey about Option 2, representing 26.90 percent of total survey respondents. Overall, 59 percent of readers liked the design, 13 percent did not like it, and 28 percent were unsure; this represents the lowest percentage of likes, and the highest percentage of "unsure" responses. Using the "net likes" metric, 46% of readers liked the design - again, a far lower percentage than for Options 1 and 3.

Option 3

"Quick and easy and as a result I decided to do the quick survey, which took less than a minute!"

"You could improve the feedback form by completely removing it from the article. It's like having a popup in front of the article that I can't block. Why the hell is it here?"

"It was great. I don't think it needs changing."

66% like

14% don't like

52% net likes

The last form, “Option 3”, asked readers “Did you find what you were looking for?” and contained a non-mandatory text box. Like the existing system, it includes a five-star rating system, which meant we were comfortable with making the text box optional – even if readers only submit a rating, useful data can still be gathered. 498 readers chose to respond to the Option 3 survey, constituting 33.83 percent of the total responses. Of these respondents, 66 percent liked the form, while 14 percent disliked it, and 20 percent were not sure. This represents both the highest percentage of likes, tied with Option 1, and the highest percentage of dislikes. Using the "net likes" metric, 52 percent of readers approved of this design.

Conclusions

Based merely on the percentage of respondents who approved of the design, Option 1 and Option 3 are tied, with 66% of readers liking each form. However, the Option 1 survey was responded to by more users - making up 38.63 percent of the responses, as opposed to 33.83 percent for Option 3 - meaning that, numerically, more readers liked Option 1 than Option 3. In addition, Option 3 garnered a higher dislike rate - 14 percent to Option 1's 11 percent. Using the net likes metric, Option 1 comes out slightly in the lead, with 55 percent of readers approving of the design compared to 52 percent for Option 3. In conclusion, Option 1 is the design readers prefer overall, albeit narrowly.

Metric	Option 1	Option 2	Option 3
Percentage of likes	66%	59%	66%
Percentage of dislikes	11%	13%	14%
Percentage of unsures	22%	28%	20%
Net likes	55%	46%	52%

Appendix

Survey Questions

• (1) Why did you fill in the article feedback form today?
  

I want to: (check all that apply)

• improve the article I was reading
  
• make a suggestion
  
• ask a question
  
• give praise
  
• report a problem
  
• share my opinion
  
• other

• (2) What do you think of that feedback form?
  

• I like it.
• I do not like it
• I am not sure

• (3) Why? How can we improve that feedback form?
  

[add a comment]