Research:Article feedback/Team survey

Article Feedback v5	Data & Metrics	Stage 1: Design (December 2011 - March 2012)	Stage 2: Placement (March 2012 - April 2012)	Stage 3: Impact on engagement (April 2012 - May 2012)
WP:AFT5 (Talk) Feature requirements Dashboards Feedback Feedback moderation	Overview Article samples Feature data Clicktracking data	Volume analysis Quality assessment Reader survey / Team survey Usability testing	Volume analysis Quality assessment	Conversions and newcomer quality Final tests Quality assessment Research report (2012 Q4) Moderation tools usability study

Overview

The Wikimedia Foundation, in collaboration with editors of the English Wikipedia, is developing a tool to enable readers to contribute productively to building the encyclopedia. To that end, we started development of a new version of the Article Feedback Tool (known as AFTv5) in October 2011. Once the AFTv5 feedback forms were deployed in late December 2011, we began gathering data on how our editors, readers and team members felt about each design. This report covers the survey responses we received from team members.

From 30 January to 6 February 2012, article feedback team members were invited to take a short survey distributed by email, asking for their perspectives on each proposed Version 5 design. Invitations were also sent to external developers involved in this project, as well as members of our workgroup - a core set of volunteers who have been helping develop Version 5. 19 individuals responded in total - 12 Wikimedia staff members, three developers, two editors and two anonymous respondents - providing feedback on each design, both quantitatively (by rating how useful they would, as a reader, find it) and qualitatively (through a free text box asking what made the design useful, and how it could be improved).

Overall, Option 1 was preferred by these team members, with 84 percent considering that feedback form useful, compared to 73 percent for Option 2 and 47 percent for Option 3. When asked to pick a favorite, the differences were even more pronounced: 68 percent of team members favored Option 1, versus 26 percent for Option 2 and 5 percent for Option 3.

Option 1

"it's a simple, straightforward question. I also like that the question is objective."

84.2% found it useful

0% found it average

15.8% did not find it very useful

The first form we asked readers to comment on, “Option 1”, asked readers “Did you find what you were looking for?”, with “yes” and “no” check boxes, and contained a non-mandatory free text field; readers were only required to either the check boxes or the free text field, not both. This design was deployed in December on 0.3 percent of all Wikipedia articles, and the sample size was doubled on 4 January to 0.6 percent. Of the 19 respondents, 84.2 percent of them found it to be either very useful or merely useful; 15.8 percent considered it not particularly useful, or not useful at all. This marks the highest level of support for any of the 3 options. Respondents praised the simplicity and similarity of the design to feedback forms on other sites, noting that to follow "patterns that have been hardwired into web users is sensible".

Option 2

"more attractive than #1. I like the icons."

73.7% found it useful

15.8% found it average

10.5% did not find it very useful

The second form we tested, “Option 2”, clearly indicated what sort of feedback we were looking for by subdividing the form into “suggestion” “praise”, “problem” and “question”. A reader is required to select one of those tabs (“suggestion” is checked by default) and then enter text into the free text box. Of the responses we received, 73.7 percent found it useful to some extent, 10.5 percent did not find it very useful, while 15.8 percent considered it to be of average utility. This represents the lowest level of dislike for the design, but it was not, overall, considered to be as useful as Option 1. While respondents praised the attractiveness of the interface and the level of detail, others worried it would prove too complex and distracting for readers.

Option 3

"A combination of ratings and free text seems to me to be the most useful, as you can associate a rating with more data."

47.4% found it useful

31.6% found it average

21.1% did not find it very useful

The last form, “Option 3”, asked readers “Did you find what you were looking for?” and contained a non-mandatory text box. Like the existing system, it includes a five-star rating system, which meant we were comfortable with making the text box optional – even if readers only submit a rating, useful data can still be gathered. Of the responses, 47.4 percent found it useful - the lowest, by far, of all the forms - while 31.6 percent found it to be of average utility, and 21.1 percent considered it useless or not very useful. This represents the highest average and useless ratings of any design. Respondents were generally split over the utility of the five-star rating system; some considered it a useful additional metric, while others maintained that our experiences with versions 1-4 show that such a system does not work. It was also felt that using two different ways of gathering data is likely to confuse readers, and split the amount of feedback we get rather than adding to it.

Conclusions

Based on the quantitative data, Option 1 is considered the most useful by our staffers and core volunteers, with 84.2 percent considering it useful or very useful, compared to 73.7 percent for Option 2 and 47.4 percent for Option 3. This lines up with the results of our reader survey, which also found that Option 1 was the preferred form; similarly, for both surveys, Option 2 scored unimpressive, and Option 3 accumulated the largest amount of dislike. Qualitatively, respondents praised Option 1 for the simplicity and clarity of the design, which it was felt would make it easier for readers to use. The result is backed up by the responses from one question, which asked respondents which design was their favourite. 68.4 percent picked Option 1, compared to 26.3 percent for Option 2 and 5.3 percent for Option 3.

Metric	Option 1	Option 2	Option 3
Percentage who found it useful	84.2%	73.7%	47.4%
Percentage who found it to be of average utility	0%	15.8%	31.6%
Percentage who found it to be useless/not very useful	15.8%	10.5%	21.1%
Favourite form	68.4%	26.3%	5.3%

Appendix

repeated three times, once for each design

1. Is this feedback form useful for improving Wikipedia?
2. Why? what did you like most? least? How could this form be improved?