Community Liaisons/Product Surveys

See mw:Community Tech team/All Our Ideas for further work, using the results from this survey.

The Product Roadmap survey intends to offer a lightweight, data-informed way for communities across the entire Wikimedia movement to contribute ideas the Wikimedia Foundation's (WMF) product roadmap.

The question we are currently wanting to answer is: If the communities were to collectively decide on a gadget or tool which could be expanded into an extension for use across all Wikimedia-supported projects, what would it be?

Background

With limited resources, the WMF needs to take a data-informed approach to prioritizing feature requests across the movement. There have been multiple lists and requests in the past, many of which are contained within individual communities and projects.

In order to effectively prioritize community feature and product requests, we first need a way to gather requests across the movement. After reviewing survey systems which fit a certain criteria, we've decided to run a pilot survey with AllOurIdeas with the English and Spanish-language Wikipedia communities, though of course, anyone seeing this is welcome to participate.

What this would be

• an attempt to bring more community influence into the Foundation’s product roadmap by ranking submitted ideas
• a means to encourage a broader range of users into telling us what their needs are
• a means of including data directly in the process

What this would not be

• a replacement for any products currently in development
• a promise that all of the ideas submitted will be built

All Our Ideas Tool Survey

Summary

The All Our Ideas Tool Survey was a pilot product survey launched in December 2014 which ran for two weeks. It was targeted to active contributors on the English and Spanish Wikipedia projects via watchlist notifications. The survey asked community members "Which tool or gadget would you want the WMF to improve to make contributing to the projects easier?" The results can be seen at the following URLs:

• http://www.allourideas.org/wikimediagadgets/results (English)
• http://www.allourideas.org/wikimediaaccesorios/results?locale=es (Spanish)

Work related to the results of the survey is being tracked at Community Tech team/All Our Ideas.

Planning

Quarter 2 testing phase

In December 2014, the Foundation piloted a two-week test to users in two communities. The general goals of testing this were to check on:

• Ensuring proper language support
• Checking on success of outreach tools (watchlist notifications, village pump notifications, etc).
• Early trends to compare across entire movement (are our communities all voting for similar things? Do some communities have vastly different needs?)
• Checking on the possibility of abuse or system gaming

After this, we reviewed to see if this is a helpful and accurate way of gathering ideas.

Update:

• Results: The most popular items were all related to automatic formatting of bibliographic citations. This is being developed in mw:Citoid, which will be available in VisualEditor during the coming months, and later (after its stability and scalability has been demonstrated) in the wikitext editor. Dozens of other ideas were considered and received varying levels of support. In general, the more popular and familiar the idea to users of the English Wikipedia, the more support it received.
• Technical limitations: Significant technical limitations in the system meant that descriptions were limited to 140 characters, without any external links. This meant that users were sometimes presented with choices that they did not recognize and could not easily get more information about.
• Style: AllOurIdeas uses w:Pairwise comparison, in which voters responses to individual pairs is used to produce a ranked list. By statistically combining the all voters' ranked preferences, it determines the most popular items across the entire population of voters. This is a validated method of research that meets some important principles of voting theory, such as the strong favorite-betrayal criterion (it should never be in your best interest to rank your first choice as anything other than first) and the later-no-harm principle (if you ranked your favorite as highest, then ranking something else as second should not hurt your favorite). Pairwise voting also reduces intentional manipulation of the outcome by canvassing for support of a particular item: Pairs are presented randomly, and the canvassed item might not be an option offered to any given voter.
  However, some participants strongly disliked pairwise comparison. They felt like they never completed the task. The main suggested alternative was showing all ideas in a long list, and asking users to either rank each idea manually, or to vote for and against each idea. These methods are familiar and give voters a definite feeling that they have finished voting. However, they do not scale easily to many dozens or hundreds of items, and they are more susceptible to strategic voting and other forms of gaming.
• Collecting ideas: AllOurIdeas allows participants to suggest new ideas until the very end. Because the absolute number of votes is irrelevant, even ideas submitted on the last day could have a ranking estimated from just a few votes—indeed, a late-submitted idea could win.
  However, some early participants disliked this, because they personally didn't vote on the ideas that were submitted late. Because commenters were unfamiliar with pairwise comparisons, some of them also worried that having fewer people vote on ideas that were submitted late would reduce the likelihood that these items could win. The main proposed alternative was collecting all ideas in advance and refusing to permit any ideas to be submitted after the deadline. The main drawback is that this rejects the input of people who learned about the vote after the early deadline for submissions, regardless of the quality of their ideas, which is not consistent with some of our communities' cultural values.

Quarter 3 Survey

Should this prove effective, the Foundation will launch a broad scale to ask all communities for gadget or tool expansion ideas.

FAQ

How many times should I click? How does it work in practice? Can I take it multiple times to make my favorite option "win"?

Click as many or as few possible pairs as you want. There is no minimum or maximum required number, so you can stop at any time. There are several options, and as you suggest applicable ideas (existing gadget and tools), they will be added to the options. You can read more about AllOurIdeas here.

Why is this called a pilot?

This is a test survey working with English and Spanish language Wikipedia communities. The intention behind this pilot is to see if the system that the Foundation is testing will scale to as many Wikimedia-supported communities as possible.

What will the WMF do with its results?

The Product Department at the Wikimedia Foundation will be taking the top ranking option/s into consideration when planning the roadmap in upcoming quarters. Using a data-informed approach allows the Community Engagement (Product) (CE(P)) to scale the decision with as many communities as possible.

Why did you choose English and Spanish to run this survey?

Launching the pilot in two languages tests the difficulty of translations for this particular survey. As the Community Engagement team at the WMF has no native Spanish speakers and es.wp is a significantly large community, we wanted the challenge in handling translations with this external system before expanding to more communities in Q3.

Can I suggest a tool/gadget/etc which is not in the list? Are there any limitations?

Please do! Keep in mind that only currently existing gadgets will be added to the list (if you do not know the name of the gadget or tool specifically, try to describe it and the team will re-write it as the name of the tool with a simple explanation). Please try to post a simple description of the tool in non-jargon language so that others may easily understand what it is.

What happens to ideas that do not get entered into the list?

The Community Liaisons are maintaining a list of all tools and their status within the survey, with explanation if needed.

Legal Notification

Your privacy is important to us. Please note that this survey is hosted by AllOurIdeas, a third-party service – please read the AllOurIdeas privacy policy to find out more about their privacy practices. By answering these questions, you permit us to record your responses and agree to donate them to the public domain. This allows us to freely share your answers with others for the purpose of open analysis, research, and study. We will not share publicly your personal information, like your email address, except as permitted by the Wikimedia privacy policy. This commitment assumes you do not incorporate your personal information in response to a question that doesn't ask for it. By answering our questions, you consent to the transfer of your responses to the United States and other places as may be necessary to carry out the objectives of this project and you permit us to record and indefinitely retain your responses for research purposes.

In order to bring your ideas to life, if you submit ideas or proposals to us in this survey, you grant to the Wikimedia Foundation a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable license to any right, title, and interest that you may have in any and all patentable subject matter and derivatives thereof described or conceived of in the survey, including patentable subject matter later derived from the survey.

To go back to the survey, click here.