Research:Wikipedia Editors Survey 2012

Contact

Wikimedia Foundation

This page documents a completed research project.

The Wikipedia Editor Survey 2012 was a survey conducted by the Wikimedia Foundation, the non-profit foundation that runs Wikipedia and its sister projects, to take the pulse of the community and monitor pressing issues or concerns. This was the third survey of Wikipedia editors as envisaged in the 2010-15 strategic plan, after the April 2011 and December 2011 surveys.

The questionnaire is available here (including translations). The survey has now concluded.

General questions about the survey

What are the objectives of this survey?

The first main purpose of this survey is to continue the work of the 2011 studies (conducted by Mani Pande and Ayush Khanna), with a focus on identifying trends rather than exploring new topics:

Understand how editor experiences have changed since the last survey
Gather data about any changes in the editor demographic
Understand editors' perception of the community (and how it has evolved) using metrics such as the Wikipedia Editor Satisfaction Index

The second emphasis in this instance of the survey is to measure the satisfaction of the editing community with the work of the Foundation, and help us to better understand how we can support the Wikimedia community.

Which Wikimedia projects does the survey cover?

Most of the questions are specific to Wikipedia projects, as in the previous two surveys, but this time we aim to run a part of the survey (apart from the introductory part, the general demographics questions and those about satisfaction with the Foundation) on Commons as well. The questionnaire was originally written in English, and translated into other languages; the translations were reviewed by community members.

When, and how often will the survey be conducted?

The survey ran from late October until November 2012, for varying amounts of time on each language project.

How can I take the survey?

Every registered user (editor) who during the survey visits Wikipedia or Commons in one of the available languages while logged into their account (and has not opted out of banners via their user preferences or other technical means) should see a banner notification once with a link to participate in the survey (which is hosted off-site, on wikimedia.qualtrics.com). It is also possible to provide a custom link that logged-in users can use to take the survey, if they miss this one-time notification: to request it, send a wiki mail to user "Editor survey" on your local project (type "Special:EmailUser/Editor_survey" in the search box).

How will the survey data be handled? What about privacy?

The data will be anonymized before conducting analysis to ensure that individual responses cannot be associated with specific respondents. The data will be analyzed collectively and responses will not be associated with any one respondent.

Will the report from the editor survey be shared with the community?

The report from the survey will be shared publicly, and placed into the public domain (Creative Commons Zero).

How can I provide feedback?

You can provide feedback about the survey on the talk page.

In which languages is the survey available?

Apart from English, the questionnaire has been translated into Arabic, ‎German, ‎Spanish, ‎French, ‎Hebrew, ‎Indonesian, Italian, ‎Japanese, ‎Dutch, ‎Polish, ‎Portuguese, ‎Russian, ‎Swedish, ‎Ukrainian and ‎Chinese (simplified and traditional). While taking the survey, respondents can change the language using a menu.

Survey design

Who designed the survey, and how?

The first part of the survey (and the demographics part at the end) is a revised and significantly shortened version of the 2011 Editor surveys with very few new questions, designed by Wikimedia Foundation staff and contractors, mainly from the Global Development department. Several staff members, including active editors, provided input for the original design of this part of the survey, and for its current revision.
The second (satisfaction) part was likewise designed by Wikimedia Foundation staff and contractors, shepherded by the LCA (Legal and Community Advocacy) Department. The statistical design for several of these questions was done by Elizabeth MH.
This was followed by a feedback phase, facilitated by Christine Moellenberndt, where input from both the global editing community and the wider WMF staff was sought (e.g., via mailing-lists, e-mail and talk page messages and in an IRC Office hour). Several changes were made based on the feedback received during this phase.
Research contractor Beckie Neumann is carrying out the technical implementation of the survey (as in the December 2011 survey) and was involved in the design phase as well. The main contact for this survey is Tilman Bayer.

How can I help design or improve the survey?

You can provide feedback on the talk page.

How is this survey different from the 2011 surveys?

In the first part of the survey, about 90% of the questions continue from December 2011, since we would like to conduct a longitudinal analysis, and look for any shifts since 2011. The "satisfaction" part is entirely new.

Survey implementation

What software will be used? Will it be hosted by the Wikimedia Foundation?

The survey will be conducted using Qualtrics, a proprietary service that was chosen among various others, including Lime Survey, an open-source tool that was used for the first Wikipedia Editor Survey, but was found to have critical security flaws.

Will the referring page be tracked?

It will be recorded whether respondents reached the survey via a banner on Commons or on Wikipedia — to avoid asking the Wikipedia-specific part of the questionnaire from Commons users —, and from which Wikipedia project. There will be no information stored about the specific article or page where the user saw the banner with the invitation link.

Will unfinished surveys be saved? Will the answers be counted?

Yes, unfinished surveys will be saved, so people may resume them later. We will compare unfinished responses with the finished ones after the survey is taken off-line. However, the final analysis will only include complete responses.

Can the respondent "skip" questions?

Based on feedback from the community, we have decided to make all questions nonmandatory — with the exception of very few numerical questions like the "$100" question where the technical limitations of the survey software do not allow to check the consistency of the input while at the same time offering to skip the question. Respondents can simply leave out the corresponding replies, and proceed to the next page of the survey by clicking the ">>" button on the end of the page.

Why will a user see the link to the survey only once? How?

This will ensure that all users (editors) have an equal probability of participating in the survey, and the survey is not biased towards those editors who edit more frequently. The Foundation is interested in responses from all types of editors. We'll set a cookie within the CentralNotice banner for the survey to guarantee that it will only show up once per user.

In case you miss the banner, you may email us, and we can generate a custom survey link for you (see #How can I take the survey?).

Privacy & transparency

Is the survey covered by the Wikimedia Foundation's privacy policy?

In collaboration with the WMF legal team, the following privacy policy has been drafted for the survey:
- By answering these questions, you permit us to record your responses and agree to donate them to the public domain. This allows us to freely share your answers with others for the purpose of open analysis, research, and study. This survey does not require you to provide any personally identifiable information such as your name, address, phone number, or e-mail. (Please do not incorporate your personal information in any response to a question.)

Who will have access to the raw data? Will the raw "anonymized" results be shared publicly?

Researchers working at the Foundation will have access to the raw data. We will share data with other researchers for analysis, but we will take steps to account for outliers where identity can be revealed. For example, if only one woman editor from Macedonia takes the survey, we will anonymize the country data for her response before it's shared, to protect her identity.

To learn more about the process used to anonymize editor data and protect their identities in the previous instances of this survey, see here for the April 2011 Editors Survey and here for the December 2011 Editors Survey.

How long will the data be kept?

Since we will use data from the survey to perform longitudinal analysis, we will keep the data for about 2-3 years.

What do you mean by "anonymized"?

Individual responses will not be associated with any user.

Results

Topline report for the first half of the survey (editing experience), PDF
Topline report for the second half (satisfaction with WMF), including some additional analysis, PDF
Blog post on gender ratios in general, containing gender data from this survey and a trend comparison with the preceding 2011 survey
Blog post about the 2012 survey presenting some results like the average time editors spend contributing each week
WMF Priorities (free-form response analysis), PDF
Motivation to contribute (free-form response analysis), PDF
Some technical notes about the survey process and data
Other material on Wikimedia Commons (including presentation slides and individual charts)
Anonymized response data which everyone can use to do their own analyses

Gallery of some example results
Histogram of responses to the question "How much time have you spent contributing to Wikipedia during the last seven days (approximately)?"
Histogram (population pyramid) of respondents' ages
"If you had to name one thing that motivates you most to contribute to Wikipedia, what would it be?"
"All in all, do you think the Foundation is generally headed in the right direction, or is it off on the wrong track?"
"How much do you agree with the following statements? ... The Foundation and its staff are part of the Wikimedia community."