Learning patterns/Asking the right questions
What problem does this solve?Edit
The most important step in performing a survey is figuring out what it is you want to learn. This sounds obvious, but many researchers realize after they have already deployed their survey that they failed to ask an important question. If you want to know how many people on different Wikimedia projects subscribe to public mailing lists, you will need to ask people which project they currently contribute to most right now. If you want to know how many people migrate from one project to another during their editing careers, you need to ask which project they joined first and which one they contribute to most right now.
What is the solution?Edit
To make sure you cover all of your bases, frame your survey around three or four types of question:
- background questions
- activity questions
- experience questions
- opinion questions
You don't need to have an equal number of questions in these four categories, and you might not need them all, but many surveys will be composed of a mix of all four types.
Questions about the participant themselves: who they are, where they're from, what they do off Wikipedia, etc.
- Teahouse is intended to be a friendly, supportive environment for all new Wikipedians, but we make a special effort to welcome female editors, who are underrepresented within the Wikipedia community. If you're comfortable telling us, what is your gender? (Teahouse Pilot project guest survey)
Questions about what they do now, or have done in the past.
- Have you ever sought help on any of the following topics on Wikipedia? (Help Project survey)
Questions about how they feel (or felt), or what they think, about the specific activities they participated in.
- Was there anything in particular you liked about the review process or the feedback? Tell us about it. (IEG submitter survey)
Questions about what they think in general about the subject you're interested in, beyond their own specific experiences.
- What ideas do you have to make dispute resolution overall more effective and efficient? (Dispute resolution survey)
When deciding which questions to ask, and how many, it is also helpful to keep these considerations in mind:
- Keep your survey as short as possible. People have short attention spans. If you ask them to fill out a 50 question survey, many of them will get bored or frustrated and quit partway through, leaving you with gaps in your data and lingering ill will. Don't ask every possible question; figure out which questions are necessary and only ask those. Realistically, keep the length of time it takes to fill out your survey (in the amount of detail you desire) under 10 minutes.
- Make multiple choice answers required and free text answers optional. No one likes being forced to write. Making basic questions multiple choice allows respondents to give you critical information quickly and easily. Making answering multiple choice questions required to finish the survey (and most online survey tools will allow this) gets you good data, as long as your questions are clear and the choices you provide make sense. Free response questions can be very useful, and most surveys will contain at least a few. If people have strong opinions about something, you can be sure that they will take advantage of this—no need to force them.
- Use opinion questions sparingly. Opinion questions are useful for some surveys, but in many cases it is better to ask primarily Experience questions and Activity instead--people's opinions don't always reflect their behaviors!
- Question sets from previous Wikimedia surveys.
- IEG Round 1 surveys for reviewers and proposal submitters
- Teahouse Pilot wrap up survey questions
- Dispute resolution project survey questions
- Help Project Fellowship survey questions and results
- Wikimedia Strategic Planning former administrators survey questions
- Wikimedia Strategic Planning former contributors survey questions
- 2011 Wikimedia Editors survey questionnaire (PDF)
- Dentaleconomics.com, The Art of Interviewing.