Learning patterns/Framing survey questions

A learning pattern forsurvey

Learning Pattern Library

Framing survey questions

problemIf people misunderstand your survey questions, they may not answer them the way you intended.

solutionWhen asking multiple-choice questions, make sure you provide choices for all the major types of answers you expect to receive, at the right level of specificity. Ask open-ended questions in a way that encourages people to elaborate, rather than give one word answers.

creator• Jtmorgan

created on27 August, 2013

What problem does this solve?

When people misunderstand survey questions, they often don't answer them the way the survey creator intended. By the time the survey creator realizes this, it's usually too late.

Most people fill out surveys quickly, and don't always read every question carefully. The survey creator usually has just one chance to ask these questions—most people are not willing to fill out the same survey twice! So it's important to ask the right questions, and to ask them in the right way.

The "right questions" and the "right way" may be different for each survey, but there are some good general principles that you can follow. Above all, try to make the survey as short as possible: that way, people are more likely to concentrate on giving reliable responses.

What's the solution?

When asking multiple-choice questions, provide choices for all of the major types of answers you expect to receive, at the right level of specificity. Ask open-ended questions in a way that encourages people to elaborate rather than give one-word answers.

Examples

Asking multiple-choice questions

Take a look at an example of a multiple-choice question, and think about how it could be better (don't type!). Then click on the hint and think again. Then click on the possible solution, and so on.

Example 1

How active are you on Wikipedia?

• 1–10 edits per month
• 11–20 edits per month
• 21–30 edits per month
• 31–40 edits per month
• 41–50 edits per month
• 51–60 edits per month
• More than 60 edits per month

• Perhaps the wording of "How active are you on Wikipedia?" could relate more clearly to the choices you give them.
• Will people have to think too hard about whether they make 30–40 or 41–50 edits per month? And will it really be useful for you to know this fine distinction?
• Can you identify where repeated words could be removed to make it simpler to read?

Possible solution
In a normal month, how many edits do you make on Wikipedia? 1–10 edits 11–50 edits 51–100 edits more than 100 edits

Comments

The new question directly relates to the wording in the options. The options are now shorter and easier to answer, and here we're assuming the four new ranges tell you more useful information about their edit-rates than the seven in the original. (You could adjust the ranges higher if you're surveying editors you believe are more active on average.) We've removed the repeated "per month", but retained the repeated "edits" for clarity. But we've forgotten an important option. Can you think what it is? And maybe we could put a note at the bottom to make it easier to relate their daily editing experience to whole months. What's a short, simple wording for that?

Better solution

In a normal month, how many edits do you make to Wikipedia? I've never edited Wikipedia 1–10 edits 11–50 edits 51–100 edits more than 100 edits Note: 100 edits is a little more than three per day, on average. But wait—there's one more thing. Let's not make people feel inadequate if they've never edited Wikipedia. For this purpose, we could soften the wording and do something else, too. Can you think of these two things?

Even better

In a normal month, how many edits do you make to Wikipedia? 1–10 edits 11–50 edits 51–100 edits more than 100 edits I don't edit Wikipedia regularly Note: 100 edits is a little more than three per day, on average. OK, this is better: the "no edits" option is now last, not first, so they won't feel they're on the low end of a scoresheet; and the wording is now softer because it refers to a larger range of potential personal roles to explain non-editing ("regularly").

Points to remember

Ask for an appropriate level of detail. Most people won't be able to accurately tell you whether they average 20 or 30 edits per month; and if you're just trying to get a sense of how active your participants are, you probably don't need that much detail. Asking for unnecessary detail can be confusing, distracting, and lead to survey fatigue. Use clear, brief wording. The opening example asked "how active are you?" and then specified edit counts. Relate the question closely to the options where it's easy to do so, and remove unnecessary repetition in the options. (And let's not give them the idea that edit counts are the only measure of activity on Wikipedia!) Capture important exceptions. What if a participant doesn't edit Wikipedia at all? Give them an option. But if you know that everyone edits, this option could be omitted.

Now have a rest and come back for the next set of examples. We need you to be fresh!

Asking open-ended questions

Open-ended questions can draw out more detailed, personal responses from people in their own words. But people may skip over these because they don't have anything to add or they don't feel like writing a lot.

But you can increase both the number and the quality of their open-ended responses by phrasing the questions in ways that encourage storytelling.

Again, let's take an example and think how it could be improved:

Example 2

Has the kind of editing you do changed since you first joined Wikipedia?

Hint
This is yes–no question, so strictly speaking they could answer with one word. That wouldn't be useful to you, and if it's all you want to know, make it a multiple-choice question. By adding just one word to the question, you're likely to get more detailed responses. What word?

Possible solution
How has the kind of editing you do changed since you first joined Wikipedia? Even so, it's still a hard question, and if coffee and cookies are around the corner after a seminar, they may skip over this with a short non-useful response. Let's think of what you could add to the question to encourage them to give more.

Even better

How has the kind of editing you do changed since you first joined Wikipedia? There are many possible areas. Just a few might be: article creation, improving articles, new topics, adding references, adding images, how you participate on talkpages. But tell us about areas that relate to your experience. Giving examples like this does bring the danger of funneling them down the pathway of your suggestions. So think carefully before you provide examples—often it's better to give no examples at all.

Example 3

What's the most difficult thing about editing Wikipedia?

This is asking people to come up with something quite abstract; but people generally find it easier to remember concrete things than abstract things, and specific examples can produce especially useful data for you.

And does "difficult" mean a negative experience, or one that was challenging but ultimately positive? Asking someone how they were affected by an experience (how they felt) will usually lead to more accurate and useful feedback.

So, think how you might change the question to make it easy for them to recall something specific in their experience? We've come up with three ways, and our possible solution is only three words longer, too!

Possible solution

Tell us about a recent editing experience where you felt frustrated. Here, we've gone for three qualities that are more likely to be easily recalled: a specific experience one that was recent a question that is expressed in terms of immediate feelings at the time ("frustrated") There's a technique to get specific abstract responses to open-ended quesitons, but we recommend that you use it only once in a survey, if at all:

Another technique
Briefly, what are the two best things and the two worst things about editing Wikipedia? Two best: Two worst: This can be reduced to just the best or the worst, depending on what data you want. It normally won't get you the how and the why.

Points to remember

Avoid the yes–no trap unless it's a multiple-choice question. The triggers to getting useful information are How", "Why", "When", "Where", and "Which". With the exception of "how", these are known as wh-questions, as opposed to yes–no questions. Ask for concrete examples. Concrete rather than abstract is often easier, although there are ways of getting abstract information; but don't push your people too hard or too often on the abstract. Ask for their experience, not their opinions. In most cases, personal experiences are more valuable survey results than vague impressions, gut reactions, or unsubstantiated opinions. The recent is often easier for them.

Endorsements

• Trevj
• Libcub
• Humetheresa
• Titodutta

See also

Related patterns

• Asking the right questions
• Surveys at different points
• Who to survey

Other resources

_{other resources (links to surveys, articles about survey design, handy tools) related to this pattern}

• Survey Questions 101: Do you make any of these 7 question writing mistakes? (Qualtrics.com)
• How to ask survey questions (Wikihow.com)
• Guide to writing survey questions (Minnesota State University)