Survey best practices

See also: Surveys

This page appears not to have been actively updated for some time, and may not represent an international perspective.
Please take care to research good practices, and add them to this page!

Chapters, the Wikimedia Foundation and other organisations have conducted surveys of Wikipedia editors. This page collects best practices for those conducting surveys. These best practices are taken from governments and international organisations.

Demographic information

To aid with systemic bias and to understand the needs of Wikimedia contributors, some surveys have asked demographic information questions, on areas like age, gender, marital status, sexual orientation and household income.

Age

If considered necessary, use the question: What is your age?

The UK Office of National Statistics provides four general use age bands:

Harmonised Standard 1	Harmonised Standard 2	Harmonised Standard 3	Harmonised Standard 4
0-24 25-64 65-74 75+	0-24 25-44 45-64 65-74 75+	0-15 16-24 25-34 35-44 45-54 55-64 65-74 75-84 85+	0-4 5-10 11-15 16-19 20-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 75-79 80-84 85+

Use of narrower age bands with small sample sizes is not recommended as it has the potential for personal identification.

Sexual orientation

If considered necessary, use the question: Which of the following terms best describes your sexual orientation?

Provide as possible answers:

• Heterosexual
• Gay/lesbian (or "Homosexual")
• Bisexual
• Other
• Prefer not to say

Sex/Gender identity

First approach

If considered necessary, use the following questions:

What is your gender?

• Male
• Female
• Other (please specify)

Are you transgender?

• Yes
• No
• Other (please specify)

If you are transgender, what was your (assigned) gender status at birth?

• Male
• Female
• Not applicable
• Other please specify

Taken from Gay and Gender research website survey (Social and Political attitudes of LBTQ individuals)

Second approach

The ONS 'Trans Data Position Paper' (Released in 2011)' suggests some alternative questions

Is your gender identity the same as the gender you were assigned at birth?

• Yes
• No

Do you live and work full time in the gender role opposite to that assigned at birth?

• Yes
• No

(With an additional suggested follow up question)

Do you sometimes adopt the gender role opposite to that assigned at birth?

• Yes
• No

The briefing paper clarifies under no circumstance should the data ever be attributable to an individual (page 16) - so these questions would have to be anonymous, and notes that at the time, the ONS decided that data collection of gender or transgender status was not be included in standard household surveys. The second set of questions suggested are similar to those recommended by Unison

Third approach

The recommendation of the Equality and Human Rights commission in this regard are quite nuanced. These are undoubtedly the most thorough questions about gender, and were developed ...with a sample of the general public who identified with a range of gender identities and with the explicit aim of replacing the question 'Are you male/female?' on UK monitoring forms, national surveys or census.

Education

In the UK the ONS measures levels of qualification in the population between 16 - 64 years of age. This measure doesn't take into account type of degree, and therefore for example doesn't measure the difference between undergraduate, postgraduate and doctoral degrees

The levels of qualification against which data is collected are:

• Degree or equivalent
• Higher education
• A Level or equivalent
• GCSEs grades A*-C or equivalent
• Other qualifications
• No qualification
• Don't know

Household Composition

The Office of National Statistics defines 'Family' as 'A married, civil partnered or cohabiting couple with or without children, or a lone parent with at least one child. Children may be dependent or non-dependent.' and a household as 'A person living alone, or a group of people living at the same address who have the address as their only or main residence and either share one main meal a day or share living accommodation (or both).'

The latest data can be found here

Families

The Labour Force Survey defines family by the following criteria:

• Married Couple family
• Civil partner Couple family
• Opposite sex cohabiting couple family
• Same sex cohabiting couple family
• Lone parent family

Against which criteria respondents are asked to specify if they have One child, Two children and Three or more children.

Households

Are defined by single residences and number of occupants as follows:

• One person
• Two people
• Three people
• Four people
• Five people
• Six or more people

Tools

Online survey tools run by third parties may have different privacy policies, and these policies may get in conflict with Wikimedia users privacy expectations. The initial list of tools was defined at Phab:T94807.

Recommended services

• Qualtrics has been vetted by the Wikimedia Foundation Legal and Contracts teams.
• AllOurIdeas has been vetted by WMF Legal.
• LimeSurvey may be a more ethical possibility than Qualtrics (Free/Libre software)

Not recommended but allowed

• Google Forms is sometimes used for events feedback and by the community because it's very easy to set up, sometimes widely advertised e.g. id.wikipedia.sitenotice. Google's privacy policy is not consistent with Wikimedia's since they collect much more information from users. That being said, if we clearly indicate to users that the survey they are taking will be on Google and thus subject to the Google privacy policy (preferably before they click on the link to the survey), you can use it to administer the services.

Not evaluated

• SurveyMonkey has some privacy policy, as well as certifications and EU branches for EU users plus special articles for Canada, Japan, Australia and Brasil users.

Surveys on wiki

• It is possible to post questions in a survey format on a wiki page, just like any other discussion. This works best if you ask only a few questions, and if the questions do not touch sensitive subjects (e.g., demographic information or contentious subjects). However, you should be aware that on-wiki surveys will attract a different audience (e.g., fewer inexperienced editors and fewer mobile users, because they struggle with talk pages) and may result in fewer responses. On-wiki surveys also may not be a popular option: In 2015, one eight-question survey was offered in both Qualtrics and on wiki, so that editors could choose their preferred format. The result was that 508 editors chose Qualtrics, and only two responded on wiki.