Research talk:Surveys on the gender of editors

Selection bias

The design of this survey does not allow to correct for selection bias. From the Hill & Shaw paper: «the fact that the WMF/UNU-MERIT survey includes data on Wikipedia readers allows us to take advantage of demographic data from a nationally representative phone survey of US adults». Please do the same, otherwise this study will be once again useless and we'll be stuck with 2010 data. Nemo 23:46, 18 July 2019 (UTC)

Just in case others have the same question, I followed up here: https://phabricator.wikimedia.org/T201707#5338885 --Isaac (WMF) (talk) 14:01, 7 August 2019 (UTC)

US adults are not all Wikipedians, even on the English language Wikipedia. However you could use this methodology as a benchmark against the US part of the response to see how that skews, but and here lieth the problem, we suspect that Wikipedians are a skewed subset of humanity, weighting our sample back to the US general population doesn't help us, it just masks the skews in the community. WereSpielChequers (talk) 13:56, 14 August 2019 (UTC)

Opt me out

I've been selected for this survey. I don't want to answer it. I don't even want to interact with it. There's no collapse bar and no way do get rid of the survey. How long should I expect the survey to be around? Schetm (talk) 20:22, 5 August 2019 (UTC)

@Schetm: Thanks for your patience and apologies that you find it disruptive. The intent was that you could select "Prefer not to say" if you did not want to provide any information, but I understand that desire not to send any information. While far from perfect, you have two options: you can 1) execute mw.storage.set('ext-quicksurvey-editor-gender-1-en', '~') in your browser's javascript console (on Mozilla, I open the hamburger menu, go to Web Developer, and then Web Console), or, 2) turn on Do Not Track. Either method will remove the survey without sending any information to the servers. We cannot do it from our end because the survey is designed specifically to not be tied to any specific user accounts for privacy reasons. Otherwise the survey will need to run for at least a week to collect sufficient responses. --Isaac (WMF) (talk) 02:27, 6 August 2019 (UTC)

Thanks for your speedy response! I really do appreciate it. Schetm (talk) 03:05, 6 August 2019 (UTC)

Yes, very offensive, and "prefer not to say" is quite different from "I want nothing do with this". My reaction is here. Well done on finding a quick way to drive editors away. --Bonadea (talk) 21:16, 9 August 2019 (UTC)

Yeah, I think it would have been better worded as "prefer not to respond" which, as Bonadea says, is a different response than "prefer not to say." Schetm (talk) 21:31, 17 August 2019 (UTC)

Offensive

This is extremely offensive and contrary to what our project is supposed to be doing. We are supposed to be providing information, not collecting personal information. Ched (talk) 21:45, 5 August 2019 (UTC)

@Ched: Thanks for raising your concerns. A few thoughts though I understand I likely cannot fully address them: 1) while we are collecting information, this data is not linked to individual user accounts. We are specifically testing out this approach to surveying because it does not require us to store any information about who receives/takes the survey but we can still reduce the likelihood of re-surveying the same people in future efforts. The gender statistics from this survey will be aggregated and published to give us insight into minimally invasive ways to understand the impact of edit-a-thons or projects such as the Teahouse on the gender gap (or potentially other ways in which the community is diverse) on Wikipedia. The data will also help us understand how to contextualize the data from efforts like the Community Insights surveys, to see whether it is or is not biased towards certain populations. --Isaac (WMF) (talk) 13:59, 7 August 2019 (UTC)

Thank you for taking the time to respond. I won't bother you with the entirety of my thoughts at this time, but TY for at least acknowledging my post.Ched (talk) 17:29, 7 August 2019 (UTC)

Does

suffixing ?quicksurvey=internal-survey-editor-gender-1-en work for non-chosen editors? Winged Blades of Godric (talk) 16:03, 8 August 2019 (UTC)

Yes. And to emphasize for others who might read this, editors are not chosen. Whether an editor sees the survey or not depends on a random token created by their browser, so we made no choices about who would respond beyond setting the sampling rate. This means that if you use multiple browsers, there is a chance you'll see the survey multiple times. --Isaac (WMF) (talk) 06:29, 13 August 2019 (UTC)

Potential flaw that may invalidate this survey

Over at the enWP Village Pump (misc), we just had a question from an editor who received this survey more than once. If this can occur, then any data compiled is potentially flawed. And if the data is flawed... so will be any analysis or conclusions based on that data. I think you need to fix this bug and start over. Sorry to be the bearer of bad news. Blueboar (talk) 20:39, 8 August 2019 (UTC)

To close this loop, this comment was replied to by RoySmith on VP. --Isaac (WMF) (talk) 06:25, 13 August 2019 (UTC)