Research talk:Wikipedia + Politics

How many surveys will you need to be completed?

--EpochFail (talk) 20:38, 9 May 2017 (UTC)

We will need 100 responses. We're thinking of targeting 500 editors initially. --Porteclefs (talk) 17:55, 9 June 2017 (UTC)

How will you operationalize "team composition"?

--EpochFail (talk) 20:38, 9 May 2017 (UTC)

Team composition is all editors involved in the creation and ongoing maintenance of a Wikipedia page. Composition will simply be a basket of political alignments. --Porteclefs (talk) 17:56, 9 June 2017 (UTC)

General thoughts

I think that a lot of editors would like to vent about their frustrations in working with others who have strongly opposing political opinions and allow those opinions to affect their work. This should be an interesting study. --EpochFail (talk) 20:38, 9 May 2017 (UTC)

Thank you! --Porteclefs (talk) 17:56, 9 June 2017 (UTC)

Please draft an invitation to participate in the survey

I suggest creating Research:Wikipedia + Politics/Invitation and drafting the content of your invitation there. I'll help you make sure you have enough information for editors and that you avoid saying things that might be upsetting or confusing. :) --EpochFail (talk) 20:40, 9 May 2017 (UTC)

Here is the draft text of the invitation:

"Dear Wikipedia Editor,

My name is Eamon Duede and I am conducting research that aims to understand collaboration within teams of diverse individuals. Given the increased political polarization in the United States and elsewhere, we are particularly interested in how team members' political diversity affects collaboration and its outcomes.

I am asking you to participate in this study because you are a frequent editor of pages on Wikipedia that are of interest to us.

Using Wikipedia as a natural setting in which diverse editors co-create content, we would like to learn about your political preferences and your experiences in dealing with editors who share or don't share your preferences. Finally, we would like to examine whether your preferences and experiences affect, if at all, the editing of pages on Wikipedia.

This short survey should only take you about 5 minutes to complete and is designed to learn about the preferences and experiences of editors of English Wikipedia. We will use these data for research purposes only and will report them only in aggregate, never individually. Complete anonymity will be maintained.

More information about our project, as well as contact information and our institutional review board (IRB) authorization, may be found here: https://meta.wikimedia.org/wiki/Research:Wikipedia_%2B_Politics

If you have any other questions you or would like to participate in the research you can reach me at: eduede@uchicago.edu

Sincere thanks for your help! Eamon Duede" --Porteclefs (talk) 17:57, 9 June 2017 (UTC)

Hi Porteclefs! Thanks for sharing the message. I've copied it over to Research:Wikipedia + Politics/Invitation and made an an edit to trim it down to what I think a busy Wikipedian would prefer to read. Please review and put anything back that you think is critical. --EpochFail (talk) 13:21, 27 May 2017 (UTC)

Thanks! I like this. --Porteclefs (talk) 17:57, 9 June 2017 (UTC)

condifentiality

i don't recruit on talk pages as a general rule because then it's public data that a particular person was solicited to participate. --Andicat (talk) 15:38, 24 May 2017 (UTC) Andicat (talk) 15:38, 24 May 2017 (UTC)

Thank you! If not on talk pages, then where? --Porteclefs (talk) 17:57, 9 June 2017 (UTC)

sorry - not used to mobile interface! so anyway, you might think about recruiting people for whom you can use direct email either through wikipedia or because they are active on public lists if confidentiality is important. Andicat (talk) 15:40, 24 May 2017 (UTC)

I respectfully disagree with Andicat. We (the leftovers from R:Committee) generally recommend talk pages so that Wikipedians can more easily track the activities of researchers. If Wikipedians get upset about the talk page messages, that's a good indication that the posting is disruptive and we should re-consider whether the study should continue. --EpochFail (talk) 17:00, 5 June 2017 (UTC)

Hi, one thing that our research assistant just thought of is that, when posting a link on a talk page, it's difficult to be certain that the person filling out the survey is, indeed, the targeted respondent. What are your thoughts on this? Does it make sense to solicit participation but share the link privately upon assent? Or are we needlessly concerned here. Thanks! --Porteclefs (talk) 20:41, 9 June 2017 (UTC)

Sorry for the late reply. I'd just ask users to type their username into a box on the survey. If you get multiple submissions from users with the same username (I doubt it) then you can deal with the problem after the fact. Alternatively, I think it's OK to send an email to a user once they have agreed to participate in the survey. --EpochFail (talk) 18:00, 24 June 2017 (UTC)

Recruitment

I noticed Bill had tried to use talk pages to recruit, apparently with little luck. How is the recruitment progressing? What assumptions are being made about self-selection?

Rich Farmbrough 18:09 4 June 2017 (GMT).

Thanks for posting about this, Rich . I didn't know that the recruitment had been moving forward for this study. We still have open conversations about their recruitment message. I'm guessing that the researchers aren't watching this page or they might have noticed I had concerns before moving forward. User:MishaTeplitskiy and User:Bill10000, please respond here and engage with us. We have mostly minor concerns, but the concerns may become more substantial if you don't respond to us here. --EpochFail (talk) 18:07, 5 June 2017 (UTC)

We pilot tested on a dozen of random users, but we haven't started the recruitment yet. Our plan is to use the talk pages to recruit people as EpochFail recommends. Could you say more about your concerns, EpochFail? --Bill

Hi Bill10000, first, please consider logging in and signing your messages using four tildes (~~~~) so that we know we are not talking to some random troll editing through an IP and claiming to be you. :) But primarily, I made a draft of your invitation message that trims down a lot of the content that I think would be too much in a first message to a Wikipedia editor. See Research:Wikipedia_+_Politics/Invitation. Please review and make your own adjustments. As you can see, pilot testing will still get you attention, so it's important to have the documentation for the project settled before sending out the first invite. --EpochFail (talk) 14:18, 6 June 2017 (UTC)

Thanks for the edits. The invitation looks good to me. I would like to change one thing: we are sampling not only editors of political pages but also editors of science pages. The idea is to see if people who didn't edit our "political pages" are similar to those who did. Bill10000 (talk) 04:25, 11 June 2017 (UTC)

I just noticed that your answer to my question re. recruitment is a goal of 100 surveys, but on the project page, it currently says "We aim to survey a set of 200-300 people". I see that as far too many and your recruitment would be very disruptive. The most active editors in English Wikipedia are surveyed very often and they would not appreciate seeing so many survey invitations. With a response rate of ~15%, getting 300 surveys would require sending invitations to 2000 (!!!) people. --EpochFail (talk) 14:21, 6 June 2017 (UTC)

Hi --EpochFail, we were planning to target ~200-300 people. Not ~2000. Would posting to ~300 pages be considered disruptive? --Porteclefs (talk) 17:58, 9 June 2017 (UTC)

If you message 300 people, I think you'll likely get about ~45 responses (assuming 15% response rate). Will that be enough surveys? In previous discussion, you'd said 500 invitations with a goal of 100 completed surveys.

EpochFail Sorry for late response, was out of town for 2 weeks. Agree with all comments made so far. We'd like to target ~500 random users (not top editors) from a set of 10,000(?) or more pages. Hoping for a 20% response rate, which is what we got with our tiny pilot. Does that sound reasonable? And again, thanks for all this guidance. WikiResearcher (talk) 15:23, 12 June 2017 (UTC)

Generally, if you're messaging any active editor randomly, I don't think that 200-300 invitations is overwhelming. But if you end up targeting the most active editors, 200-300 postings is far too many. The effect of disruption grows with the activity level of the editor. So does the density of survey requests. Every other survey proposal I review is targeting the Top N editors by edit count and we generally prevent those surveys from happening at all because of how disruptive and frustrating they are for those editors. --EpochFail (talk) 18:01, 9 June 2017 (UTC)

I know that it appears that we are waffling on the targets -in truth, we're doing everything we can to be respectful not only of the conventions surrounding surveying but also of the community's time and patience. I sincerely appreciate the guidance! --Porteclefs (talk) 19:11, 9 June 2017 (UTC)

Another question concerning recruitment: would it be deemed inappropriate to use a headless browser, signed in as one of our team, to automatically post the solicitation to user talk pages under the condition that the script sleeps for several minutes between posts? --Porteclefs (talk) 20:44, 9 June 2017 (UTC)

That would not be acceptable unless you want to go through a time consuming process where you'd likely not find support. And honestly, I wouldn't recommend it. I'd recommend that you manually post invitations to participate -- no more than 25 per day. If you post all of the invitations in rapid succession that will certainly attract a lot of attention. With 25 invitations per day, you'll learn quickly how many invitations you'll eventually need to post based on the response rate. --EpochFail (talk) 21:23, 13 June 2017 (UTC)

Good question, Rich . Self-selection is a potential issue. We are also concerned that the people who respond will be those who are strongly biased. So we did a small pilot test, and it turned out that two people who don't have strong political preferences also responded. We will also randomize our recruitment as much as possible. Hopefully our sample of editors will not be biased. Any thoughts on dealing with the self-selection problem? Thank you. --Bill

Related paper

This may be already on the team's radar, but just in case: A 2013 paper ("Jointly They Edit ...", summarized here in the Wikimedia Research Newsletter) investigated a similar question. Regards, Tbayer (WMF) (talk) 23:24, 23 June 2017 (UTC)