Grants talk:IdeaLab/Survey women who don't contribute

Interesting, but it won't bring me back

As a woman, I am interested in this discussion/grant/idea.

I no longer post on the English Wikipedia because of several unpleasant run-ins with other writers arbitrarily undoing hours of work because they did not understand what I was doing. At the end, this happened twice in a row. I doubt, however, that the problem was gender-based. It was just plain rude behavior that I do not have to tolerate and chose not to endure any longer. No, I will no longer post on en:Wikipedia, even though I had been a contributor for many years.

The penultimate antagonist had the grace to apologize to me eventually and ask me to redo my work once he understood what I was doing, but I did not have the heart to redo it. I had been adding Hymenoptera articles and categories, which he started rolling back without talking to me before he did it. He treated me as if I were a very young errant child, scolded me, and called me on the carpet to tell him what I had done wrong. Once I explained what I was doing, he backed off and apologized, and admitted I had more domain knowledge than he had. It was humiliating and infuriating.
Interacting with the last one, who had been banned on at least one occasion (but I believe more), was the last straw. He was, quite simply, a petty, insecure, argumentative tyrant with very poor "people skills" and with a history of undoing anyone else's work on his articles.
Contributing to Wikipedia is not worth dealing with that type of behavior. No matter what policies you enact, nothing will entice me back into that environment with the kind of people who (over-)react without first trying to voice their concerns, or engage in honest conversation between peers. Not after those back-to-back experiences.

It is much more pleasant on gd:wikipedia (Uicipeid) and gd:Wictionary (Uiclair). People are courteous. If anyone has concerns, they raise the issue and a conversation ensues. Once a understanding and agreement have been achieved, action is taken. Standards are very high, but feelings are not ruffled. There is a civilized atmosphere of cooperation, respect and camaraderie which is sadly lacking on en:Wikipedia.

I think that a great deal of the difference is that there are so many contributors on en:Wikipedia that the level of anonymity is high and some people feel that gives them the right to throw temper tantrums. There are so few contributors on gd:wikipedia and gd:Wictionary that people know one another. Kibi78704 (talk) 11:19, 6 March 2015 (UTC)

Thanks for sharing your story, Kibi78704. What a frustrating, discouraging experience :( If nothing else, stories like yours teach us that women are frustrated to the point of leaving and never coming back, and that the only way to fix that is fix the root causes so as not to chase editors off in the first place. Your comments about cooperation, respect and camaraderie are also valuable. It makes me wonder what be done to encourage those traits in larger communities. Surely work has been done on this in other fields (social psychology, organisational behaviour, etc) that we could learn from? --Skud (WMF) (talk) 00:20, 16 March 2015 (UTC)

About ex contributors

I think we do have a way to get these answers in our community: if we search our user database for females (indicated either by preferences or userbox), who haven't contributed since <1 year?> and have more than X contributions, each of them has invariably a related email, right? --Langus-TxT (talk) 03:50, 13 March 2015 (UTC)

Yes, but do we have access to that email address, and should it be used for this purpose? I think it would probably be appropriate to drop a note on their talk page saying "hey, we see you haven't edited since X date, we're surveying ex-editors, could you answer these questions?" but whether or not they see it will depend on whether they have set their notifications to receive an email for each talk page message, and of course whether they still use that email address! --Skud (WMF) (talk) 00:14, 16 March 2015 (UTC)

Hi Skud (WMF), I think this is a great idea, and it's something I've wanted to do as well! Perhaps finding ex-contributors via snowball sampling would be best. People in the Wikipedia communities *know* editors who self-identify as women and who have left. It's more of a matter of whether these ex-contributors would like to share their time and stories. You may want to consider 1) how you will protect their privacy 2) what you will give back to them via findings. If you need help, please ping me. :) --Mssemantics (talk) 21:48, 25 March 2015 (UTC)

Not a promising research

Not a promising research approach imo. The vast majority of people who could edit WP don't. Without a very large sample covering both genders I don't see that very useful conclusions can be drawn. Johnbod (talk) 16:04, 20 March 2015 (UTC) -- Moved from Idea page by Langus-TxT

Eligibility confirmed, Inspire Campaign

This Inspire Grant proposal is under review!

We've confirmed your proposal is eligible for the Inspire Campaign review. Please feel free to ask questions and make changes to this proposal as discussions continue during this community comments period.

The committee's formal review begins on 6 April 2015, and grants will be announced at the end of April. See the schedule for more details.

Questions? Contact us at grants(at)wikimedia.org.

Related research and sampling suggestion

Hi Skud! I think your proposal is relevant and very doable. There are a couple of pieces of related research that I think you would find it helpful to draw on when you design your survey questions. The first is the Wikimedia Former Contributors Survey, which was undertaken in 2010 (see here and here). We're definitely due for updated data around former female contributors :) The second study is Collier and Bear's Conflict, criticism, or confidence, which re-analyzes the UNU-MERIT data and comes up with some compelling new findings. The third related research study is the paper Readers are not free riders, which includes results from additional surveys/interviews with readers about their perceptions of Wikipedia.

A couple of additional questions:

1. do you plan to send this survey out to non-English speakers? In my view, the more languages/projects represented, the more impactful this study will be. However, if you do plan to do a multi-lingual survey, you will need to budget time (and possibly money) for translation.
2. Have you considered the possibly of soliciting respondents via a CentralNotice banner, like we did for the Inspire campaign, but focused on logged-out readers rather than logged-in editors? This would increase your reach significantly, and would help reach people throughout the world and across projects.

Best, Jmorgan (WMF) (talk) 16:54, 22 April 2015 (UTC)

To add: the paper by Collier and Bear is paywalled, but we had a summary in the research newsletter here: "Gender gap connected to conflict aversion and lower confidence among women", and the paper's authors seem to have made it available in full elsewhere. Regards, Tbayer (WMF) (talk) 06:21, 25 April 2015 (UTC)

There are a couple accessible PDFs of the paper floating around... Jmorgan (WMF) (talk) 17:12, 27 April 2015 (UTC)

Thanks for your feedback, and the links to other useful research, @Jmorgan (WMF):! In answer to your questions:

Firstly, I don't think I'm able to take on a multilingual survey, especially if I'm hoping to ask long-form questions, as understanding the answers would be very difficult. Just for reasons of practicality and what I think is within my capabilities, I would prefer to just do English language.

With regard to banner advertising: it sounds like a good idea but frankly I can't even begin to imagine how you'd word such a banner, given that (AFAIK) there would be no way to separate anonymous editors from non-editors. Or is there? If there is I'd be much more into the idea. Then again, maybe catching a small percentage of people who aren't in the target audience, and having to filter them out, is not such a bother? (Edited to add: and of course (duh!) there's no way to only show the banner to women, which I suspect would mean having to filter out a *lot* of inappropriate responses. Not sure if this would be a win overall, tbh. :-/)

On another note: talking to Siko (WMF) today, she suggested I might approach you to act as an advisor to this project. Would you be up for reviewing my survey design and so forth? I could really use some help from someone with more formal research experience.

--Skud (talk) 10:26, 28 April 2015 (UTC)

Hi Skud. Yes, I'd be happy to help out any way I can. I added myself as an 'advisor'. Jmorgan (WMF) (talk) 17:47, 28 April 2015 (UTC)

Questions

Hi Skud, Thanks for sharing this idea. Agree that it seems useful to have more precise and updated information on this topic than we do now - so much of the evidence is either old or anecdotal and having new data to point to could be helpful in prioritizing future strategies. With that in mind, there are a few areas I would like to understand better in terms of this proposal:

• Sample: (recognizing that with an important topic like this, the results could be more likely to be dismissed, and so good methodological rationale to back them up could be extra useful) Do you think 200 respondents will be enough to draw useful conclusions? Do you have ideas for who you'd target to take this survey, in terms of non-contributors? Without being clear from the start about where respondents will be targeted, I'm wondering if you'd either end up with a skewed set of results or not enough results to show meaningful patterns.
• Timeline: Great that you're building in community engagement time throughout the project. The timeline for execution feels a bit brief to me, I wonder if you'd want to bake in some more time especially for consulting with other wiki-researchers on the survey before running it widely?
• It would be great to understand more about how you're thinking to share back results with the community and beyond.

Cheers, Siko (WMF) (talk) 00:32, 25 April 2015 (UTC)

Thanks @Siko (WMF):, lots of good stuff to think about here. In response to your question:

The sample size of 200 respondents was chosen more or less out of thin air ;) Looking at some other surveys and research out there, and considering what would be a useful amount of data to analyse, I'm going to increase this goal to 1,000 respondents. However I must admit to some trepidation about outreach and how to get 1,000 respondents who are 1) women, and 2) non-contributors. Jmorgan (WMF) suggested a CentralNotice banner, but I think it would be shown to too many people who aren't the target (i.e. aren't women, or are contributors) which could mess up the data. I'd appreciate further advice on this!

I've expanded the timeline throughout (again, it was just pulled out of thin air at first) and especially added more time in the early stages for review of the survey design, and at the end for review of the analysis. I'm keen to connect with some experienced researchers in the wiki community who may be able to share their expertise here, as I don't have academic research experience!

In terms of sharing results with the community and beyond: I envisaged creating a downloadable PDF report, a Meta wiki page with substantially the same information, and a downloadable data-set, and publishing them all under CC-BY-SA as the main deliverable of this project. I also envisaged writing a blog post or posts for relevant Wikimedia blogs, and posting about the results in relevant community spaces (gendergap mailing list, community noticeboards, or whatever else is appropriate). Another thing I've done in the past is to ask survey respondents if they'd like to be notified of the results of the survey, and if so to leave an email address (which will be used only for this purpose), and let them know when it's published, giving them a link to a relevant post that they can share on social media or the like. Finally, I imagine that keeping an eye on discussions coming out of this, and responding to followup questions, would be part of the work.

I hope that answers some of your queries! I'll try and fold most of this into the proposal. Please let me know if you'd like any more detail. --Skud (talk) 10:46, 28 April 2015 (UTC)

More on related research and sampling strategy

I think that the basic idea of this proposal is excellent and that such a project has a good chance to yield some valuable, actionable information. That said, I would like to second Siko's suggestion to think a bit harder about the sampling strategy and the amount of work involved.

Apart from the papers already mentioned by Jonathan above, I would suggest to look at these two as well:

• The student conference poster reviewed here in the research newsletter: "Non-participation of female students on Wikipedia influenced by school, peers and lack of community awareness" (as an example of a qualitative study using in-depth interviews, at the price of a rather small sample size of just 11 women that lets the results border on the anecdotal).
• The "skills gap" paper by Hargittai and Shaw summarized here which already unearthed some significant insight of the kind that this project is after. It had a more rigorous sampling method though with a larger sample size (547 college students, male and female).

Back to the sampling ideas listed in the proposal: The first two suggested venues in particular seem to correspond to comparatively small and specialized communities which may not be very representative of a larger population (it might still be worthwhile to post survey invitations there, as long as one keeps track of the provenance of responses and maintains transparency about which communities the results refer to). The following two suggestions - large women-centric forums and mainstream media - sound more promising in that regard. Still, I would encourage thinking about college students as a possible target group too, which are not only a large and relatively diverse population and an important target group for editor recruitment efforts, but also often easier to access as a "convenience sample", in particular for academic researchers - not coincidentally they are used in all the four papers mentioned above.

Like Siko, parts of the timeline seem a bit ambitious to me, even if implemented by someone well versed in the matter. Although I fully agree with the argument that both ends of the pipeline are important, I would suggest to focus first on respondents who have never contributed to Wikipedia. Surveying former contributors is a worthwhile but entirely separate project, with quite different problems to solve regarding sampling etc.; it could be addressed in a second phase of the project provided that the first part is completed or well on its way.

(Disclaimer: I am not involved in the decision process for these grant proposals; I was invited to comment on this and other research-related Inspire proposals by my colleagues from the grant team.)

Regards, Tbayer (WMF) (talk) 06:01, 25 April 2015 (UTC)

Thanks for your very useful feedback, Tbayer (WMF)! I've answered some questions above which cross over quite a lot with yours, so you might want to read those too, but I'd like to particularly address your suggestion of using college students. I think it could be very useful as one source of respondents, but I have two issues with it: 1) I'm not an academic, and have no idea how to go about accessing a pool of college students, and 2) surely college students are generally very young? So if we did use such a group, I'd definitely like to balance it with outreach to professional and social groups that have a wider age range. What do you think? --Skud (talk) 11:10, 28 April 2015 (UTC)

Actually, I'd just like to respond to the "never contributed" thing as well! I think it would be surprisingly difficult to find those people and exclude those who have made minor or casual contributions in the past, and I'm not sure it would be entirely useful to do so in any case! Those who've tried once or twice and found it didn't appeal to them are likely to have insights that will be useful to us, perhaps more than people who have never even tried! I agree that in-depth study of people who used to be heavy contributors but quit is a completely different story. However I think we can probably design this survey so that in an early question we ask whether they've ever contributed (and at what level) and focus the questions accordingly. --Skud (talk) 11:36, 28 April 2015 (UTC)

Thanks everyone, proposal has been updated!

Thank you everyone for your great feedback so far. I've updated the proposal significantly! Originally I just tossed it out there to see if anyone wanted to run with it, so it was a bit sketchy, but since people seem to be genuinely interested and I've been encouraged to improve it, I've taken onboard most of the suggestions made above and incorporated them into the proposal.

In particular I have:

• Written about the scope I envisage for the project, including language, contribution status, depth of investigation, and proposed sampling/outreach methods.
• Increased the timeline (roughly doubled) and expanded on some of the activities -- in particular I've added more review time for both the survey design and the analysis phases.
• Adjusted the requested budget accordingly.
• Increased the goal for number of respondents to 1,000 as a more useful sample size.

I've also responded to a few people's questions/ideas above. I'd definitely like to talk more about whether a banner ad could work for this! I'm also keen to find people with more academic research experience to help out with reviewing my work as I go, as I don't have that kind of background.

Thanks again, and please let me know if you have any other questions. --Skud (talk) 11:30, 28 April 2015 (UTC)

Aggregated feedback from the committee for Survey women who don't contribute

Scoring rubric	Score
(A) Impact potential Does it have the potential to increase gender diversity in Wikimedia projects, either in terms of content, contributors, or both? Does it have the potential for online impact? Can it be sustained, scaled, or adapted elsewhere after the grant ends?	7.3
(B) Community engagement Does it have a specific target community and plan to engage it often? Does it have community support?	7.3
(C) Ability to execute Can the scope be accomplished in the proposed timeframe? Is the budget realistic/efficient ? Do the participants have the necessary skills/experience?	7.9
(D) Measures of success Are there both quantitative and qualitative measures of success? Are they realistic? Can they be measured?	7.3
Additional comments from the Committee: Project would increase understanding of the gender gap (specifically why relatively few women edit Wikipedia) and thereby potentially indicate alternative ways for WMF sites to recruit and retain women. Definitely has potential for impact, especially if scope is broadened to survey women who contribute/create/participate on other sites, women who spend a lot of time volunteering, and women from other language groups Indirect research which may or may not give results that could be useful to address the gender gap. Would be interesting for the participants to create survey that could be translated and shared across as many communities/languages as possible. Appreciate the different phases of posting work on wiki to get review/feedback, as well as the level of detail provided for how to reach out to a variety of women both online and offline Thus far, there's evidence of community engagement via Participants and Endorsements. Might like to see community notifications, too. Would want the grantee to collaborate with someone experienced in designing surveys Appreciate the way activities have been broken down into manageable chunks; this seems very feasible. The budget is reasonable for doing all the work or doing the project management and teaming with other people to do the rest. The timeframe may be too ambitious to get widespread involvement during each phase of the project. In terms of budget efficiency - WMF should consider alternative costs of having a consultancy firm perform the study. I think the costs are reasonable. Survey of 200 people may be too small to be useful. Size of the survey should be increased Might like to see a measure about the diversity of women responding (i.e. it'd be nice to have balance in the 200 respondents so they're not all coming from the Systers mailing list, for example) Would like to see a strategy for using the data in a meaningful way. Would like to know more about what is meant by "in-depth interview answers." Would provide us with valuable data about what recruitment strategies are likely to be successful. Appreciate that this project provides a platform for women's voices/experiences to be heard

Inspire funding decision

Congratulations! Your proposal has been selected for an Individual Engagement Grant.

The committee has recommended this proposal and WMF has approved funding for the full amount of your request, $4,000

Comments regarding this decision:
We’re glad to see a research advisor on-board to offer strong survey expertise, and look forward to the findings! You may want to consider how such research, if successful in English, could be expanded to other languages in future. Will be in touch soon about project setup.

Next steps:

1. You will be contacted to sign a grant agreement and setup a monthly check-in schedule.
2. Review the information for grantees.
3. Use the new buttons on your original proposal to create your project pages.
4. Start work on your project!

Questions? Contact us.

Congrats! and, Gardner post

Congratulations on getting funded!

People interested in this research topic might also be interested in Sue Gardner's blog post "Nine Reasons Women Don't Edit Wikipedia (in their own words)". Gardner selected quotes by women from comments on several sites to illustrate 9 things that they found reduced or stopped their contributions. Sumana Harihareswara 13:25, 19 May 2015 (UTC)

Vanished?

Was this project withdrawn? Nemo 14:06, 3 June 2016 (UTC)

Project incomplete

Given there has been no response to multiple attempts to contact the grantee by email, and no updates or reports have been posted on Meta, please be advised that project will be marked incomplete on June 15, 2016. An incomplete grant will make the grantee ineligible for further funding from WMF so long as your project remains unfinished.

Please let us know if you have any questions.

--Marti (WMF) (talk) 20:21, 8 June 2016 (UTC)

As of today, this grant is marked incomplete. Please contact me or the Grants Administrator (grantsadminwikimedia · org) with any questions. --Marti (WMF) (talk) 17:37, 15 June 2016 (UTC)