CivilServant Initial Data Analysis For Community Outreach
For CivilServant's work with Wikipedians in multiple languages (outside English) to test machine learning based support for newcomers and gratitude systems, we are starting by having conversations with Wikipedians who can introduce us to the culture and needs of different language Wikipedias. Because we knew that not all languages would be large enough for the kind of A/B tests we're able to support, Julia, Max, and Nathan did some early analysis to identify the chance that we would be able to work with a given language. This page documents our process and our results as of July 2018.
Requirements for a Language Wikipedia to Be Part of A StudyEdit
As CivilServant reaches out to language Wikipedias about possible research, we want to make sure that we're respecting everyone's time by talking primarily to communities with the basic conditions for conducting an experiment. At the least, this means:
- Newcomer study:
- The ORES machine learning system is available for that language
- The community has enough newcomer editors per month for a newcomer-welcoming experiment to be viable
- ORES identifies enough damaging and goodfaith edits from newcomers to be useful
- Gratitude study:
- Thanks, Love, or both are available for that language
- The community has enough editors who haven't yet sent or received thanks & love
Collecting an Initial List of WikipediasEdit
To answer these questions, we did a review of language Wikipedias for the presence of the needed features. Here's what we compiled in late May 2018.
Evaluating Statistical Power for Experiments with ORES, Thanks, and LoveEdit
Working from the subset of languages that included support for ORES goodfaith scores, we collected data about newcomers in November 2017 and the subsequent six months, using that data for power calculations to estimate, however imperfectly, the chance of observing a statistically-significant result of possible experiments over a six to eight month period this coming year. We based our power analyses on prior research involving welcoming newcomers and rewarding editors. Because we expect that communities will be co-developing experiment design with us, we see this as an indication of whether communities will have the flexibility they need to imagine something that works for them.
We're still consolidating our codebases, so you can see our code in the following places for now:
- Data collection & preparation:
- We manually conducted power analyses using Alexander Coppock's power calculator at EGAP
Initial Language Wikipedias for OutreachEdit
Based on this early data analysis, we developed a simple heuristic for our confidence that we might get enough statistical power for a study with a given language for a given study type, rating them high, medium, low, and unlikely.
|Code||Language||ORES Confidence||Gratitude Confidence||Notes|
|fr||French||High||High||Does not have Love currently|
|ru||Russian||High||High||Does not have Love currently|
|pl||Polish||Medium||High||Does not have Love currently|
|nl||Dutch||Unlikely||Low||Does not have Love currently|
|cs||Czech||Unlikely||Low||Does not have Love currently|
If Your Language Is Not On this ListEdit
If your language Wikipedia is not on this list, it still may be possible to work with us, if there are enough people in your language Wikipedia who are eager to try community-led experiments. If your language receives ORES support in the next few months, we may be able to include you. Also, if a smaller language Wikipedia is expecting to run a substantial campaign or expects to have a large influx of newcomers for a predictable reason in the later part of 2018 and early 2019, we may be able to include you- just drop us a note!
- ↑ Morgan, J., & Halfaker, A. (2018). Evaluating the Impact of the Wikipedia Teahouse on Newcomer Retention. (preprint)
- ↑ Restivo, M., & Van De Rijt, A. (2012). Experimental study of informal rewards in peer production. PloS one, 7(3), e34358.
- ↑ Restivo, M., & van de Rijt, A. (2014). No praise without effort: experimental evidence on how rewards affect Wikipedia's contributor community. Information, Communication & Society, 17(4), 451-462.