Research:How role-specific rewards influence Wikipedia editors’ contribution

This research proposal was withdrawn on January 2, 2019, after discussion with members of the English Wikipedia community.]]

Created

20:36, 25 October 2018 (UTC)

Contact

Diyi Yang

Carnegie Mellon University

Collaborators

Robert Kraut

Carnegie Mellon University

Duration:  2018-10 – 2019-2

---

Research:Projects

This page is an incomplete draft of a research project.
Information is incomplete and is likely to change substantially before the project starts.

Our research examines whether labeling and rewarding editors for their role-specific behaviors increase their positive behaviors on Wikipedia. Doing so could help Wikipedia retain editors and encourage them to do needed work. Previous experimental research has shown that when editors who contribute heavily to Wikipedia receive public, informal rewards, such as barnstars, they subsequently contribute more . However, this prior research examined only the diffuse, motivational effects of the rewards, by getting editors to contribute more. But it did not examine specific, cognitive effects, in which the label helps these editors decide what to do. Our previous research has shown that many editors specialize in the types of work that they do on Wikipedia, such as adding substantive content or performing copy-editing tasks, and that one can computationally identify the roles they occupy. The new experimental research we would like to do will examine whether labeling and rewarding them for the type of work they perform and the roles they occupy (e.g., as a substantive editor or copy editor) will have effects beyond simply telling them that they are good Wikipedians. If activity-specific barnstars reinforce the actions that they reward, these barnstars provide Wikipedia a mechanism to encourage prolific editors to perform the distinct types of work they are most interested and skilled in performing.

Specifically, we plan to a conduct random-assignment experiment in which editors who are eligible to receive an activity-specific barnstar will either receive no barnstar, receive a generic barnstar that rewards them for contributing to Wikipedia generally or receive a barnstar that is specific to the type of work they do most frequently. Then we will investigate whether editors who receive barnstars show higher productivity and greater commitment to Wikipedia compared to those who receive no barnstars. In addition, we will examine whether editors who received activity-specific barnstars subsequently specialize more in the activities for which they were rewarded compared to those who received generic barnstars. This research will help us understand how informal labels sustain and direct volunteer effort in online production communities.

Methods

Our experiment will be conducted on the English Wikipedia. First we will run role-identification algorithms to identify approximately 1200 editors who heavily add substantive content to Wikipedia articles or who perform copy editing work. These comprise a pool of active editors who are eligible for receiving activity-specific barnstars. Within each role type, we will randomly give one third role-specific barnstars and give one third of them generic barnstars by posting reward messages on their user talk pages on Wikipedia, and we will not give barnstars to the remaining third.

Experimental design (N)

Barnstar condition/ Role type	Control No barnstar	Generic Barnstar	Role-specific Barnstar
Copy editor	100	100	100
Substantive editor	100	100	100
Total	200	200	200

Inclusion criteria:

• Editors who are active in the last three months
• Editors in the top 10% of those making edits adding substantive content or performing copy-editing actions

Exclusion criteria:

• Exclude users who had administrative or elevated privileges
• Exclude editors who have previously received a barnstar
• Exclude editors who are or control bots
• Exclude editors who have participated in our project discussion

We plan to observe the behavior changes among editors who receive the informal rewards, in order to investigate whether specific informal rewards positively influence editors' commitment to Wikipedia and production. To examine this, we will collect edits made by editors who received the different types of barnstars and compare their work with edits from editors who do not receive barnstars.

Our barnstar-giving random experiment will involve three conditions:

1. Control group: 200 editors who are active contributors will not receive any barnstars
2. Generic barnstar group: 200 editors who are active contributors will receive barnstars that are generic (e.g., The Original Barnstar).
3. Role-specific barnstar: 200 editors who are active contributors will receive barnstars that label and reward them for the specific types of editing behavior they perform most frequently. The half who add substantive content to articles will receive the Tireless Contributor Barnstar and the half who make copy-editing edits will receive the Copyeditor's Barnstar.

Here is an example message that will be given to editors in Generic barnstar group:

{{subst:The Original's Barnstar|1=. You have been doing a great job of improving Wikipedia. In the last 3 months, you have made XX edits. Wikipedia needs your contributions. Keep up the good work! Diyiy (talk) 20:36, 25 October 2018 (UTC)}}

Here is an example message that will be given to editors in Role-specific group :

{{subst:The Copyeditor's Barnstar|1=. You have been doing a great job of improving Wikipedia. In the last 3 months, you have made XX edits and the majority of them were copy editing edits fixing errors. Wikipedia needs these kinds of contributions. Keep up the good work! Diyiy (talk) 20:36, 25 October 2018 (UTC)}}

Research by Restivo and Van De Rijt suggest that an N of 200 editors per cell will provide enough power to reliably distinguish the effects of barnstars on subsequent behavior .

Note that we used the account of Diyiy in the example message above.

In the final deployment, we will seek help from some experienced Wikipedia editors to help post the barnstar message. We plan to post around 50 barnstar messages per week.

Timeline

• Nov 1st - Nov 30th, 2018: Brainstorm project ideas/proposal, and applying for IRB approval and Wikipedia's approval.
• Nov 1st - Nov 30th, 2018: Identify candidate editors and collecting their edits; applying our prior edit intention framework to identify editors' skills.
• Dec 1st, 2018: Give barnstars to editors who are assigned to different experimental conditions (receiving no barnstar, receiving generic barnstar and receiving skill-specific barnstar)
• Dec 2nd, - Dec 31st, 2018: Observe editors' behavior changes
• Jan 1st - Jan 15th, 2019: Analyze those editors' editing behaviors in Dec 2018, and compared it to their editing behaviors in Nov of 2018.
• Jan 15th - Jan 31st, 2019: Summarize results as a link on the Wikimedia Research:Projects page and wrap up the project
• Jan 15th - March 10th, 2019: Write up the manuscript for external publication

Policy, Ethics and Human Subjects Research

Our project has been approved by the institutional review board (IRB) at Carnegie Mellon University.

Volunteers Needed

We'd love to seek help from some experienced Wikipedia editors to help us post the barnstar message. If you are interested, please sign up here or message Diyi Yang. We plan to ask each volunteer to help us with around 100 barnstar messages, and will prepare the barnstar recipient candidates and template messages for you.

Results

N/A

Contacts

Diyi Yang, Diyi Yang, diyiy@cs.cmu.edu

Robert E. Kraut, Robertekraut, robert.kraut@cmu.edu