Research:Effects of Feedback on Participation in Wikipedia

Contact

Robert Kraut

Carnegie Mellon University

---

Research:Projects

This page documents a planned research project.
Information may be incomplete and change before the project starts.

Key Personnel

Robert E. Kraut, Carnegie Mellon University

Aniket Kittur, Carnegie Mellon University

Haiyi Zhu, Carnegie Mellon University

Jenny He, Carnegie Mellon University

Amy Zhang, Carnegie Mellon University

Project Summary

The WMF's strategic planning process identified the health of the Wikipedia community as a major source of concern. Wikipedia is now losing experienced editors faster than it is gaining newcomers. Despite this, the community is harsher to newcomers now than it was in the past, for example in terms of reverting or deleting new articles they create. We are conducting research on the effects that the feedback Wikipedians receive on their contributions has on their subsequent participation in Wikipedia. Our goal is to find ways to increase the quantity and quality of contributions to Wikipedia.

Previous longitudinal, correlational research has shown that the feedback editors get predict the amount and quality of their subsequent contributions. For example, Choi et al has shown that the initial communication that new project members received from more experienced ones strongly predicts how long the newcomers stay in the project and how much they edit per month . Halfaker et al showed that having ones edits reverted is associated with editors subsequently contributing less, with this effect greatest for newcomers . However, those who survive this 'hazing' seem to produce better quality edits than they had in the past . Finally, Zhu et al showed that different types of feedback was associated with different amounts of subsequent contribution . She examined the effects of positive comments to a contributor for creating a well-written new article (positive task-based message), criticisms or reprimands to a contributor for not complying to Wikipedia guidelines (negative task-based message), giving directions to correct an error (directive task-based message), and feedback with a sociable, person-focused, friendly, and supportive tone or feedback that was not sociable.

However, one cannot conclude from this research that feedback actually changes the behavior of those who receive it, because people who get different kinds or amounts of feedback were probably initially different from those who get know feedback. For example, Wikipedians who whose work is reverted or who receive negative feedback are probably different from those whose work remains standing or who receive positive feedback. Halfaker et al demonstrated a similar phenomenon, showing that people whose work is reverted had historically made poorer quality contributions Our research is intended to resolve this ambiguity and to better understand whether the effect of feedback differ for newcomers than for more established editors. We propose a true experiment in which we will give appropriate feedback to editors who create new articles or provide them no feedback at all. We will then look at the their subsequent participation in Wikipedia -- the quantity and quality of their editing.

Methods

We will identity new Wikipedia articles and post appropriate feedback about the articles (or post nothing) to the authors' Wikipedia User Talk page. The researchers will identify a list of newly created articles that are two to five days old. We will only post a response to a newly-created article that is at least two days old, in accordance to Wikipedia's policy of "don't bite the newcomers".

We will screen these articles and only include in the sample articles that have done something well (e.g., are well organized and have inline citations) and that violate some Wikipedia substance or style guideline (e.g., have deadlinks or too few citations or references). This will allow us to craft relevant feedback messages that focus on positive aspects of the articles, lacks in the article or both. On a random basis, the creator of the new article will either receive feedback from us or not. If they receive feedback, the feedback message can include any combination of the following (16 conditions in total):

1. positive feedback focusing on something done well
2. constructive criticism, pointing to a lack in the article,
3. suggestions for other work the author could do in Wikipedia
4. personalized, social components, like a personalized greeting and closing.

To provide experimental control, all messages sent by the experimenter will be prepared beforehand in templates (see the following section), which are messages sent by other Wikipedia users. We are mimicking other feedback messages in Wikipedia both to increase the naturalness of our templates and to include that the risks associated with the experimental messages are no greater than the risks experienced by a Wikipedia user during everyday interactions on Wikipedia. We will then extract the users' activity data after one to two weeks in order to examine the users level of contribution; we will also extract some user info, such as date of first edit and prior editing history, in order to assess the recipients' level of experience and skill. However, we will not collect any private, identifiable information.

Because this is a field experiment, we need a large N to overcome random sources of error (e.g., participants not reading the messages addressed to them). Therefore we will attempt to send ~850 messages to editors (50 in each condition and another 50 for control group).

Here are a set of example messages our RA will send to Wikipedia editors.

• Positive feedback without social components:

I saw your article XXXX in the New Articles list--the image greatly improves the article. You may already know this, but here are some ways to draw attention to new articles. Jipinghe

• Positive feedback with social components:

Dear XX, I just took a look at your recently created article XXXX-- the article looks well put together: the information is presented clearly and is easy to understand. I don't know if you know this, but your article was one of XX that was created this week. Happy editing! Hope your day is going well and you are having fun. Jipinghe

• Constructive criticism without social components:

I'm posting this message on your talk page because you've recently created the new article XXXX-- however, I noticed the article contains errors: the article currently does not contain any references. As a new article, the most important thing is to find reliable references for all existing information. As you know, there’s always room for more work to make articles even better. Jipinghe

• Constructive criticism with social components:

Hi XX. I see you’ve created the new article XXXX--however, there might be some holes that need filling: the article currently does not contain in-line citations, and so doesn’t follow Wikipedia referencing guidelines. You may already know this but new articles are typically reviewed by other users, such as users on the ‘New Page Patrol’. If your article has not been reviewed, it should happen soon. I’m kind of new here myself but let me know if there is any way I can help. Have a nice day! Jipinghe

• Directive message without social components:

I'm posting this message on your talk page because you've recently created the new article XXXX--it would be great if you could also Wikify the related article XXX. You may already know this but new articles are typically reviewed by other users, such as users on the ‘New Page Patrol’.If your article has not been reviewed, it should happen soon. Jipinghe

• Directive message with social components:

Hello, XX! I just took a look at your recently created article XXXX--it would be great if you could also improve the related article XXX. I don't know if you know this, but your article was one of XX that was created this week. I’m kind of new here myself but let me know if there is any way I can help. Kind regards and happy editing! Jipinghe

See complete set of messages at /Messages

Wikimedia Policies, Ethics, and Human Subjects Protection

This study is minimal risk for participants. We will post appropriate feedback to Wikipedia editors who have created a new page in Wikipedia. The research assistants providing the feedback are members of the New Page Patrol, a collection of Wikipedia editors who evaluate and comment on new articles similar to those editors encounter regularly on Wikipedia. Our experimental stimuli-- online messages to Wikipedia editors who create new articles--are designed to provide feedback to the article creator and will be based on an evaluation of article created. Our research assistants sending the messages have experience editing in Wikipedia and can recognize good or poor article content or style. In addition, we will use tools such as spelling checkers, readability indices and Wikipedia style guidelines to provide article-relevant feedback. Thus, all feedback messages will be appropriate. The feedback messages have been modeled after messages that Wikipedia users are typically exposed to. All of the messages will follow guidelines of civility in Wikipedia. Thus, these messages are very similar to those that Wikipedia users might encounter in their everyday interactions on the website.

In sum, our RAs are legitimate Wikipedia contributors and will be acting as such to improve contribution by providing feedback to other Wikipedians (or not), though in a structured form as dictated by the experiment design.

We will not announce to editors receiving feedback that they are participants in an experiment. Doing so would undermine the authenticity of the feedback and therefore its effect. If participants receiving feedback knew they were in an experiment, we would not be able to generalize the results to understand how editors react to feedback from other Wikipedians. We believe these feedback messages will have their effects in part because they imply a relationship between the sender and recipient. They imply the sender cared enough to provide feedback. This relational meaning will be undercut if the recipients believed they received feedback simply because they were part of an experiment. For the same reasons, the research assistants will not sign their messages with a modified username that discloses their status as a researcher.

However, we will not be hiding the experimenter's research affiliation, and will prominently place a description and links to the study on their user page [See this example]. Their user pages will also contain instructions on how to withdraw from participation, for users who may feel uncomfortable being a part of our experiment and wish to opt-out. Any user who contacts the research assistants to opt-out will not have any further communication with the research assistants, and their data will be discarded. In addition, we will provide a description of the experiment and results at its conclusion. We will post a full description of the research and results to the Academic Research page, the RA's user page, and a link to the description on the participants' user talk page.

Benefits for the Wikimedia Community

First, participants in our experiments will receive legitimate feedback about their contributions to Wikipedia, helping them improve the quality of their work. Furthermore, messages that are sociable are designed to enhance the participants' sense of community on Wikipedia; directive messages that suggest additional activities for participants identifies areas in which they can contribute for them.

In general, our research can help the Wikipedia community develop guidelines about the type of feedback that contributors to Wikipedia should get. For example, it can discover whether the community should be especially nice to newcomers. Our goal is to increase the quantity and the quality of the contributions to Wikipedia.

Contacts

Jenny He, jipinghe, jipingh@andrew.cmu.edu

Amy Zhang, Amyxz, amyz@andrew.cmu.edu

Haiyi Zhu, haiyizhu, haiyiz@cs.cmu.edu

Robert E. Kraut, Robertekraut, robert.kraut@cmu.edu

Aniket Kittur, Nkittur, nkittur@cs.cmu.edu