Research:Framing Support for Newcomers

Created

Contact

Yla Tausczik

Carnegie Mellon University

Collaborators

Rosta Farzan

University of Pittsburgh

Robert Kraut

Carnegie Mellon University

John Levine

University of Pittsburgh

Jonathan Morgan

Wikimedia Foundation

Duration:  2014-07 – 2014-10

---

Research:Projects

This page documents a proposed research project.
Information may be incomplete and may change before the project starts.

Key Personnel

• Yla Tausczik
• Rosta Farzan
• Robert Kraut
• John Levine
• Jonathan Morgan

Project Summary

One of the major challenges Wikipedia faces is recruiting and retaining new editors. An important factor in whether a newcomer makes it through this transitional period is whether they feel comfortable seeking out the information that they need to become a good member of the community. In the current study, we plan to experimentally test who newcomers feel most comfortable turning to for this information, peers or experts.

Newcomers may feel a trade-off in seeking help from experienced, veteran members. On the one hand, newcomers may feel that experienced members are knowledgeable. On the other hand, they may feel these users are judgmental and have difficulty relating to inexperienced newcomers. If so, they may feel more comfortable seeking help from other newcomers, who will be less knowledgeable, but perhaps more sympathetic.

Currently, mentoring programs such as the Teahouse arbitrarily advertise their program as a place to seek help from experts or peers. In the current study, we plan to run a controlled experiment in which new editors are invited to participate in the Teahouse. New editors will be randomly assigned to either receive an invitation that emphasizes the Teahouse as a place to seek help from other newcomers (peers) or an invitation that emphasizes the Teahouse as a place to seek help from experienced Wikipedians (experts). By monitoring how many newcomers choose to participate in the program depending on the invitation that they receive, we can determine who newcomers feel most comfortable seeking information from, peers or experts. This study will inform the Wikimedia community about the best way to recruit newcomers to a mentorship program.

Methods

New editors who have recently joined Wikipedia will be invited to participate in the Teahouse. Participants will be identified automatically by HostBot. HostBot identifies new editors who have recently registered an account, made an edit, and are classified as making good-faith edits by Snuggle. HostBot will then send an invitation to these new editors to join the Teahouse and to get help. These participants will be randomly assigned to receive one of two invitation templates, either one emphasizing Teahouse as a place to get help from peers or as a place to get help from experts.

Previous experiments with the wording of Teahouse invitations (testing the impact of personalized and generic invitations), and other welcome and warning templates left on new users' talk pages, have yielded mixed results. In both of these cases, it was proposed that editors may not read the templates closely enough to notice the difference. In this test, we attempt to make the difference between the templates more salient by using bold text.

Invitations

Newcomers will be sent invitations to join the Teahouse that are very similar to the existing invitations, but carefully worded to either frame the Teahouse as a place to seek help from other newcomers (peers) or as a place to seek help from experienced Wikipedians (experts). Here is an example of two invitations, each framed in a different way:

Peer Framing

Hi New editor! Thanks for contributing to Wikipedia. Come join other new editors at the Teahouse! The Teahouse is a space where new editors can get help from other new editors. These editors have also just begun editing Wikipedia; they may have had similar experiences as you. Come share your experiences, ask questions, and get advice from your peers. We hope to see you there! J-Mo (I'm a Teahouse host) Visit the Teahouse This message was delivered automatically by your robot friend, HostBot 15:46, 26 June 2014 (UTC)

Expert Framing

Hi New editor! Thanks for contributing to Wikipedia. Come join other experienced editors at the Teahouse! The Teahouse is a space where new editors can get help from experienced editors. These editors have been around for a long time and have extensive knowledge about how Wikipedia works. Come share your experiences, ask questions, and get advice from experts. We hope to see you there! J-Mo (I'm a Teahouse host) Visit the Teahouse This message was delivered automatically by your robot friend, HostBot 15:46, 26 June 2014 (UTC)

Data Collection

We will examine the proportion of participants who choose to ask a question in the Teahouse after being invited to join, and we will test whether this proportion is different for the two conditions (peer or expert invites). We will also examine the kinds of questions newcomers ask when they expect the audience to be primarily peers or experts.

Estimate of the Number of Subjects

The proportion of newcomers who ask a question after being invited to join the Teahouse is very low, between 3-4%. We estimate that the difference between the two types of invitations will probably be small, perhaps as small as 1.5%. Thus, to measure a 1.5% difference using a "difference between two independent proportions" test with sufficient power (0.95) we will need to recruit and invite at least 8,334 participants. We anticipate being able to recruit and send invitations to between 100 and 500 new editors per day, which means to achieve the necessary sample size we will need to conduct the study for a minimum of 2 weeks and maximum of 3 months.

Dissemination

The results of this study will be published in academic journals.

Wikimedia Policies, Ethics, and Human Subjects Protection

We have received approval from Carnegie Mellon University's Internal Review Board as an exempt study (HS14-387).

To conduct this study, we will not be recruiting any additional Wikipedia editors or asking for any additional work on the part of these editors. The only change is that the invitations currently being sent by HostBot to new editors will be randomized and have slightly different wording than the current invitation.

Benefits for the Wikimedia community

The goal of this project is to understand who newcomers feel most comfortable seeking information from, peers or experts. The results of this experiment may help the Wikimedia community in two ways. First, it will directly inform the Wikimedia community of the best strategy to use to recruit and invite newcomers to participate in mentorship and socialization programs. Second, it may help the Wikimedia community to design mentorship programs that matches newcomers with their preferred type of mentors (peers or experts).

Timeline

• June/July 2014: Finalize Experiment Design, Obtain IRB Approval
• July-September 2014: Conduct Study
• September 2014: Analyze data and prepare publication

Funding

This project is funded by NSF grant 110790 titled "Collaborative Research: Supporting Newcomer Socialization in Online Production Communities".

References

This work builds on the co-investigator's previous work examining peer support and socialization programs online, for example see .

Contacts

Any questions can be directed toward Yla Tausczik, Postdoctoral Fellow at Carnegie Mellon University.