Research:Shallow Entry: A problem-centric approach to new editor orientation

Contact

Shilad Sen

Macalester College

---

Research:Projects

This page documents a planned research project.
Information may be incomplete and change before the project starts.

Key Personnel

• Shilad Sen
• Aaron Halfaker
• Rosta Farzan
• John Riedl
• Robert Kraut

Project Summary

Wikipedia has developed a vast array of resources to help new Wikipedia editors become productive Wikipedians. With some notable exceptions these resources ask users to first learn technical aspects of Wikipedia such as WikiMarkup.

This technology-focused approach to nurturing newcomers is reminiscent of approaches historically used in introductory computer science courses. In recent years, research has shown that many students - and women in particular - are not inspired by the technology-focused introductions . These students do not receive satisfaction from seeing the computer do what they want. Instead, these students find inspiration through solving problems.

Our project compares Wikipedia's technology-focused approach to acclimating newcomers to a problem-centered approach. The problem-centered approach begins by teaching users how to critically assess article quality through the lens of Wikipedians. Next, users will identify specific parts of the article that need to be changed and suggest general strategies for changing them. Finally, when users understand exactly how the content of an article can be improved, we introduce them to the editing interface and ask them to carry out edits.

Methods

We will recruit subjects who have never edited Wikipedia from MovieLens, a movie recommender website maintained by GroupLens_Research. Participants will interact with Wikipedia through a portal on MovieLens that directs them to specific activities on Wikipedia when necessary.

The study will be divided into two main experimental groups. Each group will be walked through a series of three steps designed to help them become effective editors. All members of each group will start each stage at the same time. The stages will be chronologically separated by approximately one week. Users will only interact with Wikipedia in Stage 3.

Stages in Technology Focused Group (1/3 of subjects)

We are still developing the steps for the technology-focused introduction introduction to Wikipedia, but we will base them on the interactive lessons available to online ambassadors for the Public Policy Intiative. Tentative stages:

1. Users will walk through a tutorial in which they perform a set of simple Wiki markup tasks (make something bold, a list, etc) (on MovieLens).
2. Users will create a userpage (on MovieLens, with option of pushing edits to Wikipedia).
3. Users will edit a movie article (on Wikipedia).

Stages in Problem Focused Group (2/3 of subjects)

1. Users will rate the quality of pages using a rubric similar to the Public Policy Initiative's Quantitative Metric. We will also provide general comments about the quality of the page (on MovieLens).
2. Users will suggest specific section-level comments using WikiGnome. The WikiGnome tool enables users to insert cleanup templates and section-anchored comments without WikiMarkup (on MovieLens, with the option of pushing changes to Wikipedia).
3. In the third stage, participants will create an account on Wikipedia and edit Wikipedia pages based on the WikiGnome comments (on Wikipedia).

Online environment: MovieLens (private) and Wikipedia (public)

For both groups, each of the three stages of the experiment take place online either at MovieLens or Wikipedia.

• The first stage in both top-level groups occurs on MovieLens, so users will have no impact on Wikipedia.
• The second stage for both groups occurs on MovieLens, but users have the *option* of making their work public on Wikipedia.
• The third stage for both groups occurs on Wikipedia.

Although the second stage is optionally pushed to the public Wikipedia, we expect few users to choose the option. In addition, the work subjects complete (adding markup templates or creating a user page) seems to have relatively low risks to Wikipedia.

Estimate of Number of Subjects

We estimate that users participation will decrease at each stage of the study:

• 400 users will complete stage 1
• 100 users will complete stage 2
• 30 users will complete stage 3.

Actual numbers will depend on the initial response rates and the fall off rate at each stage.

Involving the Wikipedia Community

Once we have completed the RCom approval process, we will describe our study to http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Film and ask for their support and feedback.

Dissemination

We will publish our results in academic journals and make them freely available on [1]. The source code for our project will be shared under a GPL license. We will also make the Wikipedia editing data associated with the participants in our project available under a CC license.

Wikimedia Policies, Ethics, and Human Subjects Protection

The University of Minnesota IRB has approved a Category 2 exemption for this experiment. Their internal tracking number for the study is 1204E13543.

Some portions of our study will be conducted on the MovieLens website. We will recruit MovieLens users through email and a solicitation on the website. MovieLens users are aware that MovieLens is maintained partially as a research platform, and we commonly ask them to participate in experiments on the site. Users have the option to opt out of this experiment (and any other experiment on MovieLens) at any time. Users will rate articles associated with movies, provide comments about them, and provide feedback through WikiGnome through web pages that we host. This data will generally covered by the MovieLens privacy policy.

A group of users will be placed in a "cohort" experimental condition. These users will be able to view other cohort members ratings, comments, and edits. We will clearly inform cohort participants of the privacy levels associated with their contributions.

To effectively lead users through the "problem-focused" approach we need to know their Wikipedia identity and carefully shape their Wikipedia interactions. For example, we need to ask users to edit a specific series of movie pages, and we need to install javascript associated with WikiGnome in their profile. Participants will create a Wikipedia account through pages hosted on our server. An outline of this process follows:

1. Our server will provide the participant with Wikipedia's account creation recommendations and policies and carefully explain how and why we need their user information.
2. The user will provide a requested user name.
3. The server will check to make sure the username is not in use.
4. The server will create an account for the user name and assign it a randomly generated password.
5. The server will install the WikiGnome javascript plugin.
6. The server will provide the user with the randomly generated password.
7. The server will direct the user to login using the username and password they supplied and edit a particular series of articles.
8. When this process is finished, the server will use the "reset password" form to create a new randomly generated password and email it to the user.

Benefits for the Wikimedia community

Our goal is to help more people become effective Wikipedians, and we believe the problem-centered introduction to Wikipedia will support this. If our hypothesis is correct, Wikipedia may be able to alter their signup procedure and introductory tutorials to nurture a more diverse editor base.

We also hope to design our open-source software portal so that outside researches can extend it to study other newcomer strategies.

Time Line

• April 2012: Finalize Experiment Design, Obtain IRB Approval
• May 2012: Conduct study
• June 2012: Analyze data and prepare publication

Funding

This project is funded by NSF grants IIS 11-11201 and IIS 08-08692.

Related Research

Unfinished.

External links

• MovieLens
• GroupLens

Contacts

Any questions can be directed towards Shilad Sen, Assistant Professor of Computer Science at Macalester College.