Research:Teahouse/WSOR research sprints

This page in a nutshell: Summaries of research sprints conducted during the 2011 Wikimedia Summer of Research that informed the design of the Teahouse

Newbie teaching strategy trends

Full report: Newbie teaching strategy trends

Summary

There is a steep learning curve to contributing productively to Wikipedia. New editors, even those who edit in good faith, often make mistakes when they start out. We were interested in finding out how more established community members responded to these initial edits by new editors:

do they take the time to tell them personally what they had done wrong, and why?
Is the criticism they offer specific enough to be constructive, instructive and actionable?
Do they give them any kudos for the things they'd done right?
And, most importantly, how have these teaching practices changed over time?

We were interested in various approaches to teaching new editors how to contribute most productively to Wikipedia. We coded messages on new editors’ talk pages for moments of teaching/instruction, praise/thanks, criticism, and warning. We noted when these were templated or personalized interactions, and when editors referred to specific edits, types of edits, or editing Wikipedia generally.

Question: How have community strategies for teaching new editors to be effective contributors shifted since 2004?

Methods

We coded a sub-set of the data used in a previous research sprint that analysed the rise of warning templates on new users' talk pages. We adapted the coding scheme used in the previous sprint to focus specifically on identifying messages related to teaching.

Results

We found Wikipedian teaching strategies shifting in two significant ways. We saw a significant drop in messages including praise and thanks corresponded with an increase in the overlap of teaching with criticism, and a decline in personalized teaching corresponded with an increase in templated instruction. Implications. Messages posted to new editors’ talk pages reflect attempts to socialize these new community members, but the socialization tactics have become increasingly negative and generic in recent years.

Alternative Lifecycles of new users

Full report: Alternative lifecycles of new users

Summary

The process of socialization that new users of Wikipedia undergo is often portrayed as a gradual, stepwise process. For example, a new Wikipedia user may begin by copyediting, then contribute article content, then learn about project norms, and finally then join 'high-level' community processes like Wikiprojects, Did You Know (DYKs),Articles for Deletion (AfD) debates , and Requests for Adminship discussions. Well-established theories of organizational learning such as legitimate peripheral participation are fundamentally based on this assumption.

This project investigates to what extent new users experience these traditional modes of socialization, identifies the different kinds of spaces in which new users are participating, and to what extent they undergo alternative lifecycles— such as a new user creating an article and immediately thrust into an deletion debate.

Question: Broadly, how are new users introduced into the Wikipedian community, and has this changed over time?

Methods

The unit of analysis for this study is the individual user, 30 days after they make their first edit. A random sample was generated containing 200 new editors to Wikipedia per each 6 month period between January 2004 and June 2011, for a total of 1,400 editors. Researchers then manually coded each of the new users based on the schema of different community processes and coordination spaces, relying on the list of messages left for them as well as their contribution histories within their first 30 days.

Results

We found that new users are receiving substantially more notifications that their articles and images are being deleted. However, new users in 2011 are much less likely to participate in deletion processes (for instance, by disputing the deletion of their content). We also found that new users are participating substantially less in community processes in general, across almost all areas of activity. Implications. Like SubSubPop in the scenario above, new users may be confused or intimidated by the impersonal nature of deletion notifications and unsure of how to respond to them or how to participate in community process such as contesting a deletion.

New user help requests and participation in help spaces

Full report: New user help seeking behavior

Summary

New users receive a lot of inbound communication from the Wikipedia community in the form of (often templated) welcome and warning messages to their talk pages. These welcomes and warnings usually contain a variety of helpful links to resources, many of which were created to help teach new editors how to be successful contributors to Wikipedia. A lot of this information resides in 'help' namespace. However, despite the plethora of these helpful links, it is not known to what degree new users actually use these help resources.

Furthermore, as others have noted, the help infrastructure on Wikipedia is fragmented and confusing. For instance, with major help portals—Wikipedia:New Contributors Help Page vs. Help:Contents—divided between different namespaces, it may be difficult for new users to know where to ask a question in the first place. The help resources also suffer from inconsistent page layouts and navigation. And perhaps more fundamentally, participating in help usually requires editing pages with Wikitext, which is often challenging to new users.

Questions: What kinds of questions do new users have? Where are they asking them?

Methods

We gathered a random sample of new users from 2004-2010 (divided into half-year cohorts) who had made an edit to a non-article namespace sometime in their first 30 days. We analyzed the messages on the user's talk page (if any), and also any edits they made to namespaces other than 'Article' during that time period, for help requests. In order to gather the widest variety of data, we defined 'help' very widely to mean "Any question or request for information or assistance related to reading or editing Wikipedia pages, whether or not it was directed at a particular person."

Findings

New users by and large don't ask for help in help spaces, despite exposure to help resources through user talk page templates and the 'help' link in the global Mediawiki navigation menu. The three most common places where new users ask for help are: other users’ talk pages, their own talk page, and article talk pages.
More than half of all help requests that relate to norms, policies or guidelines for editing involve issues of notability or conflict of interest. 22% of new user help requests were about markup complexity or other technical issues.

Implications: New users need help on a wide variety of issues, and they don’t know where to ask for it, or else they’re not comfortable asking for it there. They need to be provided with places where they feel comfortable asking for whatever kind of help they need.

WikiProject Participation and mentorship

Full report: WikiProject participation and mentorship

Summary

Research from previous sprints shows that new users don't have a lot of opportunities to interact positively with veteran Wikipedians. However, WikiProject participants form close-knit groups of people at all levels of editorship who have shared interests, so they should provide ideal sites for this kind of interaction. This sprint follows up on previous research on how participating in Wikiprojects helps new users learn the ropes of Wikipedia.

Questions: What kinds of contributions do new users make, and what degree of support or mentorship do they receive from other WikiProject members, during their first 30 days after joining a WikiProject?

Methods

Our sample consists of the new user talk pages and contribution histories of editors whose first edit was after 2008 and who had edited the member list of a WikiProject within their first 100 edits. We examined these users' user talk page and user contribution histories 30 days after their first edit to one of the WikiProject member list pages

Findings

56% of joiners made at least one edit to the project workspace or an article within the WikiProject’s scope within 30 days of joining. Although newcomers exhibited a variety of participation patterns, many of their contributions were minor edits, such as adding the WikiProject template to an article talk page or updating information in an article in-fobox. Most newcomers did not participate in discussions on the WikiProject talk page.
When these editors did participate in project discussions, either on the project talk page or on the user talk page of project members, it was usually to ask a question. In the context of these questions, they often introduced themselves as newcomers and described their motivation for joining the project.

Implications: Many new editors who join WikiProjects make minor, peripheral contributions at first. They look to the project for assistance or guidance, and contextualize those requests by introducing themselves.

Visualizing WikiProject Activity

Full report: Visualizing WikiProject activity

Summary

In order to get a better sense of new user participation in Wikiprojects, as well as overall WikiProject activity. We have created a set of database tables that list various activity metrics for WikiProjects (e.g. # of joiners, pages claimed, and WikiProject page activity). We use these tables to track how WikiProject participation overall has changed over time. This may give researchers and community members a better sense of whether WikiProjects as a whole are still proving to be a vital mechanism for both onboarding new users and helping existing users coordinate their work activities, as well as provide the Foundation with tools for identifying Wikiprojects that are currently inactive or that are struggling to recruit new members.

During the course of this work, we decided it would be worthwhile to structure our research around the idea of a 'dashboard' for visualizing the various metrics we'd collected, at the level of individual WikiProjects. Many of these metrics had never been tracked before, at least not on such a grand scale, and for good reason .

Questions: How much coordinating activity are Wikiproject pages experiencing? What tools do Wikiproject members currently use to communicate, coordinate, advertise, track work?

Methods. However, before we started building a tool, we needed to know what kind of problems current Wikiprojects are facing--whether in recruiting new members, tracking progress towards goals, or measuring activity levels--in order to make sure the visualizer we created would actually be useful to the community. So we conducted a series of interviews with core contributors to several Wikiprojects, and asked these Wikipedians what their Wikiproject experience was like, why they contributed, what they did, and what challenges they experienced.

Findings

There are some tools and templates for tracking work towards project goals, but there is no unified framework and it's costly to set up and maintain these existing mechanisms. Also, they don't cover everything that members want to see/do.
In general there is poor support for internal processes of WikiProjects: maintaining a project requires a great deal of maintenance and coordination work, but that work isn’t engaging so it is often neglected.

Implications: Teahouse should find ways of surfacing project activity and automate critical but boring activities to reduce the burden on editors.