Research:A game of tome: Gamifying the experience of Wikipedia writing

Created

03:34, 30 August 2019 (UTC)

Contact

Dr. Direwolf & Deryck Chan

Cantonese Wikipedia

Collaborators

Professor Christopher Jackson, University of New South Wales; Dr. Chaak-ming Lau, Chinese University of Hong Kong.

Duration: 2019-Sept – 2020-December

Research:Projects

This page documents a planned research project.
Information may be incomplete and change before the project starts.

The Lead: Introduce and describe your project at a high level in one or two paragraphs. Will the output of this project provide tangible benefits for our community (in the form of data, software, Web services)? If the output of this project mainly consists of scholarly publications, what aspects of Wikimedia projects will they help to understand

In the past 10 years, Wikipedia has consistently been among the top 10 most visited websites in the world. Wikipedia is a free online encyclopedia that serves as a general reference work for people around the globe. Due to its open and free-to-use nature, it is a potent tool for transmitting scientific and philosophical knowledge to the general public. However, Wikipedia lacks a professional group of writers for its articles, and instead relies on contributions by the general public, resulting in at times low motivation to contribute or suboptimal quality of the resultant work. The current project seeks to tackle this issue by applying gamification to Wikipedia writing.

The researchers in this study adopt some principles from the game design literature and design a bot that gamifies (see gamification) the Wikipedia-writing experience for Wikipedians. Data will be collected to investigate how this bot impacts users’ willingness to contribute and the quality of their contributions (as measured by several metrics).

Note: According to our current plan, we will use the Cantonese Wikipedia as the research context, but that is subject to change.

Methods

Describe in this section the methods you'll be using to conduct your research. If the project involves recruiting Wikimedia/Wikipedia editors for a survey or interview, please describe the suggested recruitment method and the size of the sample. Please include links to consent forms, survey/interview questions and user-interface mock-ups.

Basics

Target sample size: ≥ 120

Design: This study will have a 2 x 1 experimental design.
- Group 1 (high flow group) ≥ 60 participants
- Group 2 (low flow group) ≥ 60 participants

Procedure

Recruit participants via snowball sampling. Participants will be native Cantonese speakers from Hong Kong who have had no prior Wikipedia-editing experience.
Host a face-to-face (or virtual, whichever is more viable) workshop: In this workshop, one of the researchers will help participants create their wiki accounts, give basic instructions on writing Wikipedia articles, and administer a questionnaire that measures several personality variables (see 3.3 “personality measures”) and ask each participant to list up to 3 topics that interest them.
Input the account names (which will not be the same as their real names) of the participants into the bot, so that the bot will keep track of them and recommend articles for them to write. Group 1 will be recommended articles within the topics they list as of interest to them, while Group 2 will be recommended articles that are not under the topics they list as of interest.
Use bots to gather data on participant behaviors and contributions to the Cantonese Wikipedia over the course of the next n month (n will be 6 according to the current plan, but that is subject to change).
At the end of the experimental period, give participants questionnaires to respond to. The scales will measure their experience when writing articles (see 3.4 “experiential measures”).
Conduct data analysis.

Bot descriptions

The key manipulation of this experiment is the presence of a bot whose function is to gamify the Wikipedia-writing experience. Gamification can generally be defined as the adoption of game design principles and theories to non-game contexts (such as management), usually with the aim of enhancing participants engagement in a target activity (Deterding et al., 2011 ; Huotari & Hamari, 2012). The current bot will be coded primarily by Dr. Direwolf, and based on several principles from the game design literature:

Theoretical basis

According to the flow theory (an influential theory in game design), a task is most likely to elicit high levels of engagement when it provides (1) clear goal(s) (Kapp, 2012), (2) clear feedback on progress (Lee & Hammer, 2011), and (3) optimal challenges (i.e., challenges neither too easy nor too difficult). Such activities are likely to induce psychological flow, a state characterized by high level of concentration and greatly enhanced motivation to engage in the flow-inducing task again afterwards. The ability to induce flow has thus been considered to be of importance in designing engaging games. Adopting principles from the game design literature, gamification naturally places emphasis on the ability for gamification practices to induce flow. Gamification in educational contexts, for instance, frequently use explicitly stated challenges (clear goals) and progress bars (a hollow bar that gradually fills as the student progresses towards their assigned goals; clear feedback) – both of which common in modern-day video games – as part of their practices (Kapp, 2012; Lee & Hammer, 2011).
Competition and cooperation are common elements in games. The concept of social engagement in games has been much discussed. The concept of social engagement suggests when players decide whether to play a game, they are partly motivated by the desire to interact with other players, an idea particularly important for the design of multiplayer games (e.g., Zyda, 2005 ). A multiplayer team-based shooter game, for instance, typically pit teams of players against each other. In such a game, there will be between-team competition, between-team cooperation (e.g., two teams working together to attack a third team that is leading), within-team competition (e.g., players of the same time competing to score kills), and within-team cooperation. It has been suggested that competition and cooperation are both intrinsically rewarding and thus a game needs to provide opportunities for both.
Customization is a common feature in many digital games. In a game, for instance, the player is allowed to choose an avatar to represent him/herself. Some games allowed an even greater degree of such customization, such as allowing players to create their characters with distinct appearances, clothing, and voices (Gordon et al., 2013). Some studies have shown that allowing players to customize their in-game characters have positive effects on player engagement, which has been interpreted to be due to the effect of autonomy: According to the Self-Determination Theory, humans are more likely to be intrinsically motivated to engage in a task if they think they are given the freedom to make choices when doing the said task.
Unlockable contents are also common in game design. In many games, for instance, a player can gain levels by continuously playing a game and finishing specified in-game tasks. Gaining levels often help a player earn more in-game currency, with which the player could purchase contents, such as additional clothing for the in-game characters, a greater variety of avatars, and in some cases, more powerful weapons. Gamification practices often involve unlockable contents. Iosup and Epema (2014) , for instance, conducted a study where they gamified a higher education course, and their practices involve some of their course contents being accessible only to students whose scores have surpassed a certain level. In the Wikipedia context, users who have made enough contributions can be nominated to become editors or admins, and if they succeed in attaining these positions, they will have access to more functionalities.

Based on the above principles, the bot will have several functions:

The bot identifies the users to be tracked. It will use an array to store the names of users it is tracking.
For each user, the bot proposes articles under 500 words for them to write. Group 1 participants will be recommended articles that are under the topics they list as their interests, while group 2 participants will be recommended articles that are not under their topics of interest. According to the flow theory, activities that are perceived to be relevant to oneself are more likely to induce flow. When we do data analysis, we will run an independent sample t-test to compare the two groups on level of flow, as manipulation check.
Each user will earn points based on the quality of their contributions (see 3.2 “quality of contributions measures” for the equation). At the end of each week, the points will be displayed to them on their user talk page, alongside tips and hints for improving their scores (flow theory, principle of feedback).
After the user has added a set milestone of word count, the bot will leave comments at the user’s talk page or the talk page of the article they are editing, to give comments. The comments will be randomly taken from a set of prepared comments, which are to be written by the researchers. The comments will be individualized to a degree, e.g., a comment that suggests an user to utilize more references will be more likely to appear if the article(s) written by that user has lower-than-average citations per unit word counts (flow theory, principle of feedback).
If an admin or editor edits an article by the focal user, the bot will prompt the admin or editor to give comments to the user (flow theory, principle of feedback).
If the user’s score has exceeded a certain threshold, the bot will prompt admins and editor to nominate him/her to become new admin and editor, so that he/she will be able to access new functionalities (principle of unlockable contents).
The bot will keep doing 1-6, until a milestone (total word count of 30,000, according to the current plan) has been reached. After that, the bot will leave a final comment on the user’s talk page.
The bot will create a leaderboard, listing the names of the top 20 users who have the highest scores in that month, and send messages to the focal users informing them of their positions on leaderboard (principle of competition). Leaderboard is a common feature in gamification to provide opportunities for competition (e.g., Deterding, 2013).
In the context of Wikipedia, researchers cannot prohibit certain users from using the features of customizing one’s user page. Hence, the manipulation cannot be ‘whether the user can customize’. The bot can, however, instruct and prompt the users it keeps track of to customize their own user pages, thus creating more opportunities for these users to use customization features. For each user, the bot will record the number of edits and word counts he/she made to his/her user page (principle of customization).
For each user, the bot will give him/her the names of under-bot users who edit similar article types as him/her and prompt them to contact each other to discuss potential opportunities for cooperation (principle of cooperation).

Variables

3.1 Users’ eagerness to contribute measures

Words written per unit time (day or week);
Words written per unit time during daytime (HK time 09:00-18:00);
Words written per unit time during evening (HK time 18:00-00:00);
Words written per unit time during nighttime (HK time 00:00-09:00);
Number of unique articles edited;
Number of articles created;
Number of times multi-media materials are used in an article;
Number of edits to templates;
Words written in talk pages;
Number of times mentioning an editor/admin’s name in talk pages;
Number of edits to one’s own user page;
Words written on one’s own user page;

3.2 Quality of contributions measures

Number of references added per article (num_ref);
Average number of categories added to each article (num_cat);
Number of inter-article links created per article (num_lin);
Number of articles chosen for “Do you know?” (num_dyk);
Number of articles submitted to “good article” or “featured article” (num_fea);
Double-blind ratings by editors/admins not involved in the study(?).
For each user, user.points = num_ref + num_cat * 3 + (num_lin) * 0.5 + (num_dyk) * 50 + (num_fea) * 20; Hence, a user’s point total is a function of the quality of their contributions. Some of the metrics are multiplied because they are influential (e.g., managing to produce a good/featured article is considered a huge achievement in the Wikipedian community).

3.3 Personality measures

The Jackson-5 scale, which measures sensitivity to rewards and punishments;
The Need for Cognition Scale;
The Dark Triad scale;
The Big 5 Aspects Scale;

3.4 Experiential measures

The flow state scale, which measures flow in a specified activity;
The PANAS;
According to the current plan, these measures will be administered once, at the end, as mentioned in step 5 of the methodology section. This is due to practical concerns, as it might not be viable to ask participants to fill in scales at multiple time points.

Timeline

Please provide in this section a short timeline with the main milestones and deliverables (if any) for this project.

Oct 2019 Recruit participants
Nov 2019 Host workshop
Apr 2020 Conclude and data analysis

Policy, Ethics and Human Subjects Research

It's very important that researchers do not disrupt Wikipedians' work. Please add to this section any consideration relevant to ethical implications of your project or references to Wikimedia policies, if applicable. If your study has been approved by an ethical committee or an institutional review board (IRB), please quote the corresponding reference and date of approval.

We are planning to obtain ethic approval from University of New South Wales (UNSW). Proposals and relevant forms have been prepared. However, according to UNSW policies, we are required to provide a letter of support from relevant authorities in case of studies that involve collaboration with an organization outside UNSW. Thus, we would like to request a signature from relevant Wikipedia authorities on one of the UNSW forms.

Results

Once your study completes, describe the results an their implications here. Don't forget to make status=complete above when you are done.

The result of this study will be of value for not only the gamification literature, but also for the expansion of Wikipedia’s functionality, which will ultimately result in greater transmission of scientific and philosophical knowledge to the general population.

References

Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011, September). From game design elements to gamefulness: defining gamification. In Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments (pp. 9-15). ACM.

Gordon, N., Brayshaw, M., & Grey, S. (2013). Maximising gain for minimal pain: Utilising natural game mechanics. Innovation in Teaching and Learning in Information and Computer Sciences, 12(1), 27-38.

Huotari, K., & Hamari, J. (2012, October). Defining gamification: a service marketing perspective. In Proceeding of the 16th international academic MindTrek conference (pp. 17-22). ACM.

Iosup, A., & Epema, D. (2014, March). An experience report on using gamification in technical higher education. In Proceedings of the 45th ACM technical symposium on Computer science education (pp. 27-32). ACM.

Kapp, K. (2012). Games, Gamification, and the Quest for Learner Engagement. T+D, 66(6), 64–68.

Lee, J., & Hammer, J. (2011). Gamification in Education: What, How, Why Bother? Academic Exchange Quarterly, 15(2), 146.

Zyda, M. (2005). From visual simulation to virtual reality to games. IEEE computer. IEEE Computer Society, pp. 25–32.