Research:Understanding Wikipedians’ Perspectives on Discussions and AI Tools to Improve Discussion Effectiveness
This page documents a research project in progress.
Information may be incomplete and change as the project progresses.
Please contact the project lead before formally citing or reusing results from this page.
Introduction and Background
editWikipedia talk pages are important places for the work to improve articles, achieve consensus and build community among the editors. Talk pages also can be places of dispute and disagreement, which are sometimes productive, but sometimes problematic.
One common source of disputes on talk pages involves Wikipedia policies. These policies encompass rules guiding Wikipedians on their conduct, profile management, article content creation, and so on. Notably, policies related to article content, such as 'Verifiability,' 'Neutral Point of View,' and 'No Original Research,' suggest a focus on enhancing encyclopedic content. In that sense, discussions citing these policies are inherently productive and essential for a richer Wikipedia. However, the full potential of disputes about content related to these policies often goes untapped due to a lack of human resources to thoroughly assess and resolve them. Understanding whether or how AI can support Wikipedians on talk pages requires a clear and in-depth understanding of Wikipedians' perspectives on talk page discussions, disputes, disagreements and whether or not they are effective. Our preliminary research aims are 1) to gain insight into Wikipedians' thought processes about potential talk page disputes, 2) understanding the characteristics of discussions in Wikipedia, 3) identifying opportunities to leverage AI-assisted tools, and 4) discovering opportunities to benefit the Wikipedian community.
Research Questions & Scope
editThe objective of our research is to gain a clear and in-depth understanding of Wikipedians' perspectives on talk page discussions. We have four questions that we hope to answer through this research:
- Thought Process – How do Wikipedians navigate and assess the main points and productivity of discussions with the potential to become disruptive disputes?
- Characteristics – What are key features of discussions with the potential to become disputes in Wikipedia and how can they be identified?
- Effectiveness – Can or how can AI generate assistive contributions to a discussion?
- Areas for Improvement – What editor needs are unmet and could be addressed?
Participants
editWe want to interview active editors of Wikipedia who have experience participating in, or reading/reviewing the discussions on article talk pages. We plan to recruit ~15 Wikipedia editors for this study.
We will recruit participants by sending out a Participation Questionnaire. If you are interested, please click on the link to express your interest: https://docs.google.com/forms/d/e/1FAIpQLSerXUz0XYYKUTdsWxGRdFVAqE6bO6kG0WK--YPbjznwCbJ2gA/viewform?usp=sf_link
Participant Incentive
editParticipation in the study is voluntary, but in recognition of the participant's time and effort, a $25 donation will be made to a 'like-minded' organization. Participants completing the interview can choose whether the contribution will be made to: Wikimedia Foundation, Creative Commons, or Internet Archive. If not otherwise specified, the default choice will be Wikimedia Foundation.
| Organization | Selection | Total |
|---|---|---|
| Wikimedia Foundation | 3 | $75 |
| Creative Commons | 0 | $0 |
| Internet Archive | 11 | $275 |
Methods
editThis project is based on qualitative research mainly involving semi-structured interviews conducted via video calls.
Recruitment
editWe will recruit participants who are aged 18 and above and who speak English through the "email this user" feature. We will identify individuals with experience in article editing and engagement in discussions based on publicly available edit histories. Our study team will send a direct email invitation, including a participation questionnaire that explores participants' experiences and opinions regarding Wikipedia discussions. Those agreeing to take part will be contacted again by our study team to schedule interviews.
Interview Procedure
editA 60 minute semi-structured interview will be conducted using a video call. For the interview, the participant will be provided a discussion extracted from an article talk page to read and consider. During the interview the participant will answer questions about their thought processes and perspectives. The researcher will then introduce an AI-generated discussion contribution. The participant will read the AI-generated contribution and answer questions about their thoughts.
Data Collection
editParticipants’ edit histories will be collected for screening and analysis. Before the interview, consent to audio record the video call will be obtained. After the interview, the audio recordings will be transcribed for further analysis. After the transcribing has been completed, the recordings will be deleted.
Research Ethics
editThis work has been reviewed by the Institutional Review Board (IRB) at the University of Washington and given an "exempt" determination. Before conducting the interview, we will inform the participants of the basic details of the study and ask for agreement to participate. During the interview, participants may decline to answer questions, or they can stop their participation and ask to have the interview deleted. We will maintain the confidentiality of the interviews by anonymizing the data and deleting direct identifiers. The data will not be shared beyond our research team and will not be used for other purposes.
Impact of the Study
editWe will gain a deeper understanding of how Wikipedians navigate discussions and the potential disputes that arise in them. We will also be able to understand how Wikipedians react to and ask for AI tools that may impact discussions. These findings could have broader impacts on building AI-powered tools for distributed contributor systems like Wikipedia.