Research:System Design for Increasing Adoption of AI-Assisted Image Tagging in Wikimedia Commons
This page documents a research project in progress.
Information may be incomplete and change as the project progresses.
Please contact the project lead before formally citing or reusing results from this page.
In this research, we aim to investigate designs to increase the adoption and satisfaction of AI- assisted tools within commons-based peer production (CBPP) projects, with a specific focus on Wikimedia Commons. While AI- powered automation tools have long been integrated into CBPP projects for indirect tasks like content moderation, the utilization of AI for direct content generation has surged with recent advancements in generative AI algorithms. However, the impact of AI-assisted tools on human contributors and the design considerations to enhance their interaction, adoption, and satisfaction remain uncertain. This study proposes to co-design an AI-assisted image tagging tool with Wikimedia Commons contributors and users to increase adoption and satisfaction. We will perform a study of the prior WFM attempt to provide a computer-aided tagging (CAT) tool to understand the factors that led to its deactivation. We will then investigate technology designs to improve AI-assisted image tagging for structured Commons. The successful completion of this project is expected to advance the development of an AI-assisted image tagging tool on Wikimedia Commons, promoting greater adoption, usage, and satisfaction among contributors. Additionally, the insights gained from this study can be generalized to enhance interaction and collaboration between human contributors and AI-powered automation tools in the broader Wikimedia tools ecosystem and other CBPP projects.
Introduction
editAI-powered automation tools have long been integrated into commons-based peer production (CBPP) projects for indirect tasks such as content moderation and contribution quality. In recent years, with the rapid advancement of generative AI algorithms, there has been a surge in attempts to utilize AI-powered tools for generating direct content in CBPP, including creating Wikipedia articles and generating image annotations. AI holds the promise of enhancing content creation efficiency and consistency, thereby addressing content gaps in areas that may have received insufficient human contributions. However, the impact of AI-assisted tools on human contributors as well as the technology designs to enhance human contributors' interaction, adoption, and satisfaction with AI-assisted tools in CBPP remain uncertain. In this proposed research, we will fill in this gap by co-designing an AI-assisted image tagging tool with contributors and users of Wikimedia Commons, with the goal of increasing adoption and satisfaction.
Wikimedia Commons is a WMF project that makes multimedia resources available for free copying, usage, and modification. However, a lack of structured, machine-readable metadata about media files has hindered its accessibility, searchability, usability, and multilingual support. Recently, WMF researchers attempted to introduce computer-aided image tagging (CAT). Unfortunately, our prior research revealed low adoption of the CAT tool by Commons contributors. Participants reported unsatisfactory usability and performance of the tool and resistance to changing their existing workflow of creating and maintaining the local category system. The CAT tool was deactivated in September 2023.
In this project, our aim is to study the previous WFM attempt to provide the CAT tool, understand the factors that led to its deactivation, and explore technology designs that enhance the adoption, usage, and satisfaction of AI-assisted tagging among Commons contributors. Our research questions are:
- How do AI-assisted image tagging and structured data on Commons affect the work and workflows of diverse contributors and user communities in related Wikimedia projects, such as Commons, different language versions of Wikipedia, and Wikidata?
- What are the perceptions, concerns, and preferences of Commons contributors and users regarding the quality of tags suggested by AI algorithms?
- What usability issues and challenges do contributors encounter when using the CAT tool on Commons?
- What technology designs can enhance the quality of suggested tags, identify appropriate tags for a Depicts statement, and improve the overall user experience with AI-assisted image tagging on Wikimedia Commons?
The successful completion of this project would advance the development of an AI-assisted image tagging tool on Wikimedia Commons, fostering greater adoption, usage, and satisfaction among contributors. This enhanced tool will provide Commons contributors with a more efficient, accurate, and user-friendly method of adding structured data to multimedia files. Improved structured data will enhance the searchability and usability of multimedia resources across WMF projects, promoting inclusivity among diverse language communities. Furthermore, the design insights from this study can be applied broadly to improve interaction and collaboration between human contributors and AI-powered tools across the Wikimedia tools ecosystem and other CBPP projects.
Methods
editResearch Ethics
editThis work has been reviewed by an Institutional Review Board (IRB) at the University of Washington. In July 2024, the University of Washington Human Subjects Division (HSD) determined that this study is human subjects research and that it qualifies for exempt status. This exempt determination is valid for the duration of the study.
Qualitative Analysis of Discussions Related to the Deactivated CAT (Computer-Aided-Tagging) Tool
editTo understand the Commons editing community's needs, experiences, and concerns regarding the deactivated CAT tool, we conducted a qualitative analysis of Wiki discussions about the tool. We identified 11 Wiki pages (Table 1) where the CAT tool was discussed. These pages contained 595 comments across 172 topics, contributed by 160 unique users. Table 2 shows the distribution of user engagement in CAT-related discussions: 7 users left more than 10 CAT-related comments, 15 users contributed 5–10 comments, 59 users made 2–4 comments, and the majority—79 users—left only one comment.
| Page | #Topics | #Comments |
|---|---|---|
| Page 1 | 6 | 37 |
| Page 2 | 24 | 58 |
| Page 3 | 111 | 349 |
| Page 4 | 12 | 15 |
| Page 5 | 9 | 11 |
| Page 6 | 5 | 10 |
| Page 7 | 1 | 7 |
| Page 8 | 1 | 15 |
| Page 9 | 1 | 5 |
| Page 10 | 1 | 8 |
| Page 11 | 1 | 80 |
| Total | 172 | 595 |
| #Comments | #Users |
|---|---|
| More than 10 | 7 |
| 10-5 | 15 |
| 4-2 | 59 |
| 1 | 79 |
The first author manually copied all 595 comments, along with the corresponding usernames and links to the user pages on Wikimedia Commons or Metawiki, into a spreadsheet for analysis. We then conducted open coding of the comments using a thematic analysis approach. This process identified seven emergent themes regarding Commons editors’ experiences with the deactivated CAT tool:
- Goals of the Structured Commons project
- Evaluations of the quality of suggested tags
- Definitions of “depicts” statements
- Differences between image tags and “depicts” statements
- Existing infrastructure (image titles, categories, descriptions)
- Documentation and instructions for the tool
- User interface (UI) issues, including difficulties skipping images, overwhelming notifications, editing suggested tags, adding additional tags, error messages, and waiting periods
Findings from this qualitative analysis informed both the recruitment of participants and the development of our interview protocol.
Interviews
editTo gain a deeper understanding of Commons editors’ experiences with the deactivated CAT tool and the issues identified in our qualitative coding of CAT-related discussions, we conducted interviews with Commons editors who had participated in those discussions.
Recruitment
We aimed to recruit all 160 unique users we identified as having participated in CAT-related discussions. However, we did not send invitations to 41 users for various reasons:
- Six user pages appeared to belong to Wikimedia Foundation researchers responsible for designing or developing the CAT tool.
- Twenty-six users did not have a user page on Wikimedia Commons or Meta-Wiki.
- One user page was attributed to a deceased Wikipedian (WP:RIP).
- One user was banned indefinitely from editing all Wiki projects.
- One user was marked as retired from editing Wiki projects (WP:RETIRE).
- Six users had user pages on Commons or Meta-Wiki but had not enabled the “Email this User” function.
We contacted the remaining 119 Commons editors through the built-in “Email this User” function on Wikimedia Commons and Meta-Wiki. Each editor received a personalized message that included their username, the purpose of the study, a link to our Meta-Wiki study page, the Wiki page where their participation in CAT-related discussions was identified, and an invitation to participate in an interview. We emphasized that participation in the study was voluntary and offered a $25 Amazon gift card as a token of appreciation for their time, effort, and expertise.
Of the 119 editors contacted, 29 responded. As of December 2024, we have completed interviews with 16 of them. Table 3 shows the progress of our interview recruitment.
| Editors Contacted | 119 |
|---|---|
| Editors Replied | 29 |
| Editors Interviewed | 16 |
Data Collection and Analysis
Guided by the findings from the qualitative coding, we developed an interview protocol consisting of an opening script, four interview phases, and a closing script.
In the opening script, we introduce ourselves, explain the purpose of the interview, and outline how we handle and maintain the confidentiality of participants' data. We also request consent to audio-record the interview. Before participants provide consent, we clarify that, despite our efforts to anonymize data, Wikipedians may sometimes identify individuals or situations from the discussion. This ensures that participants are aware of any residual risks to confidentiality.
In the first phase of the interview, we ask introductory questions to understand participants’ engagement with Wikimedia Commons. These questions include how long the participant has been editing Wikimedia Commons and what types of work they typically do on the platform. We also ask participants to explain, in their own words, what they understand to be the goals of Structured Data on Commons. These questions help establish rapport and gather foundational information about the participants’ perspectives and experiences.
The second phase of the interview focuses on a discussion of sample images that participants recently uploaded to Wikimedia Commons. We show them three images they recently contributed and, for each image, ask them to describe the story behind the contribution, what the image depicts, and how they would tag the image themselves. Following this, we provide a list of tags generated by the Google Cloud Vision API for the image, translated to Wikidata items, to replicate the type of suggestions made by the deactivated CAT tool. We then ask participants to reflect on these tags, including whether they agree with any of the suggested tags, whether they would accept and add any of the tags to the “depicts” statements, how the tags could improve the discoverability of the image on Commons, and what metadata they find important for tagging such images.
The third phase of the interview shifts to reflections on the deactivated CAT tool itself. We begin by asking participants what they remember about the tool and how it impacted their work on Wikimedia Commons. We then ask them to explain their understanding of the role of the category system on Commons, the purpose of the “depicts” statements, and the differences between these two elements. We also ask them to reflect on the goals of the CAT tool on Commons, whether they think the tool achieved those goals, and why they believe it needed to be deactivated.
In the final phase, we invite participants to share their thoughts on potential improvements to the CAT tool and explore broader issues, such as the risks associated with using AI/ML for tagging images and ways to mitigate these risks on Wikimedia Commons.
In the closing script, we invite participants to share any additional questions or comments. We also acknowledge their time and effort by offering a $25 Amazon gift card as a token of appreciation and confirm their preferred email address for sending the gift card.
As of December 2024, we have conducted 16 semi-structured interviews. Fourteen of these were conducted in English using participants' preferred teleconferencing applications (Zoom or Google Meet) or via phone calls. During 13 of the teleconference interviews, we shared our screens to display and discuss participants' example contributions and the suggested tags. For one phone interview, we sent participants a list of relevant links in advance and asked them to open the materials on their own devices prior to the session. The duration of these interviews ranged from 45 to 90 minutes.
Additionally, two interviews were conducted through online chat. In these cases, the researchers emailed the interview questions to participants, waited for their responses, and subsequently asked follow-up questions. We believe this asynchronous approach provided flexibility for participants who preferred written communication or needed a translator.
These interviews resulted in a data collection of 16 interviews. The first author transcribed all interviews and documented relevant pages discussed during the sessions, such as user pages, user contributions pages, and Commons talk pages, to support data triangulation. The first author also open-coded the transcripts using a thematic analysis approach and recorded analytical memos. We plan to refine the themes iteratively and present our findings in subsequent reports.
Findings
editThis project will produce three outputs:
- Research reports for the Wikimedia community.
- At least one paper manuscript to be submitted to leading HCI conferences such as CSCW or CHI.
- Presenting our research progress and outputs at events such as Wikimania 2024 and Wiki Workshop 2025 for Wikimedia contributors, developers, and interested community members.