Research:Reducing the gender gap in AfD discussions: a semi-supervised learning approach

Created
1st of May, 2024
Collaborators
Duration:  2023-August – 2024-August
Article for Deletion, Gender Bias, Wikidata

This page documents a planned research project.
Information may be incomplete and change before the project starts.


We propose to develop machine learning methods for matching discussions in Article for Deletion based on the type and amount of external evidence available to discussants. This evidence scoring approach would form the basis for the development of a AfD matching tool that discussants could use to review the outcomes of previous discussions, and check whether they are consistent with the current one. In doing so, we want to study the information foraging practice of AfD participants and understand how the language-specific affordances of the AfD process affect notability assessments. For example, in both the English and Bengali Wikipedia, discussants are provided with direct links to external sources (e.g. Google, NYT, JSTOR), unlike in the Italian. We will thus conduct a content analysis of a sample of AfDs to determine the most frequent external sources used in each community. Of course the above questions require an operational definition of the gender of the subjects of AfD discussions. Here, we choose to leverage Wikidata as the ground truth about gender in AfD discussions. Our preliminary analysis on AfDs in English analyzed by Tripodi [2] shows that this approach achieves >60% coverage on average. Furthermore, even though our focus is on the gap in coverage between men and women, it allows us to extend our analysis beyond traditional genders (e.g. transgender men, women, etc.) due to the availability of rich gender information.

This project could help researchers and Wikipedia contributors gain a better understanding of AfD debates on biographies of women and other genders not typically considered when dealing with the gender gap in content representation. Our AfD discussion matching service could enable researchers and contributors to compare the potential outcome of ongoing discussions with that of similar discussions based on the availability and type of external evidence. We envision such a tool could promote consistency in outcomes across debates regardless of gender. For example, WikiProjects devoted to closing the gender gap, like “Women in Red”, may benefit from the ability to identify gaps in outcomes between discussions of biographies of women and men. As a proof of concept, we will build a dashboard keeping track of AfDs of biographies with relevant stats broken down by gender. We will take a number of steps to maximize the chances of adoption of our tools. We will deploy our tools on the Wikimedia cloud services and make the source available on Github. We will advertise our research on relevant WikiProjects related to gender and inclusion, like “Women in Red”, and “LGBT Studies” and invite their members to attend participatory design sessions.

Methods

edit

We will perform a content analysis of biographicals AfDs to understand how discussants evaluate external sources. Using an open coding approach, we will train human annotators to identify notability assessments made by discussants within the AfD. These assessments will typically include citations to external sources, which will allow us to identify what sources AfD discussants use in practice, and thus to define metrics that operationalize the concept of external evidence.

Our next step will be an analysis of AfD debates of biographical articles, in which we will compare debates about biographies by gender. As the vast majority of Wikidata labels covers men and women, we will focus first on a comparison of these two genders. However, we will also test how our approach performs when using gender labels beyond these traditional genders. To match AfDs based on the strength and type of external evidence, we will experiment with propensity score matching, though we may consider alternative matching methods, like Coarsened Exact Matching. Thanks to the scoring method, we will investigate the effect of the gender of the AfD subject on group composition and stance of the debates, and on how editors assess external sources to gauge the notability of the subject of a biographical article. We will pre-register our study on OSF or a similar repository.

Finally, to achieve our goal of a cross-cultural study of AfD discussions, we will develop a suite of multi-lingual tools for AfD debates. Our initial goal will be to support three languages of interest (English, Italian, and Bengali) including a parser for AfD discussions, and a set of scrapers of core external sources used in each language community.

Timeline

edit

Please provide in this section a short timeline with the main milestones and deliverables (if any) for this project.

Policy, Ethics and Human Subjects Research

edit

It's very important that researchers do not disrupt Wikipedians' work. Please add to this section any consideration relevant to ethical implications of your project or references to Wikimedia policies, if applicable. If your study has been approved by an ethical committee or an institutional review board (IRB), please quote the corresponding reference and date of approval.

Results

edit

Describe the results and their implications here. We encourage you to share preliminary data. Don't forget to make status=complete above when you are done.

Resources

edit

Provide links to presentations, blog posts, or other ways in which you disseminate your work.

References

edit

Tripodi, F. (2023). Ms. Categorized: Gender, notability, and inequality on Wikipedia. New media & society, 25(7), 1687-1707.