Research:Facilitating Public Deliberation of Algorithmic Decisions

04:03, 21 August 2020 (UTC)
Duration:  2020-August – 2021-May

This page is an incomplete draft of a research project.
Information is incomplete and is likely to change substantially before the project starts.

Introduction & BackgroundEdit

38 Wikipedia languages are currently using ML-based systems such as Objective Revision Evaluation Service, or ORES to make algorithmic predictions regarding edits quality. Past research has shown stakeholders often have different and competing values regarding these decisions. For example, there is a tension between improving the efficiency of reverting low-quality edits and treating newcomers nicely. We are proposing a deliberation-driven decision making process to address those competing values. For example, participants will be presented with different model cards representing different values and will discuss, debate, and negotiate about their models. Eventually, they will be asked to select a model and offer rationale behind their choice. We will also measure the process using metrics like quality of the decision, subjective satisfaction (legitimacy) and quality of the deliberation.

Value TableEdit

Social/stakeholder group Values (priorities) System Criteria
Experienced editors Optimize experience of the experienced editors Optimize accuracy for experienced editors
Newcomers Optimize experience of newcomers Optimize accuracy for newcomers &

Maximize non-damaging predictions for newcomers

Wiki Community; Fairness advocates Fairness Keep similar accuracy, and/or false positive/negative rates for both experienced and newcomers
Patrollers Reducing overall efforts and maximize precision Minimize both damaging predictive rates & maximize fraction of actually damaging edits that are predicted as damaging
ML developers Accuracy of the system Maximize overall accuracy
Readers Content quality and maximize negative predictive value Minimize the proportion for damaging edits predicted as non-damaging

Process DocumentationEdit

Date Activities Participants
10/20/2020 Discussed with Dr. Amy Zhang from the UW about adopting Wikum she developed for Wikimedia for the deliberation process. Amy Zhang, Wesley Deng.
11/12/2020 Had a conversation with Dr. Aaron Halfaker concerning the project design and which wiki communities to collaborate with. Aaron offered his insights on the next step. Aaron Halfaker, Hong Shen, Wesley Deng.
11/13/2020 Presented the project plan for Dr. Loren Terveen and Dr. Mark Snyder from UMN for their advice on how to better design the study to benefit the community. Loren Terveen, Mark Snyder, Hong Shen, Wesley Deng, Haiyi Zhu.