Research:Community-centered Evaluation of AI Models on Wikipedia/Formative Study

This page documents the semi-structured interview questions for the formative study of the research on community-centered evaluation of AI models on Wikipedia.

Interview questions edit

  • Please describe your experience with AI tools, such as ORES, for counter-vandalism, new page review, or other tasks on Wikipedia.
  • Please share your experience as a [participant’s role] within your community on Wikipedia.
  • How did your community make decisions about the design and use of AI tools?
  • How were you involved in this decision-making process?
  • How did your community evaluate whether these AI tools fit the community’s needs and values before or after the deployment?
  • How were you involved in the evaluation process?
  • How did community members collaborate and resolve disagreements during the evaluation process?
  • Was the evaluation process effective or not? Why?
  • Are there forms of support that would be particularly helpful to have from your perspective as a [participant’s role]?
  • Are there forms of support that would be particularly helpful to have for the entire community?
  • Did the evaluation change the community’s perception and acceptance of AI tools? How?
  • Have you ever participated in data labeling campaigns on Wikipedia? If so, would you please describe your experience?
  • Can you envision ways to better support communities in evaluating AI tools before they are deployed to make more informed decisions about whether or not the community should adopt them?