Grants:IdeaLab/Full Circle Gap Protocol: Addressing the Unknown Unknowns/Final

This project is funded by an Individual Engagement Grant

Welcome to this project's final report! This report shares the outcomes, impact and learnings from the Individual Engagement Grantee's 6-month project.

Part 1: The Project

Summary

This project successfully piloted a feminist, and feminism-focused, process of doing distributed editing among non-Wikipedian experts through pairing with an undergraduate class. As a pilot, the gap finding project was designed to try out one method of doing distributed editing. At the same time, the pilot was an opportunity to reflect at a meta level on process -- and the academic experts were involved as critics reflecting on how the pilot went. The outcomes of this project include two learning patterns, "Engaging non-Wikipedian academic experts to identify content gaps" and "Using Wikipedia’s gaps as feminist teaching tools" that are based out of the projects activities and reflections.

Methods and activities

What did you do in the project?

This project is focused on developing a feminist, pilot protocol for doing a feminism-focused gap-finding project. Achievements include:

• Recruited five academic experts from the University of Washington to participate in the pilot
• Signed Letters of Engagement for each participant
• Organized and hosted a five-hour brainstorming session to discuss, and do, gap finding
• Contacted a community partner, Ada's Technical Bookstore, where the event was hosted
• Created Wordpress blog to chronicle the project
• Presented in four undergraduate classes and consulted with four professors on doing Wikipedia editing around feminism content gaps with students (three that were a part of the project, one class and lecture that was additional)
• Recruitment of one course to do Wikipedia editing and use the gap-finding list generated by the project
• Presented on successes and lessons so far at WikiConference USA (Talk titleː "Working with Academic Experts")
• Participated in UW Research Commons editing events, invited a local feminist to present a reading of Wikipedia as related to her leadership in an association for women in mechanics prior to a public editing event
• Conducted three close reading gap-finding activities with experts to augment the brainstorming session
• Worked with one instructor to assist in developing a list into a content editing assignment
• Observed 40 student group presentations about their experience
• Assisted instructor and WikiEd to use the dashboard to track student editing
• Conducted five one-hour debriefing sessions with experts
• Wrote two learning patterns

Outcomes and impact

Outcomes

The outcomes of the pilot are the following:

• Recruitment of five academic experts
• Recruitment of one course
• 21 content gaps related to "feminism and technology" identified
• 3 close readings of gaps conducted
• 4 course presentations in undergraduate classes
• 40 enrolled student editors
• 16 articles edited
• 1 article talk page edited with suggested content additions
• 1 article created
• 11,000 words added
• 45 references added

The outcomes of reflecting on the process of the pilot include:

• 1 five-hour brainstorming session with experts
• A “bestiary” of six gaps on Wikipedia identified
• A list of 21 "content gaps" on Wikipedia related to feminism and technology identified
• 3 classroom presentations with participants (see specifics in midpoint report)
• 3 experiments in using feminist methodologies for teaching Wikipedia to undergraduates (see this learning pattern for details on incorporating Wikipedia into classrooms)
• 3 close readings conducted (see the learning pattern for more details)
• 5 one-hour debriefing sessions

This project was successful on many levels in piloting a feminist, and feminism-focused, process of doing distributed editing among non-Wikipedian experts through pairing with an undergraduate class. As a pilot, the project tried distributed editing. Along the way, myself and the academic experts critiqued the process. This was intentional, treating workflow as political was a fundamental feminist value in the project. Indeed, as one participant put it, treating the Wikipedia editing process, as well as the question of missing or unknown content, as areas worthy of intervention made the project attractiveː

I became involved in this project because of the feminist focus. How do you actually begin from feminism, especially in terms of Wikipedia, which isn’t built from a feminist foundation. Coming together with a group of scholars who were not familiar with this media artifact to grapple with these questions, this was a … draw.

Another echoed this perspective, that a meta-level project about process was important to herː

How do we make an intervention at a larger level? This project tries to do that. It considers the meta-framework of editing, not just about changing a singular entry, but how entries come to be. I don’t study Wikipedia or edit Wikipedia, but I’m invested in it. I use it. People use it. It's important to think about the ways that it is becoming a credible source.

Involving feminist academics in a pilot shaped the contours of the project. As academics, the process of thinking about the "gaps" and the way that process works unfolded into even bigger questions. As described below, the benchmark metrics were achieved through the pilot, but the real lessons and learning came from reflecting on the process of doing this pilot, rather than the absolute number of entries edited. In other words, yes, content was added that wasn't previously known to be needed -- we found 'unknown unknowns.' But more importantly, the process of involving non-Wikipedian academic experts, such as feminist academics who may not have the time to learn Wikipedia, was where greater learning took place, learning that is hopefully shared in useful ways in this report, the midway report, and the two learning patterns that I've developed out of the project.

Progress towards stated goals

Planned measure of success (include numeric target, if applicable)	Actual result	Explanation
Recruitment of a minimum of five subject matter experts to participate;	Achieved	As documented in the Gap Finding Project blog in the July update, I spent time selecting participants who would be both knowledgeable about the topic "feminism & technology," from interdisciplinary backgrounds, in instructional roles, and non-Wikipedians interested in Wikipedia. The recruitment process involved personalized invitations by email, in-person meetings, and phone calls.
Hosting a ~4 hour brainstorming session to identify content “gaps” in a gender-related subject or topic, with somewhere between 5 - 25 pages analyzed in depth during the session (depending on the breadth of each page);	Achieved, as documented in the midway report	We discussed and named the meta-level gaps of Wikipedia, then built a wiki repository and compiled links and pages to edit. This list was the focus of the last hour and a half of our brainstorming session. These readings were augmented by a close reading activity. In debriefs, the majority of the participants found the brainstorming session to be the experience that was the most meaningful for them in the project. The session was useful for community building and understanding the gap finding task. Recommendationsː Hold two brainstorming sessions instead of one. Devote the first session to discussing the perimeters of the project and beginning to explain how to identify gaps in content. Hold the second session shortly after and use it as a working session for participants to bring their materials to do close readings of pages Provide extremely clear instructions to the participants on how to do a close reading, such as the instructions provided in this learning pattern Present a critical introduction to Wikipedia, including the politics of Wikipedia, and how editing works currently since non-Wikipedians may not realize how many features on the platform are open to being edited (e.g. categories)
Recruitment of a university course that will use the materials; Expanding the lists in to practical action-items for the students that make sense for the course	Achieved, as documented in the midway report	Three project participants who were instructing courses in the fall choose to include a unit on Wikipedia in their classes. One instructor decided to assign her students content editing and use our list. However, there was not a topical match between the course and the brainstorming session. The course was on "critical media literacy' while our session focused on the broad topic of "feminism and technology." Because of the lack of a clear overlap between the entries brainstormed and the course content, many students did not choose items on the list. Recommendationsː Match the topic of a brainstorming session to a class prior to the session to make it most useful and focused Give very clear instructions to the participants of the brainstorming session what needs to be accomplished, e.g. providing a description of the course that will be using it or even a syllabus Invite the instructor to include the distributed editing project in their course at least three months prior to the start of the course, to better facilitate planning Work with a project manager such as WikiEd to facilitate the match-making; this was something that four of the participants said they would be interested in having Develop a network of instructors who are doing a distributed editing project on one campus through a grant or larger project, put into place a funding schedule that will enable them to plan, meet regularly, share resources -- a suggestion made on two separate occasions by two different participants
Training and working with work these student editors for the duration of the course. At least 5 of the identified gap topics (new or existing articles) edited by these student editors, with at least 2000 characters of content and 2 citations added per topic.	Achieved	The instructor designed a three week assignment that drew on WikiEd training materials to teach her students the contours of editing. She offered students the list created at the gap-finding brainstorming session as well as the close readings, as resources. However, she did not make using the list mandatory due to the focus of the course being on critical media literacy not on feminism and technology. Despite this mismatch, students did choose five of the identified gap topics (existing articles). Other students chose articles and added new material to Wikipedia that grappled with how ideologies of difference (including gender, race, and sexuality) operate in media culture, which is a critical lens connected to the ethos of this project. Students contributed more than 15,000 characters the entries that were specifically drawn from the brainstorming session, along with 15 references. Overall, students added -- and at times deleted and replaced -- more than 11,000 words in 16 articles. However, students also encountered hiccups. Some of these were avoidable. For instance, fandom, one of the pages that was extremely promising due to the close reading conducted, did not end up have the edits made to the page. The students just inserted their contributions into the talk page, requesting a semi-protected page edit request, because of a semi-protected status that had already been given to the page. Had the students been in a position to work for a longer period of time on the projects, or work more closely with an experienced Wikipedian, they could have gotten past the semi-protected status and intervened directly. However, due to the timing of the assignments, students did not achieve the necessary permissions to make edits to semi-protected pages prior to the end of the course. Thus their edits, and requests to make them, were not followed up with. Perhaps this experience helped them see that it can be difficult to edit Wikipedia as a newcomer. At the same time, the encounter also points to the difficulties that emerge when classroom content editing assignments come at the end of the quarterː students are invested in finishing their assignments but there's not necessarily the motivation to come back and follow up on their edits, nor always the time to get the support they need from experienced Wikipedians. Recommendations: Start on Wikipedia editing assignments earlier in the quarter, if possible Encourage students to reach out by email to volunteer Wikipedia editors or WikiEd connections in order to make transparent the pitfalls they may be facing Double check that close readings or brainstorms done are on pages that are pages that newcomers will be able to edit; or, ensure that students can edit articles that are protected; or, ensure students will have time during the course to receive responses from editing requests in order to complete their edits and their assignments
Staging a final presentation session which may include the return of the subject matter experts to reflect on how their ‘gaps’ were addressed through the editing work of the undergraduates;	Achieved, however only the project manager and WikiEd were able to attend, none of the subject matter experts attended the presentations	Due to scheduling constraints, none of the participants were able to attend the presentations.
A documented workflow for running a gap identification event and using the identified gaps in a classroom editing project	Achieved	This project has been documented in blog form, in the midway and final reports for the project, and through the creation of two learning patterns.

Global Metrics

Metric	Achieved outcome	Explanation
1. Number of active editors involved	2	The goal of this project was to involve new editors and inexperienced academic experts interested in Wikipedia.
2. Number of new editors	45	40 undergraduates, five academic experts
3. Number of individuals involved	47
4. Number of new images/media added to Wikimedia articles/pages	2	While the project focused on participation and articles written/improved, not images, two images were added by undergraduate course participants
5. Number of articles added or improved on Wikimedia projects	16	Forty students worked in groups of approximately three
6. Absolute value of bytes added to or deleted from Wikimedia projects	11,000 words added	See more details on the course's WikiEd dashboard page

Did your work increase the motivation of contributors, and how do you know?

In the hour-long debrief interviews I conducted, participants relayed that their participation in this project further encouraged them to think more about Wikipedia. Each said they wished there had been more time to devote to the project. Three of the participants said they would like to, and would plan to, include a larger unit on Wikipedia in their courses. They plan to include both meta-level analysis and content editing. Four said they would be interested in having WikiEd coordinate a gap-finding session that would work in conjunction with their courses.

Students who participated in this project conveyed in their response papers and in their final presentations that working on content editing on Wikipedia opened their eyes to the ways that sharing knowledge publicly raises the stakes. For instance, one student said that knowing her work could be viewed by anyone, and not just her instructor, gave her reason to spend a little more time on making sure it was quality. At this stage, it is difficult to say whether or not students who participated will become long-term editors.

Indicators of impact

This project involved working with five academic experts who did not identify as Wikipedians. All of the participants self-identify as women. While all are in academia, they are from a diverse set of backgrounds, educational histories, and their research areas of expertise focus on issues of diversity including gender/queer studies and critical race and ethnic studies. The project thus served to increase participation of underrepresented groups, and in the process, improve the quality of WikiProjects devoted to thinking about content gaps -- including gaps due to existing systemic biases.

The project also had an unintended impact. As documented in the midway report, on October 26, I invited a local activist to do a close reading of pages related to women and aircraft/aviation mechanics prior to a UWRCWikiLab editing session at the University of Washington. (I was among the organizers of Cascadia Wikimedia User Groups weekly Wikipedia Lab sessions at the UW Research Commons). Trained as an aircraft mechanic, the speaker was the leader of a local chapter of women in aviation mechanics organization. In her presentation, she commented on entries related to women in aviation mechanics and made editing suggestions, most of which were specifically related to her work with the organization and highlighted local press. Honored to be a guest speaker, she invited ten of her colleagues to join, none of whom had ever edited Wikipedia before. Inviting a local expert to present on existing gaps on Wikipedia prior to an editing session sheds light on gaps, brings new voices to Wikipedia, and promotes partnerships with local organizations. This models how partnering an editing event with a local organization can support the reach of Wikipedia, improve the quality of content, and increase participation.

Project resources

Project resources and documents are available on the project blog.

Learning

What worked well

I've created two learning patterns to document both what worked well, and what has been learned through out this project. Both of the learning patterns focus on the educational and outreach aspects of the project.

Using Wikipedia's gaps as feminist teaching tools -- this learning pattern elaborates on the six gaps that were identified at the brainstorming session

Engaging non-Wikipedian academic experts to identify content gaps

What didn’t work

This was a pilot project. Instances of awkwardness, breakdown, even failure, were expected as a part of the project. Many aspects of the project were deemed successes, in that the outcomes of the project synchronized with the predicted goals. But at the same time, there were instances of breakdown and times when the process did not go exactly as planned. Some of these pitfalls and breakdowns have already been documented in the "Progress toward stated goals" section of this report, where I specifically make recommendations based on each goal. However, there were also features of the project to learn from that went beyond the goals, which emerge out of reflection and the debrief sessions. I'll describe them hereː

Participation incentives

As discussed, participants were recruited through networks and given personal invitations to participate. There was no open call. Participants also signed a Letter of Engagement that laid out the contours of participating. Since participation levels varied among the five involved, in the project exit debriefs, I asked whether or not they would have, or would again, participate in this project without an honorarium, and asked for general feedback on why they participated and what they would change about the project were it to run again.

All said they really appreciated the gesture.

It's not the amount, but that symbolically, [the honorarium] says that we are valued.

I was intrinsically interested in this project, so yes in theory I would have done it without it. An honorarium is nice, respectful. We spent time working on this. It wasn’t my primary motivation for participating though.

Four out of five said they would have participated without an honorarium. These four participants explained that their level of participation had to do with their investment in the project, rather than the dollar amount they were receiving. "You would have had to pay me more if I were doing this for the money," said one, jokingly. The fifth participant hesitated at the question, then said she was unsure if she would have participated without the honorarium. She cited her busy fall schedule, which also significantly constrained her ability to participate fully in the project even with the honorarium. "I do not think I would have had signed on if it had not been for [the honorarium]. I had many competing things on my plate."

These responses suggest that the incentives for participating in such project may include monetary compensation, which is understood as a symbolic gesture that honors and values the time and expertise of the participants. But at the same time, participants should have some other reasons guiding their involvement, and may have other reasons for not wishing to be involved. For instance, another participant suggested that despite the honorarium, what made engaging with regularity difficult were time constraints that money cannot solve. Other pressures such as committee work, fellowship deadlines, conference schedules (fall is heavy on conferences) and lack of personal time made it difficult to engage. Moreover, there was also a fall out of facetime between the participants after the fall brainstorming session (a breakdown/challenge that I will discuss in the next section). What can be learned from this? Participants will be most involved when the project is directly related to the work that they are already doing, which is often instruction or research, and when there is a sense of community.

Sense of community

All of the participants said they felt the most connection to the project during the brainstorming session. In the weeks after, as the close reading project was developed, there was no face-time with other participants, nor any online engagement beyond email. The quarter was in full swing and many participants had significant time commitments -- including colloquiums, travel for conferences, job talks, conflicting teaching schedules, not to mention life outside of their jobs. That said, they all relayed it was more difficult to feel a sense of community with the other participants when the communication was only through email, Wikipedia pages, or the blog.

What were the recommendations?

One participant suggested developing a cohort program that would institutionalize distributed content editing and a focus on feminism among instructors.

Maybe a cluster of instructors who are all teaching classes from a feminist perspective ... What if we were all teaching classes and we could be a part of a working group, or a collective, that had a larger grant [that included all of us], that would incentive. We would all work together. I’d love to have three other people to talk through things as I am working through doing content editing with students. If I were teaching a class and had others [instructors] to meet alongside who were committed to the same project as me, with me, and we were all doing this [distributed editing and feminist gap finding] I would have engaged very differently in this. Absolutely I would do this project again and not for the money, but having a cohort or a research cluster of instructors working together on feminist content editing and feminist approaches -- that would be something.

More recommendations and ideasː

• Use open recruitment methods and personalized invitations to reach out to possible participants, in order to ensure that participants are self-selecting
• Ensure that participants have incentives to participate
• Consider institutionalizing or initiating a cohort program for instructors doing feminist related editing the same university (or in the same city)
• Meet at least twice with the group -- do a midway check in and second brainstorming session as well
• Use a social network to keep people involved. Of course this poses issues of access, since not everyone uses the same social network (e.g. Twitter or Facebook), but it could be a way to reach some people

Next steps and opportunities

This project has opened up many ways of thinking about editing and the involvement of academic experts who are not Wikipedians. Small suggestions and recommendations on how to improve the project, and things to consider, have been woven through out this report. For future projects, I would suggest the followingː

• Use the close reading activity in future edit-a-thons or community engagement activities
• Invite WikiEd to orchestrate match-making between instructors and brainstorm events
• Develop a grant or program that would support a small cohort academics in doing diversity-related editing in their courses

Think your project needs renewed funding for another 6 months?

Part 2: The Grant

Budget

Expense	Approved amount	Actual funds spent	Difference
Project Manager	$5,500	$4,875	$625
Subject Matter Advisors	$1,000	$1,000	0
Refreshments	$200	$93.78	$106.22
Discretionary Costs (space rental, copying, childcare if necessary)	$300	$125	$175
Total	$7,000	$6,093.78	$906.22

Remaining funds

Do you have any unspent funds from the grant?

• Yes - refreshment estimates were unused

• Amount distributed total = $6,125; Amount used total = $6,093.78; Difference between totals mailed by personal check to WMF on 1/28/16 = $31.22

Documentation

Documentation emailed to grants admin with attachments on 1/28/16

Confirmation of project status

Did you comply with the requirements specified by WMF in the grant agreement?

• Yes

Is your project completed?

• Yes

Grantee reflection

It's truly been an honor to work on this project. I had the opportunity to deepen relationships with fellow Wikipedians, with WikiEd, with academics I respect and honor, and with undergraduates. This is meaningful and inspiring. Thanks to being an IEG grantee, I was able to step into a leadership position and reflect thoughtfully on the process. The project accomplished both forecasted and unforeseen achievements. I am proud of the two learning patterns I wrote. Also thanks to this project, I had the opportunity to apply for and present at WikiConference USA. This was invigorating! I'd never met so many other Wikipedia editors and organizers in one place. That learning experience led to new connections, relationships, and understandings of the Wikipedia project, including developing a relationship with the Art+Feminism organizing team. My involvement in grappling with questions about media literacy and production, public knowledge projects, pedagogy, equality, gender, bias, pedagogy, platforms, and feminism -- and taking action as a form of questioning -- will be continuing and I'm more experienced to take on the next challenges that come my way. I am extremely grateful for the grants committee, and Wikipedia community, for seeing the value and importance of this project.