Wiki Education Foundation/COVID-19 Project Evaluation

Providing the general public with crucial information during a national emergency: Wiki Education’s COVID initiative

In 2020 and early 2021, Wiki Education undertook a campaign to improve the quality of information related to the impacts of the COVID-19 pandemic. Over several months, Wiki Education ran a series of courses affiliated with our Wikipedia Scholars & Scientists program (see previous evaluation reports of the pilot of this program here and here). To our knowledge, this initiative was the first effort ever to engage subject-matter experts during a national emergency to systematically improve Wikipedia’s coverage of a topic crucial to the general public.

Our success in this project clearly demonstrates targeted investment to improve content areas on Wikipedia works. Through our Scholars & Scientists program, we’re able to recruit and train subject matter experts to make meaningful contributions to articles related to a particular topic area.

Why we undertook the project

“	By adding scholarly information to Wikipedia, 7+ billion people will have access to reliable information as opposed to fake news, pseudoscience, and conspiracy theories.	”
— Magdalena Luca, Ph.D., Professor of Mathematics and Statistics, MCPHS University

In March 2020, the World Health Organization declared a global pandemic. Wikipedians were busy improving articles related to coronavirus, especially the main article on the pandemic. Quality varied widely, however, across articles related to state and regional responses to the pandemic. Articles about states like New York or California with vibrant existing editing communities were already pretty well developed, but those on states with smaller editing bases had little information. While local media covered that day’s COVID news, it was hard for citizens to get a high-level overview of the situation in their state or region. Wikipedia could provide that overview — if editors were engaged in all states and regions.

Wiki Education stepped in to fill this void. As an organization that has spent a decade building a network of academics physically located across the United States, we were uniquely qualified to reach out to experts in nearly every state. And our existing Scholars & Scientists program had previously demonstrated success in teaching experts to add their knowledge to Wikipedia articles. Thanks to the generous sponsorships provided by one of our funders, we set out to use our network to broadly improve Wikipedia’s coverage of state and regional responses to the COVID-19 pandemic through a series of courses that ran between April 2020 and February 2021.

Our key question was: Can we empower subject matter experts across the United States to meaningfully improve the quality of English Wikipedia’s coverage of state and regional responses to the COVID-19 pandemic, providing Wikipedia’s readers information they’re seeking?

What we did

Wiki Education ran a total of six COVID-focused courses. Each course was 6 weeks long, with subject matter experts recruited through our networks. Each course was between 15 and 19 participants, who met with our course instructor each week via Zoom. Our Dashboard software walked them through training modules and exercises outside of class and encouraged participants to engage on-wiki as they improved articles related to the pandemic. We maintained a Slack channel for participants where they could get real-time support as they completed assignments and editing tasks each week. During synchronous class sessions, our course instructor walked participants through editing best practices and facilitated discussions around participants’ experiences, observations, and questions. Through these different support channels, participants learned about article development, the Wikipedia community, and how they can leverage their expertise to improve articles.

Recruiting participants

“	It was the best experience I have had to date learning through a video-based class.	”
— Nicole Knapp, COVID-19 course participant

When we decided to run Wiki Scholars courses to add COVID-19 information to Wikipedia, we knew we would need to recruit subject-matter experts across a range of disciplines. After all, the pandemic has impacted so many people and institutions. Over the span of 10 months, we recruited participants for six COVID-19 Wiki Scholars courses. We had a lot of interest in this work, connecting with 200 applicants overall. Several applicants were especially eager to join, as 42 people applied to more than one of the courses, either because they weren’t accepted in the first round or because their schedule did not work with our selected synchronous class time. Running the same course more than once reduced the resources we needed to put into recruitment for subsequent courses, as we already had interested and qualified scholars to invite. This will inform our future course scheduling, and we’ll continue running course series like this one.

So how did we connect with these scholars and scientists to bring them into our courses?

Tapping into our extensive academic database

We have been connecting with scholars and scientists in academic settings — at academic conferences, on university campuses, through partner organizations, and through online sign-ups — for nearly a decade. Wiki Education has always prioritized collecting contact information from anyone who shows interest in getting involved with Wikipedia, as we offer several paths to joining this community. When we announced the COVID-19 Wiki Scholars courses, we sent emails to our existing network, inviting them to apply.

Of our 99 COVID-19 Wiki Scholars, 46 participants (or 46%) were contacts we’d met prior to running these courses. We’ve been in touch with several of them since 2016, and the COVID-19 courses finally provided an engagement opportunity for them to sign up and learn how to edit Wikipedia. This emphasizes the importance of maintaining a network and of offering different ways for them to get involved.

Leveraging contact referrals

Over the years, we’ve repeatedly confirmed that the most successful way to bring in new program participants is through word-of-mouth. Our existing network of pro-Wikipedia academics, librarians, researchers, etc. helps bring in new participants, as word-of-mouth and personal recommendations are meaningful to people who may have never thought otherwise that they could (or should) edit Wikipedia.

Of our 99 COVID-19 Wiki Scholars, 20 participants (20%) shared that a colleague recommended they apply to the course. Contact referrals are an easy win for Wiki Education, as they’re a persuasive method of bringing new scholars into our network. We know how important it is to facilitate these, so we often contact our past program participants to ask them to share their experience on social media, mailing lists, or in one-on-one recommendations.

In one instance for the COVID-19 course recruitment, a long-time contact who has never participated in a Wiki Education program sent the course application to a mailing list for her university’s science writing program. In the end, we accepted 7 of the applicants who came through — all graduate students in the sciences who went on to add science content to Wikipedia.

Helping prospective participants “see themselves” with targeted marketing

Recruiting participants into this program requires marketing, and even though the COVID-19 Wiki Scholars did not have to pay for their seats, they did have to dedicate several hours a week to this work. Thus, one of our primary concerns is to help prospective participants see that this work is worthwhile and meaningful. By the time we were recruiting for the final courses, we had run several related courses. We were able to share case studies about work others had done to improve COVID-19-related articles, and this helped others see that they have knowledge to contribute to Wikipedia. This learning reinforces the importance of case studies and demonstrating to prospective Wikipedia editors that their research skills, writing skills, and subject-matter expertise can make a positive impact on public access to information rather than simply telling them.

Since the COVID-19 content spans so many possible academic disciplines, we used our powerful database to send targeted recruitment emails based on contacts’ area of study. Some example email language we used to describe examples of Wikipedia content they may work on during the course:

In our newest course, participants will add information to Wikipedia about impacts of the COVID-19 pandemic. You may choose to focus on your local government’s response to the pandemic, the federal administration’s actions, the impacts of sheltering-in-place on cultural institutions, cross-cultural perspectives on epidemics, the hazard of stigma toward "others" during a pandemic, or any topic that you feel intersects with your own studies and the COVID-19 pandemic. – partial marketing email to anthropologists

In our newest course, participants will add information to Wikipedia about impacts of the COVID-19 pandemic. You may choose to focus on your local government’s response to the pandemic, the federal administration’s actions, the impacts to higher education, socioeconomic disparities among children experiencing distance learning, insecurity for faculty, racial justice through the lens of a public health crisis, or any topic that you feel intersects with your own studies and the COVID-19 pandemic. – partial marketing email to sociologists

Spreading the word through partners and other Wikimedia affiliates

This brings us to one of the most effective way we bring new participants into our programs, and the same holds true for the COVID-19 Wiki Scholars courses. Over the years, we’ve cultivated relationships with academic associations, university departments, and other organizations eager to bring free knowledge to the public through Wikipedia and Wikidata. In preparation for the COVID-19 courses, we reached out to several partners, asking them to share the opportunity with their members. Scholars and scientists are extremely familiar with their member organizations, making them more likely to read emails/newsletters/other publications from them than from Wiki Education. Here are some of the partners who shared the COVID-19 Wiki Scholars course information:

• Midwest Political Science Association – They recommended members participate to add content about their regional government’s response to the pandemic.
• Louisiana State University, Communication Across the Curriculum – They emphasized these courses as a way to apply their science communication studies in real life.
• National Science Policy Network – They encouraged members to apply and add science policy related to COVID-19 to Wikipedia.
• Wikimedia affiliates – We supported members and employees of Art+Feminism and other affiliates, giving them the chance to expand their Wikipedia knowledge in a structured setting, which they will pass on to other newcomers to the movement.

Key Learnings

• Running the same course in multiple cohorts helped recruitment for subsequent courses, since we could invite qualified applicants who couldn't participate in earlier courses due to scheduling conflicts. Additionally, case studies from the first courses helped boost applications for additional courses.
• Recruiting academics is an ongoing process, so we must prioritize collecting contact information from anyone interested in open knowledge and Wikipedia so we can continue tapping into our extensive network when a new opportunity arises.
• Collaborating with partners and like-minded institutions is the most effective way to find program participants who are a great fit.

Recruitment trends

Since we recruited participants into five COVID-19 courses over 10 months, we were able to track trends related to our recruitment efforts and the number of applications we received.

The urgency of getting information to the public is an effective motivator

The COVID-19 Wiki Scholars course interest mirrored the number of COVID-19 cases in the United States. While cases and media coverage were high, we saw overwhelming interest. Our target audience of academics and researchers are incredibly busy professionals, and we often see an increase in interest when prospective participants see urgency in their participation. We’re better able to demonstrate not only that this work is necessary, but that it’s necessary right now, and that the subject-matter experts are crucial to our success. This reinforced our past learning that tying the course topic to a current event or urgent matter within the discipline is one of the best ways to persuade subject-matter experts and academics to carve out time in their busy professional and personal lives to do this work.

“Zoom fatigue”

We received more applications from qualified applicants for the first and last sets of courses. In other words, the recruitment dropped off for the courses we ran from July–October 2020. We still ran full courses that made a positive impact to Wikipedia, but we were curious about the slight decline. Our Wiki Scholars courses have always been virtual and held over Zoom. That served us well during the pandemic, as scholars could participate from anywhere to learn how to contribute to Wikipedia. However, the summer courses likely correlated with our target audience’s burnout and “Zoom fatigue,” making them less enthusiastic about committing to an optional project like the Wiki Scholars course when they could spend that time away from a screen, outdoors, or focused on curriculum development for a disrupted academic term in fall 2020.

Better tools make us more effective and efficient

In December 2020, we began using a new technical tool for mass emailing our network. Pardot is an add-on within Salesforce, our Customer Relationship Management software, that aids organizations in improving marketing reach. One of the biggest upgrades for our work is that the deliverability of emails is high, so we could notify our own massive academic network about the upcoming COVID-19 Wiki Scholars course opportunities. Additionally, we were able to filter our audience by discipline, creating more targeted recruitment emails about the types of Wikipedia articles in need of work. For example, we asked those in fields like medicine and other STEM fields to improve articles directly related to the virus or medical conditions, while we asked our contacts in the social sciences to write articles about the societal impacts of the pandemic. Pardot drastically simplified this process compared to our previous tools.

Key Learnings

• Showcasing urgency in participation helped generate a larger pool of applicants.
• Using technical tools to create more targeted recruitment messages helped boost applications.

The subject-matter experts

Pie chart depicting COVID-19 Wiki Scholars’ subject matter expertise and research background

For the COVID-19 Wiki Scholars courses, we recruited a broad range of subject-matter experts into the initiative, since we saw an opportunity to share information related to a number of fields. To date, these course participants had the most diverse research backgrounds of any course we’ve run so far.

Librarians

Librarians made up the biggest cohort in the COVID-19 Wiki Scholars courses, representing 18% of the participants. This is unsurprising, as Wiki Education has a vast number of librarians in our database, meaning we contacted thousands of librarians to invite them to apply. Librarians have extensive backgrounds in research, practice critical media literacy in their work every day, and comprise a large number of Wikipedia editors. We know that librarians have always been excellent participants in Wiki Education’s programs, and we were thrilled to see the interest from librarians and archivists who had been in our database for years prior to this initiative.

User:Catadel’s addition to the Johnson & Johnson COVID-19 vaccine article on Wikipedia

• One librarian from the University of Pennsylvania channeled her studies in Library and Information Science into improving the Wikipedia article about the Johnson & Johnson vaccine. In addition to other edits, she added information about the expectation that the Johnson & Johnson vaccine, as compared to the Pfizer and Moderna vaccines, would play an important role in low- and middle-income countries thanks to its ease of transportation, storage, and only needing to administer one shot.

User:Nick Lee 23’s contribution to the article about the Impact of the COVID-19 pandemic on education.png

• Another participant works as the Education Librarian at the University of California, Irvine, focused on student success for traditionally under-supported groups (e.g., Dreamers, former foster youth, etc.). She joined one of our COVID-19 Wiki Scholars courses and brought her subject-matter expertise to the Wikipedia article about the impact of the COVID-19 pandemic on education. Specifically, she documented the rise in private school enrollments in the United States during the pandemic, which widened the inequality gap among students like the ones she supports in her daily work.
• One graduate student in the Library and Information Science program at Arizona State University joined a course and created the article about the COVID-19 pandemic in the Navajo Nation. See below for examples of his excellent work documenting the pandemic within the Navajo Nation.
• A participating archivist from New York University libraries tackled creating a new Wikipedia article about the impact of the COVID-19 pandemic on Native American tribes and tribal communities. Her extensive experience as a researcher of primary and secondary sources positioned her to take on this ambitious article, adding information from 33 academic and news sources to Wikipedia.

Medical and public health professionals

User:Azb24’s contributions to the Long COVID article

• One of our COVID-19 Wiki Scholars with a background in public health researches health communication and the effects the media and/or technology have on public health literacy. As a part of the course, she used her new Wikipedia-editing skills to expand the article about media coverage of the COVID-19 pandemic. She added sections about media coverage in Germany and Sweden and cited sources about media coverage contributing to the politicization of the pandemic.
• Another scientist with a focus on public health improved the article about Long COVID, the potential condition of long-term effects resulting from a COVID-19 infection. Specifically, she added information about the different studies showing long-term symptoms for children.

Other scientists

• One COVID-19 Wiki Scholar applied to our final course and described himself as a Neurodegeneration researcher who recently received a grant to study the effects of CoV-2 on neurons. He brought this expertise to Wikipedia and the public, starting a brand new article about the impact of COVID-19 on neurological, psychological and other mental health outcomes. In this article, the COVID-19 Wiki Scholar summarizes effects of COVID-19 on taste and smell, distinguishes between acute and chronic symptoms so far recorded for patients with COVID-19, and covers the mental health symptoms among caregivers. We’re grateful our course gave him an outlet to share this recent research with the public, compiling and disseminating 20 scientific and medical sources.

Other scholars

User:Jkpizza’s edits to the Impact of the COVID-19 pandemic on prisons article

• A historian who studies prisons joined this initiative worked to improve the article about the impact of the COVID-19 pandemic on prisons, adding data about outbreaks in the United States.

Key Learnings

• The definition of "subject-matter expert" varies from course to course. In order to be most successful, participants need an extensive research background and writing skills. Identifying participants within relevant disciplines is necessary for each course we run.
• The emphasis on local knowledge benefited the article creation and refinement process in these courses. The nature of documenting this pandemic put an emphasis on local knowledge which has varied from state to state.
• In addition to capturing local knowledge, these courses created a space for participants to create articles or article sections about overlooked and underrepresented populations and topics.

The curriculum

“	I entered the course curious about Wikipedia, but leave an advocate and interested in the community.	”
— Katie Pfaff, Science and Medical Writer

At the start of the project, our typical course was 12 weeks long and included deep discussions of the Wikipedia community, more academic discussions of policy, and several review cycles before work went live. To respond to the needs of the pandemic, we adapted that curriculum to be a more intensive 6-week course, front-loading trainings on how to edit and getting people directly in to editing in the article namespace early in the course. Before the first course, the Wikipedia experts on our staff evaluated all of the US state, city, territory, and regional-focused pages about the pandemic to create a spreadsheet allowing participants to relatively quickly see which pages need their attention the most. We also took notes about what kinds of visuals were common, typical heading structure, and other elements in articles to remove that part of the work participants usually did.

This standard curriculum created for the first course was replicated across the five subsequent courses, with minor modifications. The curriculum is available on the timeline tab of a COVID Wiki Scholars Dashboard.

Course sessions

Similar to other Scholars & Scientists courses, these meetings typically lasted an hour and consisted of a mix of demonstrations, conversations, and question and answer exchanges, addressing individual concerns. Beyond the curriculum, the article selection process focused on state-by-state responses to the COVID-19 pandemic. This state response frame grew to incorporate communities, cities, and prominent people and legislation affecting and affected by the pandemic. Due to the evolving nature of the pandemic responses, conversations about how to determine significance and expand articles were common. More universal concerns about making edits to important articles and learning the technical aspects of Wikipedia also pervaded all of the courses.

While five of six of the courses were unqualified successes, with participants adding significant amounts of high-quality content, one course, our third, had less engagement than the others. More than two thirds of the participants never made substantive contributions to Wikipedia. We have identified three dynamics that likely contributed to this section having less engagement:

• Scheduling – More than anything else, scheduling posed a challenge for this course. Most of our participants came from higher education in this cohort. Uncertainty around reopening in person for the fall semester resulted in several emergency meetings, resulting in missed meetings, and no doubt, fewer opportunities to write and complete assignments.
• Creating vs refining articles – This course took place three to four months into the pandemic. The first wave of state-response articles was already written. Contributing to these articles became more complicated. Condensing timelines, choosing what new updates to include, and several significant national narratives to fit into articles (impact of racial justice marches on the pandemic, fluctuating rules, confusion around reported numbers, treatment updates, to name just a few)
• Wikipedia policies – This group was particularly concerned with community guidelines about using local references (city, town level) which led to several discussions about what constitutes a reliable reference. Due to the national conversation about misinformation and the centrality of equity in all of these discussions, these conversations were appropriate to explore. However, they did not lead directly to contributions.
• Survey results from this group point to an satisfying course experience for the participants, which may indicate other ways of evaluating the success of a course, especially if participants are drawn to improving Wikipedia policies rather than just content.

Key Learnings

• A six-week bootcamp style curriculum still had great outcomes to Wikipedia.
• In spite of a more flexible course structure, attendance still fluctuated due to everyone adapting to their own pandemic environment.
• Although these courses prioritized article creation and improvement, course participants wanted to devote time to the origin and improvement of Wikipedia policies, community norms, and the editing experience.
• The process of refining an article is a more advanced topic that many participants were comfortable with exploring. A modified curriculum or longer course may be a good way to improve these contributions.

Outcomes to Wikipedia

All told, 99 experts participated in this six courses. These editors added 132,000 words to 252 Wikipedia articles, which were viewed more than 16.3 million times. Participants also created 20 new articles.

Participants edited articles on:

• “COVID-19 pandemic in” 32 different U.S. states (plus D.C.)
• “COVID-19 pandemic in” 7 different countries
• “COVID-19 pandemic in” 6 different regions (e.g., San Francisco Bay Area, New York City, Navajo Nation)
• “Impact of the COVID-19 pandemic on” 10 different topics (e.g., education, migration, people with disabilities, the environment)
• COVID-19 testing, Face masks during the COVID-19 pandemic, the Johnson & Johnson COVID-19 vaccine, Misinformation related to the COVID-19 pandemic, Social impact of the COVID-19 pandemic, and more.

“	I really enjoyed learning about the state that I was editing. It made me feel way more knowledgeable about the pandemic and how to contribute factual information to the public on the topic.	”
— Megan Damico, Environmental Health Sciences PhD-candidate at UNC-Greensboro

At the start of this project, we focused more on articles like the COVID-19 pandemic in Louisiana. Louisiana, along with New York and Michigan, was one of the states that was hit very hard early in the pandemic, and students in four of the six COVID-19 Wiki Scholars courses have contribute to the article. A participant in the first course made major additions to the article and remains the third-largest contributor to the article article text at the time of writing. An entire section of the timeline remains her work. A participant in the fifth course is the second largest contributor to the article, responsible for most of the timeline from July to January, much of the government response and the entire section on the section detailing the impact of the pandemic on the state’s economy.

The COVID-19 pandemic in New Hampshire is another article which received major contributions from course participants across the sequence of courses. The second, third, and sixth largest contributors to the article were from the sixth, first and second courses respectively. Much of the “Government response” sections for March, May and June 2020 were written by a participant in the first course. Participants in the second course rewrote the lead section, expanded the timeline, and adding some archive links to some citations (which is important, because online information can be ephemeral). The participant in the sixth course filled in part of the timeline from earlier months that had been left blank. They also removed information about the size of the crowd at a NASCAR race over the summer that may have seemed significant when it was added but had little lasting notability.

Day-by-day recitation of local responses in articles (a feature of early course contributions) made more sense when we thought the pandemic would be over soon, and keeping people up to date on was the most important thing. As the pandemic continued far longer than initially anticipated, we starting pushing people toward broader impact articles, which resulted in the creation of articles like the impact of the COVID-19 pandemic on people with disabilities and impact of the COVID-19 pandemic on Native American tribes and tribal communities. As the pandemic wore on, and most people stopped following the daily case numbers, the focus on broader topics also allowed participants the opportunity to contribute to some aspect of the pandemic that was closer to their own area of study or specialization.

Knowledge equity

“	Once you realize the number of people who use Wikipedia for information and/or entertainment, it quickly becomes evident the profound impact this resource has on the formation of our society. What and who gets covered and linked to plays a large role in how users ultimately view the world around them. So, if pages on COVID-19 address the disease’s impact on sports, but not people who have been imprisoned; if pages on central topics in astrophysics link to only white men as important influences; if many in our society cannot find representations of their own identities within its pages -- what does this mean for what we believe to be 'true' about the world. Editing is a meaningful contribution to society.	”
— COVID-19 course participant Tasha Bergson-Michelson, Instructional and Programming Librarian, Castilleja School

Our impact wasn’t just focusing on improving the quality of content related to state and local responses of the pandemic; it also, like everything we did, had elements of knowledge equity woven into it. More than some other themed courses we have led in the past, these courses represented experts, volunteers, students and professionals. Pulling together a group with diverse backgrounds presents a very unique opportunity for equity in terms of authorship and creating knowledge. We encouraged participants to identify opportunities to improve Wikipedia’s equity in relation to the encyclopedia’s coverage of COVID.

An example of this is the article on the COVID-19 pandemic in the Navajo Nation. The Navajo Nation was one of the hardest-hit regions in the United States, and early in the pandemic had a higher per capita rate of COVID-19 infection than any US state. Two participants in the first COVID-19 Wiki Scholars course decided to collaborate on the article and worked together to gather sources and discuss the appropriate scope of the article. As soon as the article was moved to mainspace, other editors joined in to expand and update the article, including a participant in the second COVID-19 Wiki Scholars course. The article appeared in the ”Did You Know…?” section of Wikipedia’s Main Page on July 7, 2020. Although editing has slowed significantly since September — which is when the Wikipedian who was most actively updating the article stopped editing Wikipedia — the community has continued to update the article.

Participant Nisa Muhammed, Assistant Dean for Religious Life at Howard University, took the fourth course we offered. “I was prompted to start editing Wikipedia when I learned that most editors were older white men. That disturbed me as a Black Muslim woman because that represents a narrow perspective,” she told us. “The many diverse readers of Wikipedia should have an equally diverse cadre of editors. I edited an article on Blacks and COVID-19. I chose this article because my community was dying at greater rates than any other community and Wikipedia had no article to represent how Blacks were affected by the pandemic. More classes like this need to be offered and targeted to diverse audiences. These perspectives are crucial to building a better world that reflects the greatness America has to offer.”

Key Learnings

• What was appropriate at the start of the pandemic (day-by-day indications of local responses) became less relevant as months went by; in response, we adapted curriculum to push for more wholistic “impact of” articles, which people also read in great numbers.
• Focusing on how equity intersected with COVID-19 was something important we were able to bring to Wikipedia.
• In lieu of conferences or other traditional meeting spaces, these courses served as a space for participants to connect with colleagues and advance their expertise around a specific goal.

Outcomes to program participants

We surveyed participants about their experiences in the course after they’d completed the curriculum. A total of 35 people responded to our survey, or a 35% response rate.

Overall, 100% of respondents said the course met their expectations. 34/35 participants (or 97%) said it was worth making time for among existing professional commitments, and 34/35 also said they would recommend the course to a friend or a colleague. While a 35% response rate is generally good for a survey, it is likely that the most engaged participants were the ones who responded to the survey.

“	It helped to hear others’ experiences and listen to their questions. Knowing what other people were encountering was incredibly valuable. Specific to the time we’re in, it was also valuable to have social contact for an hour with others since we’re all isolated due to the pandemic.	”
— Hayley Heaton PhD, Research Associate, University of Michigan

We also asked a series of open-ended questions of participants. Some responses are displayed in pull quotes for the relative sections of this report. Common themes that emerged from these responses were:

• It’s worth remembering that all of these course participants managed to take these courses on top of their regular responsibilities (work, family, personal health, and beyond). While participants were writing about a pandemic, several had to pull back or drop the class because of sudden pandemic-related changes to their lives.
• These courses are a unique experience and a chance for scholars to collaborate across disciplines, regions, and career levels. This research and writing experience is not common in the best of times, let alone during a global pandemic. We asked participants to commit at least four hours of work per week to their coursework. As busy professionals, this is a big commitment they make in order to learn how to join the Wikipedia community, especially when personal lives and work environments were upended by the pandemic. Still, our participants reported a sense of community and even distraction in the courses.
• In our 10+ years of experience in bringing new editors to Wikipedia, we’ve often heard outsiders question how content is created, who maintains Wikipedia’s content, and who “approves” new edits. Though people tend to understand anyone can edit Wikipedia, they rarely question who that is or what the process looks like. In our Wiki Scholars courses, we not only put Wikipedia policies and technical skills front and center, but we always frame policies and discussions through the lens that a community of practice works together to build the encyclopedia. We make a concerted effort to highlight and praise the volunteerism of Wikipedia while revealing content gaps and the need for better diversity on the projects. Participants share that they’re pleased to learn more about the players, behind-the-scenes work, and how we resolve disputes, and the COVID-19 Wiki Scholars courses were no different.
• Wiki Scholars courses provide a starting point for new editors, and they can implement their new skills beyond the course. We see these training courses as an opportunity to bring subject-matter experts into a comfortable and comforting community as we guide them through the editing process. Ideally, they will take their experiences and continue engaging with Wikipedia, either as an active editor, an instructor who assigns students to write Wikipedia articles, or as a community organizer who gets involved with editing events and Wikimedia affiliates or user groups. According to our COVID-19 Wiki Scholars, they don’t see the end of the course as the end of their time as a Wikipedian; 34/35 survey respondents said they plan to continue editing Wikipedia.

Key Learnings

• Some participants were directly affected by the pandemic while taking the course; our flexibility in supporting participants helped them contribute as much as they could.
• Our participants felt like the course gave them the skills they need to edit Wikipedia while also understanding a bit about the broader community.
• Due to the quick-changing nature of writing about the pandemic, participants felt their contributions were especially helpful.

Conclusion

Throughout the last year, we’ve succeeded in our goal to empower subject matter experts to improve Wikipedia’s coverage of an international emergency. Wikipedia’s quality has improved substantially, and the experts themselves overwhelmingly had positive experiences. Our 99 program participants edited 256 articles (including 20 new articles), which have been read more than 17 million times since the start of the pandemic (as of March 2021).

“	I had no confidence about digging in or teaching myself. Now, I understand where to start and the role of content experts like myself in Wikipedia.	”
— Robin Kolodny, Professor and Chair, Department of Political Science at Temple University

One of the challenges of writing about contemporary events unfolding by the hour and day is it’s hard to get a sense of the relative importance of something in the context it’s happening. As a result, many articles related to the pandemic have ballooned in size, and have imbalances in coverage within the article. What is important month-to-month as the pandemic has continued also changes drastically. We devoted some class time to discussing how unreadable some articles are becoming. But it is more difficult to pare down articles than it is to add to them. Removing content from articles while preserving their flow can be challenging. Future courses could focus more on the art of editing instead of adding to COVID-related articles.

Our final courses began in early January 2021, when vaccines were still extremely limited in supply. Vaccine distribution opens a new aspect of the pandemic that needs coverage on Wikipedia. And while our work focused on many areas of equity as it related to COVID, we are confident there is more that could be done in that aspect of coverage.

Our key question was: Can we empower subject matter experts across the United States to meaningfully improve the quality of English Wikipedia’s coverage of state and regional responses to the COVID-19 pandemic, providing Wikipedia’s readers information they’re seeking?

Nearly one year later, we can firmly attest the answer is yes. By improving 32 state, 6 region, and 10 impact-of COVID articles on Wikipedia, our participants have provided critical information that was otherwise missing — information that has been viewed by millions of readers. We are deeply grateful to our excellent staff teachers, who guided these experts to making valuable contributions to highly read pages; to our course sponsor who made offering these course to participants for free possible; and to the participants themselves for dedicating hours of their live — during a global pandemic! — to improving the world’s understanding of that emergency.

*    *    *