Open main menu

Grants:APG/Proposals/2017-2018 round 1/Wiki Education Foundation/Proposal form

Proposal by Wiki Education Foundation to support its annual plan with $750,000 USD.

In 2018, the Wiki Education Foundation seeks to improve the quality and quantity of information available on Wikimedia projects by bridging the gap between academia and the Wikimedia projects. We will do this through three main programs: (1) Our Classroom Program, in which we will support more than 16,000 university students in the United States and Canada as they edit the English Wikipedia as a class assignment; (2) Our Visiting Scholars Program, in which we facilitate providing university logins for existing Wikipedia editors, giving them access to sources that would otherwise be behind paywalls; and (3) Our Wikipedia Fellows pilot program, in which we train members of academic associations to edit Wikipedia articles in their areas of subject matter expertise.


Comment on Wiki Education Foundation's proposal here


Return to the community review page.

We ask that you please don't make any changes directly to this page after the proposal submission date, but comment on the discussion page.


If you need to review the edit instructions you will find them in the edit intro.

  • Use this form if you are eligible to submit a proposal in the current round, to request funding in the current round.
  • This form must be published on Meta by the proposal submission date for each round to be considered, and no changes may be made after the submission date without staff approval.
  • This form complements your organization's annual plan, detailed budget, and strategic plan. It won't be considered complete without a detailed budget and strategic plan.
  • Organizations may apply for a funding period that does not exactly match their fiscal years. Follow the instructions carefully to understand what time period each question refers to.
  • Refer to the framework, guidance from the Board, and the FDC's recommendations before using this form, so you have an understanding of the Annual Plan Grants process.
  • Please Email FDCSupport@Wikimedia.org with questions about using the form.

A few terms used in the form:

FDC proposal form terms Wikimedia terms Learning & Evaluation terms

OverviewEdit

1. In order to support community review, please provide a brief description of your organization's work in the upcoming funding period.

In 2018, the Wiki Education Foundation seeks to improve the quality and quantity of information available on Wikimedia projects by bridging the gap between academia and the Wikimedia projects. We will do this through three main programs: (1) Our Classroom Program, in which we will support more than 16,000 university students in the United States and Canada as they edit the English Wikipedia as a class assignment; (2) Our Visiting Scholars Program, in which we facilitate providing university logins for existing Wikipedia editors, giving them access to sources that would otherwise be behind paywalls; and (3) Our Wikipedia Fellows pilot program, in which we train members of academic associations to edit Wikipedia articles in their areas of subject matter expertise.


2. Name, fiscal year, and funding period.
  • Legal name of organization: Wiki Education Foundation
  • Organization's fiscal year: 07/01/2017–06/30/2018
  • 12-month funding period requested: 01/01/2018–12/31/2018
  • Currency requested: US Dollars
  • Name of primary contact: LiAnna Davis, Deputy Director


3. Amount requested.

Table 1

Currency requested US$
Total expenses for the upcoming year $2,823,819 $2,823,819
APG funding requested for the upcoming year $750,000

Classroom Program: $400,000 (40% of total program cost)
Visiting Scholars Program: $200,000 (74% of total program cost)
Wikipedia Fellows Pilot: $150,000 (56% of total program cost)

$750,000

Classroom Program: $400,000 (40% of total program cost)
Visiting Scholars Program: $200,000 (74% of total program cost)
Wikipedia Fellows Pilot: $150,000 (56% of total program cost)

Amount of funding received from WMF for the current year $99,996[1] $99,996[1]

Table 1 notes:
[1] We received a $99,996 Simple APG that covered the period 1 May 2017 to 30 September 2017.


4. How does your organization know what community members and contributors to online projects need or want? Does your organization conduct needs assessments or consult the contributors and volunteers most involved with its work?

More than half of our staff are experienced volunteer contributors to the Wikimedia projects, especially Wikipedia and Wikimedia Commons, and all programs staff interact regularly on the English Wikipedia. Our ongoing engagement with Wikipedia in both our work and volunteer capacities gives us a solid footing for all organization discussions regarding the Wikimedia community. Nine of our staff have graduate degrees, most have experience teaching college courses, and three have Ph.D.s, meaning we are also extremely well versed in the academic community we serve.

For our Classroom Program: At the end of each term, we conduct an instructor survey, which we send to the 300+ people who teach with Wikipedia each term through our Classroom Program. In the survey, we ask them questions about the support they received from us, the support their students received from us, what additional support would be helpful, and how our Dashboard software worked for them. We have done surveys in the past with student editors as well, but they require us going through an Institutional Review Board, which adds a lot of complications. Instead, we have incorporated lightweight requests for feedback at the end of student training modules, and adapt our support based on their responses. When we develop new materials (such as our discipline-specific handouts), we consult experienced editors in those disciplines through WikiProjects and individual outreach. When an existing editor raises concerns about a student's editing on the Education Noticeboard or through another venue, we track this as an "incident", work to resolve it, and then address whether we could have prevented it through developing better materials. We have an ongoing relationship with the English Wikipedia community through which they know they can provide feedback, make requests, and raise issues knowing that we will work to understand and respond. We monitor students' interactions with the community, which gives us an idea of which individual issues need to be addressed even if editors don't report them, and allows us to see patterns that inform how we develop our support infrastructure. Finally, we track the questions we get from student editors via ask.wikiedu.org and email, and take common questions into consideration as we plan out what additional resources we need to modify or develop.

For Visiting Scholars: The foundation of the Visiting Scholars program is a direct response the Wikipedia community's frequently expressed need for access to high-quality research resources. We've solicited feedback from the community and program participants both on-wiki and off, including needs-finding conference sessions at WikiConference North America and Wikimania.

For Wikipedia Fellows: We will engage in regular dialogue to see how the pilot is working for participants, and to help resolve any issues that come up.

5. Please provide a link to your organization's strategic plan, and a link to your separate annual plans for the current and upcoming funding periods if you have them.

Our Annual Plans for the three years of our existence are all available on our website. Our last strategic direction, developed in collaboration by our board and senior leadership during a 6-month strategic planning exercise in the first half of 2015 (see our Annual Plan for 2015–16), expired in 2017. We had initially planned to launch the next round of strategic planning in 2016 but decided to postpone the process to 2017, which allows us to sync our thinking with the new Wikimedia movement strategy. Board and staff started working on our new strategic plan in July 2017. At the end of October 2017, we'll have an in-person meeting of our board and senior staff in order to set the course for the next three years. We expect our board to confirm our new strategy during its in-person meeting in January 2018.


Financials: current yearEdit

The purpose of this section is to give the FDC an idea of how your organization is receiving funds and spending funds toward your current plan. Your current funding period is the funding period now in progress (e.g. 1 January 2016 to 31 December 2016 for most organizations).


Financials for the current funding period [1]
Revenues or expenses Planned (budgeted), total Planned (budgeted), YTD Actual, YTD [2] Projected, total
Revenues $2,738,000 [3] $350,000 $403,702 $2,738,000
Expenses $2,427,800 $535,757 $412,531 $2,427,800


Table 2 notes:
[1] The numbers in Table 2 reflect revenues and expenses of our current fiscal year 2017–18.
[2] As per September 28, 2017.
[3] The difference between revenues and expenses will go toward next fiscal year 2018–19.


Programs: upcoming year's annual planEdit

This section is about your organization's programs. A program is a defined set of activities that share the same objectives and a similar theory of change. Please share the general goal of each program, and then list the specific objectives that the program will meet. Please do not include information about your organization's operating activities in this section. You may provide information about activities like administration, staff and board training, fundraising, governance, and internal IT in another section or in a supplementary document, but please do not include these activities here as programs.

1. For each program, and overall
  • Include targets for each of the three shared measures for each program, and overall. If one or more of these required metrics are not relevant to any of your programs, please consult your program officer.
  • Also choose at least two grantee-defined metrics to highlight in this section, and include targets for each of these grantee-defined metrics for each program, and overall. (Other program-specific metrics may be included in your program objectives, in the detailed program sections below.)


Table 3

Shared metrics

  1. Participants: The number of people who attend your events, programs or activities, either in person or virtually. This definition does not include people organizing activities, social media followers, donors, or others not participating directly.
  2. Newly registered: The number of participants that create new accounts on a Wikimedia project. These include users who register up to two weeks before the start of the event.
  3. Content pages: A content page is an article on Wikipedia, an item on Wikidata, a content page on Wikisource, an entry on Wiktionary, and a media file on Commons, etc. This metric captures the total number of content pages created or improved across all Wikimedia projects.

Grantee-defined metrics

  1. Quantity: Number of words added to the article namespace.
  2. Quality articles: Number of articles that have at least a 10-point improvement in an ORES-based quality prediction score, which indicates significant improvement of the "structural completeness" of an article. See the "Defining quality content" sections in each program section for a description of how this applies to that program.
Program Participants Newly registered Content pages Quantity Quality articles
Classroom Program 16,000 15,500 19,200 11.2 million 5,000
Visiting Scholars Program 15 0 680 810,000 435
Wikipedia Fellows Pilot 20 20 80 24,000 40
TOTAL FOR ALL PROGRAMS 16,035 15,520 19,960 12 million 5,475


Table 3 notes:

  • See additional targets for each program as connected to the logic model at the end of each program's section.


2. Please list your goals and objectives for each program. Please be sure your objectives meet all three criteria for each program
  • The objectives listed are each SMART: specific, measurable, attainable and relevant, and include time-bound targets.
  • Include both qualitative targets and quantitative targets, and remember to highlight your baseline metrics.
  • Provide any additional information that is important to our understanding of this program. For example, you may include needs assessments, logic models, timelines, tables, or charts. Share how this program will contribute more broadly to movement learning, or explain how your program aligns with important Wikimedia priorities such as increasing participation and improving content on the Wikimedia projects.


Classroom Program

Connection to the draft Wikimedia Movement Strategic Direction

The Classroom Program fits into both the knowledge as service and knowledge equity directions in the current draft. Creating the infrastructure to bring external partners into our movement has been a key role the Wiki Education Foundation has been playing for several years; our Dashboard software is a key example of this. It provides the structure to enable the management and tracking of programs to bring newcomers to Wikipedia.

Those newcomers represent the knowledge equity element. Our Classroom Program brings in a diverse group of students from community colleges to Ivy League schools. Our work to target content improvement on topics related to gender, race, and sexuality has meant that more people globally have access to high quality free knowledge on these topics. In addition to providing all of our students with the resources they need to successfully contribute to Wikipedia, regardless of their backgrounds and academic experience, we provide additional support through our discipline-specific handouts. These handouts ensure that students editing in certain fields, such as gender studies, can contribute more equitably to Wikipedia.

In our Classroom Program, student editors at universities in the United States and Canada edit Wikipedia articles as a class assignment. Our theory of change is that by providing support for instructors and student editors, we will improve the quality and quantity of Wikipedia content in underdeveloped subject areas.

Background of program

When we started the pilot of the program (the Public Policy Initiative) in 2010, we had seen university instructors around the world assign their students to edit Wikipedia as a class assignment — with good results when the instructor was a Wikipedian themselves, and with rather mixed results when the instructor didn’t know much about Wikipedia. We wanted to create a sustainable, volunteer-driven program in which university instructors who didn’t know much about Wikipedia could get the support from our program to successfully teach with Wikipedia.

We started the program at the Wikimedia Foundation; a few years in, we knew what we had was a good program but it wasn’t the focus of the Wikimedia Foundation, and it needed to have more attention to realize its potential. A group of volunteer Wikipedians and professors proposed a new organization, the Wiki Education Foundation, and in late 2013, the program officially moved from WMF to the Wiki Education Foundation.

 
Eight years of growth in number of classes participating in the Wiki Education Foundation's Classroom Program, 2010–17.

Nearly eight years in, we’ve learned a lot about this program in the United States and Canada. One of our big learnings is that there is a lot of enthusiasm for teaching with Wikipedia among university faculty here. Higher education trends point toward providing more project-based, active, or service learning projects; Wikipedia assignments meet all these criteria. Universities are desperately trying to teach students crucial information literacy skills. As a 2016 Stanford University study measuring information literacy skills of middle school, high school, and college students noted, “Overall, young people’s ability to reason about the information on the Internet can be summed up in one word: bleak.” Again, Wikipedia assignments build these skills. And finally, the academy has embraced Wikipedia as a starting place for research, and begun to recognize the importance of improving the information resource the world uses. As our slogan says, “Don’t cite it. Write it!” This enthusiasm for our program has enabled a sharp growth trend from 14 classes participating in our program in Fall 2010 to 358 classes participating in Spring 2017.

We’ve also learned that with the right support, Wikipedia class assignments can be successful even if the instructor has never edited an article. Assignment design is key; so are training modules, brochures, a searchable question-and-answer platform, and experienced Wikipedians on standby to provide a human backup when something goes wrong. The wide range of support offered by the Wiki Education Foundation has enabled more than 1,000 individual faculty members to incorporate Wikipedia into their classes since 2010; fewer than 5% of them had ever edited Wikipedia before our project.

Another learning is that the volunteer-driven element of our program simply didn’t scale effectively. We had initially created “Wikipedia Ambassadors” who would have a 1:1 relationship with students to provide support, with a “Regional Ambassador” volunteer who would coordinate the work in a particular region. This model worked well when we were supporting 20–30 classes, but started breaking down shortly after we started scaling. We weren’t able to find enough volunteers to meet the demand from instructors. There was a wide quality and quantity gap in the support individual volunteers provided to courses. Many well-meaning volunteer editors were also students themselves; they were eager to help during the beginning of the term when their own course work was lighter, but when students wanted help often naturally coincided with the biggest crunch time of their own academic schedule, when they had less time to give. And not everybody wanted to do all of the tasks for a class, making it extremely difficult to pair the right volunteer with the right class. We realized we were spending dramatically more staff time trying to coordinate volunteer Ambassadors, and with inconsistent support to classes, than we would be if we simply brought that role into our organization as staff; we did this in 2014. While we believe the volunteer model works well for small-scale education programs, we find it to be a better use of resources to have a staff-driven program at our scale.

A scalable education program

 
While the amount of staff time devoted to the Classroom Program in FTE equivalent terms has stayed relatively flat over the last four years (+/- 1.5 FTEs), the number of students has dramatically increased, showing that we are able to increase the impact of our program without adding staff (our biggest expenditure for the program).

The Wiki Education Foundation’s organizational goal continues to be building a scalable model: Rather than increasing our budget as we increase the number of students we add to the program, we want to instead keep our spending on the program stable while we increase our impact. To do so, this required a large investment (in our 2015–16 fiscal year) to build out our Dashboard course management software tool. We’ve tripled the number of classes we supported over a three year period (from 117 in spring 2015 to 358 in spring 2017) without increasing the staff dedicated to the Classroom Program, clearly demonstrating the success of this approach.

 
Process map describing the Wiki Education Foundation's Classroom Program as of 2014. (Click through for a larger version.)

In 2014, we approached how to scale the program by creating a process map of our program. In the process map, we walked through each stage of the Classroom Program experience, identified where there were current bottlenecks to our scaling, and then identified what potential technical solutions would be to each of those, when there was a necessity for human intervention, and identifying any dependencies that were our assumptions. Creating this process map was a key step for our organization, because it created a roadmap for what we needed to create in terms of our technology and our communications materials to support a growing number of students. Today, our Dashboard course management platform has resolved most of the bottlenecks identified in our original process map.

A note about the Program & Events Dashboard

Our Dashboard software is released under an open license, and in 2015, we installed a version of it on WMF Labs called the Program & Events Dashboard, or P&E Dashboard. Thanks to some work from WMF staff, some collaborations at past Wikimania Hackathons, some Google Summer of Code and Outreachy interns, and some development work from the Wiki Education Foundation, the P&E Dashboard has a pared-down feature suite, but works across all languages and Wikimedia projects. Program leaders can use it to see what participants in programs including education, edit-a-thons, writing contests, and photo contests have added to Wikimedia projects. Many of the program management features on our Dashboard (the Assignment Design Wizard, week-by-week timeline functionality with temporally assigned training modules, the alerts system, the survey system) are not available on the P&E Dashboard, and program leaders who’ve used the P&E Dashboard have requested a number of other features that are useful for programs like edit-a-thons that the Wiki Education Foundation doesn’t do. That means there is a separate technical roadmap for the P&E Dashboard.

We are not including any money for development for features on the P&E Dashboard's technical roadmap in our FDC proposal. Instead, we are only seeking funding for technical projects that will help the Wiki Education Foundation's version of the Dashboard. If we develop a feature for our Dashboard that is also relevant for the P&E Dashboard, we will deploy it to the P&E Dashboard. But specific technical features for the P&E Dashboard that aren't relevant to the Wiki Education Foundation's programs will be funded separately.

Here’s how our Dashboard software works:

  1. A new instructor creates a Wikipedia account, and is guided through an online orientation that introduces them to key Wikipedia policies and what teaching with Wikipedia looks like.
  2. After successfully completing the orientation, the instructor is offered a series of assignment types that they can create. Most instructors choose the “Expand or create a Wikipedia article” assignment, although some do other options like copyediting, adding images, or translation.
  3. The Dashboard’s Assignment Design Wizard walks the instructor through a series of questions about their course and how they would like the assignment to work (e.g., does the instructor want to create a list of articles for students to choose from, or will the students need to identify their own articles for improvement? Are students working individually or in groups? Are students possibly working on medical topics?).
  4. Based on what the instructor has inputted in the Assignment Design Wizard, the Dashboard creates a suggested timeline for the Wikipedia assignment. Relevant training modules are automatically added (so, for example, students who might be working on medical content will get a special module on MEDRS). The instructor can edit this timeline if they choose. Once they are happy with their assignment, they’ll click the submit button, which causes the course page to get copied to the English Wikipedia for transparency’s sake, and notifies our staff that a course page is available for review.
  5. Our Classroom Program Manager reviews the course page to ensure any edits the instructor made meet our best practices (so, for example, if the instructor has added that they will grade only on what sticks on Wikipedia, not on what students did, we won’t approve the course page until that language is removed). This is also the point where we will discourage classes that are not appropriate for full-length writing a Wikipedia article assignments (e.g., those with more than 100 students), pushing them to consider an alternative assignment.
  6. We assign each course to one of our two Wikipedia Content Experts, who are longtime Wikipedia editors responsible for overseeing student work and helping resolve problems.
  7. Once the course is approved, the instructor receives a link to distribute to their students. Students click on this link, create Wikipedia accounts, and enroll in the Dashboard. We automatically put links on their user page to their course page, and we create their sandbox, replacing the typical sandbox template with a template that directs them to our staff for questions.
  8. On a week by week basis, the Dashboard will assign brochures to read, training modules to complete, or other references for students as needed. They learn new things about Wikipedia the week they need to know them, so for example, students learn about notability and sourcing early in the term, but they’re not served the training module about how to move out of their sandboxes until several weeks in, when they’re supposed to move their article.
  9. The Wikipedia Content Expert assigned to the class sees which classes are active each week, giving them an opportunity to proactively look into what students are doing. There are also a series of automated alerts that the Dashboard detects and notifies the Wikipedia Content Expert about, including: if a student edits an article with a discretionary sanction tag, if an article a student edited is nominated for deletion, and if an article a student edited is nominated for the Did You Know section on Wikipedia's main page. These alerts enable us to head off any potential problems without burdening valuable volunteer time.
  10. When students have questions, they’re prompted to search our help forum (ask.wikiedu.org), which is a Q&A format that offers answers to the most frequently asked questions. While anyone can log into ask and answer questions with their Wikimedia accounts, the majority of program participants simply use the “Get Help” feature in our Dashboard to reach out to the Wikipedia Content Expert assigned to their course. At the end of each term, we add additional questions and answers to ask.wikiedu.org that we received throughout the term, so the resource continues to grow.
  11. At the end of the term, instructors use the Dashboard’s metrics, authorship highlighting (which shows which students added what to each article), diff viewer, and other tools to easily visualize and grade students’ work.
  12. When the course is finished, the Dashboard emails the instructor a link to our end-of-term survey, where we ask what went well, what didn’t, and what we could do better next term.
  13. At the beginning of the next term, the instructor can either go through the Assignment Design Wizard again for their next course, or clone their previous course page and simply adjust the dates, and the process starts again.

Through this process, many instructors and students go through our program without ever interacting with one of our staff. Those who do interact with us mostly do in their first term teaching; once they have a good feel for how the program works, our tools enable them to run successful Wikipedia assignments without taking our staff time. The Dashboard has enabled us to scale such that we are able to serve them the appropriate training modules and brochures so they can be self-sufficient, but with enough monitoring and alerts that we are able to jump in to help when students run into trouble.

Which classes we support — and which ones we don't

 
We've learned seminar-style classes where students have a close connection to the instructor work best in the program; our average class size is 21 students.

We've learned in doing this program for many years that some courses are a great fit for full Wikipedia writing assignments, and some — no matter how much the instructor or we might want it to work — just aren't. The Wiki Education Foundation has worked with faculty at more than 500 universities in the United States and Canada, supporting classes at institutions ranging from community colleges to small liberal arts colleges to Ivy League schools to large public research universities.

Our experience has taught us that the most important factor in whether a course adds good content to Wikipedia is how engaged the instructor is in providing a meaningful learning experience for their students, and exceptional teachers can be found across the academic spectrum, from adjuncts to tenured professors. We avoid classes where that engagement is difficult: the class has more than 100 students or the class is primarily taught by teaching assistants (TAs). We also avoid survey courses for non-majors, where students likely haven't learned enough depth about a topic to make a good contribution to Wikipedia. We also screen carefully to ensure instructors aren't expecting students to do things like original research, requiring that students submit to the Good Article or Did You Know process even if the article doesn't qualify, or grading on what "sticks" on Wikipedia. This course evaluation process honed over many years enables us to identify the courses with proper assignment design that are most likely to have a positive contribution to Wikipedia, and head off those that are likely to end poorly.

A powerful tool to target content gaps

 
More than 60% of the Wiki Education Foundation's students in our Classroom Program identify as women.

The beauty of our education program isn’t just that we are improving the quality and quantity of information available on Wikipedia: It’s that we can target the subject of that information to match the English Wikipedia’s existing content gaps. One of the most well-publicized gaps is the gender gap; not only does this manifest itself in the contributor base on Wikipedia, but it also manifests itself within the content: topics related to race, gender, and sexuality have significant gaps on the English Wikipedia. The Wiki Education Foundation’s Classroom Program has made a major impact in the gender gap on Wikipedia over the last several years. Around 60% of our student editors are women. And thanks to our partnership with the National Women’s Studies Association (NWSA), we’ve improved more than 3,500 articles related to race, gender, and sexuality on Wikipedia in the last two years.

These partnerships are a key strategy for several of the Wiki Education Foundation’s programs. For the Classroom Program, our partnerships with academic associations prove to be fertile recruiting ground for instructors. For example, with our NWSA partnership, they provide us a workshop slot and a free exhibit hall booth at their annual conference. Our staff travels to wherever the conference is being held, leads the workshop, and collects names and email addresses of instructors who are interested in teaching with Wikipedia. We follow up with these instructors after the conference to encourage them to start teaching with Wikipedia by logging into the Dashboard software as noted above. NWSA also sends out email blasts to their members, features us in their communications, and invites us to speak at department chair sessions. The endorsement of the association gives another reason for instructors to participate. The result of this partnership for the Classroom Program is that 159 courses in women’s studies have taught with Wikipedia since the start of our partnership.

Partnerships take time to flourish, but the success of the NWSA partnership demonstrates the power of professional associations in helping us close content gaps on Wikipedia.

Demonstrated impact to student learning

 
Having data on student learning outcomes is key to bringing more faculty into the program.

While most of our instructors are motivated to teach with Wikipedia because they want to see the encyclopedia’s coverage of their field of study get better, they also need to be able to justify the decision to deans, tenure committees, and other university administrators who are more concerned with student learning than with improving Wikipedia. That’s why it’s so important to have peer-reviewed literature that measures the student learning outcomes from Wikipedia assignments. The Wiki Education Foundation is uniquely positioned to lead this research for the Wikipedia Education Program globally simply because we have a large enough sample size of students to get meaningful results.

 
Student Learning Outcomes using Wikipedia-based Assignments Fall 2016 Research Report

In fall 2016, we conducted our first student learning outcomes research project, with the results published in a report on Commons (and submitted to a peer-reviewed journal). The study results indicated:

  • 96% of instructors thought the Wikipedia assignment was more or much more valuable for teaching students digital literacy than traditional assignments
  • 85% of instructors thought the Wikipedia assignment was more or much more valuable for teaching students the reliability of online sources
  • 79% of instructors thought the Wikipedia assignment was more or much more valuable for teaching students to write clearly for the general public
  • Wikipedia assignments shift students’ perceptions of Wikipedia’s reliability to show more trust in Wikipedia
  • Students are more motivated to complete Wikipedia assignments, particularly because they perceived work to be useful beyond the classroom
  • Students’ skill development from Wikipedia assignments maps well to the Association of College and Research Libraries’ Information Literacy Framework, particularly:
    • Authority is constructed and contextual
    • Information creation as a process
    • Information has value
    • Scholarship as conversation

The results of this study have given instructors the data they need to prove the value of teaching with Wikipedia from a student learning outcomes perspective to recalcitrant university administrators. We’ve heard from several Wikimedia program leaders that the research results are valuable for them to convince decision-makers in their countries of the educational value of teaching with Wikipedia.

Defining quality content

The three core FDC metrics all measure outputs: participants, new participants, and how many articles they edit. Our organization is focused on impact as a measure of success, which is why we chose quality and quantity as our two grantee-defined metrics. Quantity is quite easy to measure for the Classroom Program: Our Dashboard software automatically calculates the number of words students add to the article namespace. Quality is a lot harder to measure.

In 2010–11, the pilot year of our program, we conducted an extensive research study. We brought in subject matter experts and Wikipedians, and asked them to assess student articles on a 26-point scale, based on the English Wikipedia's 1.0 Assessment scale. The results showed that students make substantive improvements to articles through their work. We repeated this study again in spring 2012; again, the results showed significant quality improvement. But both of these studies took an immense amount of volunteer time to assess articles. It was not scalable to keep asking Wikipedians to assess before and after articles each term.

More recently, we've started using ORES as a way to measure "structural completeness" of articles. ORES generates a "wp10" score for each version of an article, based on how well an article matches the typical structural features of a mature Wikipedia article. The machine learning model calculates scores based on the amount of prose, the number of wikilinks to other articles, the numbers of references, images, headers and templates, and a few other basic features, on a scale from 1 to 100. While ORES does not take the quality of sources and writing into account, it does give the best machine estimate of the quality level of an article currently available in the Wikimedia movement to our knowledge. Getting the first 20 or so points in an article is pretty easy: add a couple section headers, put a few sentences and references, add an infobox. And then the real work begins. At the Wiki Education Foundation, we use the ORES scores as an imperfect metric for the quality of student work: How much did existing articles improve on the ORES scale through student work? We purposefully exclude new articles from this count because getting an article to stub or start class automatically gives it so many points that it distorts our overall numbers, and we want the most accurate reflection of the impact students have.

We define the baseline for student work as articles where student editors in the Classroom Program added at least 1,000 bytes. That's almost 200 words — enough to exclude edits where students are doing copy edits, adding wikilinks, or other minor edits. In spring 2017, our students added at least 1,000 bytes to 3,253 articles (they also added at least 1,000 bytes to 806 new articles, but we have excluded that from the chart since there's no "before"). The chart below shows the ORES score of articles before (teal) and after (lavender) the students worked on them:

 

As you can see from the chart, students take a lot of stub and start class articles, and make major enough improvements to move these articles into mid-quality range. On average, our students improved these articles by 13 points on the ORES scale.

But the impact our program has is even more dramatic when you focus solely on the articles where students added at least 6,000 bytes of content. That's about 1,150 words — a significant amount of content added to an article by a student or group of students working together. In spring 2017, our student editors added at least 6,000 bytes to 1,156 articles. The chart below again shows the ORES score of articles before (teal) and after (lavender) the students worked on them:

 

The impact of our program is even more stark here: Articles went up an average of 23 points on the ORES scale. Student editors in the Wiki Education Foundation's Classroom Program are making a major difference in the quality of content on Wikipedia, as measured by ORES.

Another indicator of quality is how many student edits were reverted or deleted; the English Wikipedia community is generally pretty good at catching and reverting or deleting poor quality content. In spring 2017, only 2.6% of our students' mainspace edits were reverted and only 1.9% were deleted. This percentage is quite small, especially considering the high quantity of content our student editors add compared to other first-time editors, and is another indication that the content our student editors add to Wikipedia is of high quality.

But what about retention?

Unlike other education programs, our focus is not on retaining student editors. Our system teaches them what they need to know about Wikipedia to contribute successfully to Wikipedia as a class assignment. On student surveys, we’ve asked if students plan to keep editing; the majority say they do, but very few actually stick around. Some student editors find a new home in the Wikipedia community, and they stick around, but they are the exception rather than the rule. Nevertheless, a recent study conducted by researchers at Carnegie Mellon University and the University of Pittsburgh has proven that students who joined Wikipedia through the Wiki Education Foundation’s programs edit more, improve the articles more, and stayed longer than did newcomers who joined Wikipedia individually.[1] While we are encouraged by this study’s results, we believe our time is more valuably spent retaining instructors.

When we retain an instructor in our program, they will continue to teach with Wikipedia year after year, bringing a new crop of students to improve content on Wikipedia. As an example, Timothy Heningsen, an English professor at the College of DuPage, taught with Wikipedia for the first time in spring 2015. He had such a great experience that he’s integrated it into most of his classes since; he’s taught 17 courses with us, brought 366 students to Wikipedia who have added 279,000 words to more than 400 articles. The impact of retaining instructors amplifies over time, so we focus on ensuring we are retaining a large percentage of instructors whose students added good content to Wikipedia.

 
Year over year retention of instructors in the Wiki Education Foundation's Classroom Program.

Measuring quality of a course is something we’ve focused a lot on recently. We have a "still-being-perfected" formula that calculates a course’s ranking at the end of the term, taking into account the amount of content added, the number of students, the staff time we spent on the course, and our Wikipedia Content Expert’s ranking of the quality of content students added to Wikipedia. Then we focus our retention efforts on the top classes; for low-ranked courses, we try to determine if there’s something we can fix to make the course run more smoothly in a future term, and if not, we discourage that instructor from teaching with Wikipedia again. Retention is further complicated by academia’s scheduling, where professors might not teach a Wikipedia-appropriate course every term, go on sabbatical, have a term off from teaching while they serve on committees, or leave academia. That means we will never achieve 100% retention, but our focus on retaining our best courses’ instructors helps us ensure our programs are adding quality content to Wikipedia year over year.

Last year, we dug into the numbers, looking at the individual 177 instructors (not courses) who taught with us during the 2014–15 school year. A total of 103 of those individual instructors taught with us at some point during the 2015–16 academic year, representing a raw 59% retention rate of instructors. For the other 74 instructors, we looked into email exchanges we had with them and dug into online course catalogs on university websites to determine if they were teaching the course they'd taught with us again. From that, we determined 18 (or about 10%) were either on sabbatical or had left academia, meaning there was no way we could have retained them. Another 23 (or 13%) did teach that academic year, but did not teach the same course from when they participated in our program; the course(s) they did teach may or may not have been appropriate for a Wikipedia assignment (large survey classes led by TAs, for example, are not appropriate for Wikipedia assignments). We couldn't determine the status of 25 (or 14%) of the instructors; they may have left academia, but we weren't able to find out, and they did not respond to our emails. Finally, we found that six (or 4%) of the instructors had taught the same course again, but this time without a Wikipedia assignment. That means we failed to retain only a small percentage that we know of. Overall, we believe the numbers indicate that we are doing well in our efforts to retain instructors in our program.

References

  1. Robert Kraut et al., Production or commitment? Dilemmas in bringing newcomers on-board (currently under review)

Supporting the global education community

 
The Wiki Education Foundation's Director of Programs LiAnna Davis presents to heads of representatives from the Research, Information, and Dissemination Centers that are part of the São Paulo Research Foundation (FAPESP), in an effort to encourage more science classes to join the Grupo de Usuários Wikimedia no Brasil's education program.

The Wiki Education Foundation takes its role as the largest education program in the Wikimedia movement seriously. Education is a big part of a lot of Wikimedia affiliates' activities, but since it is the entire focus of our organization, we've been able to scale immensely. We've learned a lot along the way, and we believe it is important to share our learnings regularly with other Wikimedia movement entities. We publish regular blog posts, extensive monthly reports documenting our activities and learnings, and full evaluation reports of our pilot programs. These evaluation reports are particularly useful for global community groups who are looking to repeat our work; for example, the Grupo de Usuários Wikimedia no Brasil is interested in doing a Year of Science initiative in Brazil, modeled after our initiative last year. We periodically contribute to the Education newsletter, share our learnings to other program leaders in the Education Collaborative (which started transitioning to a user group), and have facilitated workshops at the last two Learning Days during Wikimania 2017 in Montréal. We stay in contact with other education program leaders globally, offering mentorship when we can, and learning from what they do.

Another key area where we provide leadership to the global community is in development of resources. We've ported the content of our online training slides to the Program & Events Dashboard, and with support from some community members, are working to make them translatable. We've provided our source files for our printed brochures to groups looking to translate and localize our materials for their own educational contexts.

We strongly believe in documentation and not re-creating the wheel, and we spend a fair amount of staff time and energy ensuring we are transparently sharing our successes, failures, and learnings with others in the movement.

Logic model

 

Activities in 2018

 
A student uses the Wiki Education's "Editing Wikipedia" brochure as she completes her course assignment.

In 2018, we will continue to scale the Classroom Program, growing the number of classes participating in the program and thus increasing the quality and quantity of content added to the English Wikipedia in academic subject areas.

This year, we are kicking off an initiative called Future of Facts, in which we endeavor to help close content gaps on politically relevant topic areas. In the age of “fake news”, purposeful misinformation, and lack of information literacy skills, Wikipedia stands as a beacon of fact-based truth. We want to ensure that when citizens turn to Wikipedia to inform themselves about politically relevant topic areas, they will find neutral, well-cited facts. As part of the Future of Facts initiative, we will:

  • Create partnerships with academic associations in topic areas like public policy, political science, law, history, and environmental sciences
  • Recruit new instructors to our Classroom Program through existing partnerships with the Midwest Political Science Association, American Sociological Association, and National Women’s Studies Association
  • Staff booths at exhibit halls for academic conferences for politically relevant subject areas to recruit new faculty
  • Target outreach activities at historically black colleges and universities (HBCUs). By bringing students explicitly studying topics related to African-American culture and history, we can improve Wikipedia’s coverage of these topics. This ties in to our efforts to have citizens who understand American history, culture, and topics often overlooked in primary education
  • Support instructors and students who want to assign their students to edit Wikipedia in these subject areas
  • Create new discipline-specific handouts on how to edit Wikipedia in politically relevant topic areas
 
The Wiki Education Foundation's Jami Mathewson talks to a potential instructor a the 2017 American Sociological Association conference.

We will also continue to support classes in other subject areas; our growth rate for new classes continues to increase each term due to the visibility we’ve achieved, and we anticipate the program will continue to expand. We will:

  • Support faculty and student editors who are participating in the program, intervening if there are any problems
  • Develop additional software features for our Dashboard course management software that enables us to support more students with the same number of staff
  • Create new training modules and ask.wikiedu.org content to address common student problems and questions
  • Continue to measure the impact of our classes, and focus on how to retain the best instructors
  • Publish regular blog posts highlighting the impact of our work
The "Guided Editing" project

One of the biggest pain points about our student editors for existing Wikipedia editors is that sometimes they struggle to get the tone right for an encyclopedia article. Issues with plagiarism, too few citations, and failure to meet the manual of style on Wikipedia frustrate existing editors. We want to create a Guided Editing experience that uses machine learning to review students’ edits as they make them, making suggestions to avoid plagiarism, citation issues, tone problems, and manual of style errors, before the students’ edits are made on the live article namespace on Wikipedia. Creating this Guided Editing system will enable us to address many of the most common challenges student editors face.

In 2018, we will solicit external funders to seek a restricted grant to develop the Guided Editing software. If money comes through for this in 2018, we will begin the early stages of what we expect to be a major technical project that will span more than one year. But because the timing of this is dependent on when this grant funding comes through, we are not including it in our FDC application.

Finally, we started investigating student learning outcomes with Wikipedia-based assignments, as described above. This research acts as an example, model, and framework for future research, but it is far from complete and requires more time and effort to continue. In 2018, we will expand the study. This study will benefit researchers and students by narrowing the focus of previous research and increasing the amount of data on this underrepresented subject. More data on this subject will allow a better understanding of the students' contexts that affect attitudes, productivity, digital literacy, collaboration, and skills transfer. This information is beneficial to Wikipedia and Wikipedia researchers because it helps triangulate attitudes and contexts for understanding how students engage Wikipedia, and it helps create knowledge about increasing student learning while collaborating on and creating Wikipedia articles. We believe this research is critical for our program as well as for programs globally who need this data to convince ministers of education, deans, and other decision-makers of the validity of Wikipedia assignments producing student learning outcomes; endorsements from global program leaders on an earlier unsuccessful project grant speaks to this as well. In 2018, we will

  • Design survey for both repeated and term-specific research questions
  • Continue data collection and analysis
  • Publish data in open formats
  • Present data and findings at national and international conferences
  • Publish analysis and presentations in open venues

Quantitative targets, 2018

Measure of success Type Baseline (2016–17) Goal (2018)
New discipline-specific handouts developed Input 4 4
New academic association partnership agreements signed Input 1 2
Academic association conferences attended to recruit instructors Input 20 15[1]
Number of students Output (participants) 14,136 16,000
Percentage of students who identify as women Output (participants) 60% 60%
Number of articles edited Output (direct product) 15,156 19,200
Amount of content added Impact (short term) 10.6 million words 11.2 million words
Number of articles edited with at least 10 ORES point improvement Impact (short term) 4,240 5,000
Instructor satisfaction rate (percentage who indicate interest in

teaching with Wikipedia again)

Impact (short term) 98% 98%
New student learning outcome study completed Impact (long term) 1 1

[1] We have a better idea now of what conferences are most effective for us, so we believe we can achieve an increase in impact while attending fewer conferences.

Risks

  • As our program has grown, our individual connection to each faculty member participating in the program has decreased. The less personal connection faculty have to our staff, the higher the risk that they will not return to the program. We actively track instructor retention and spend time engaging on how to retain more faculty, so we do not believe this will be a problem, but we acknowledge it is a risk.
  • As we move toward higher quality contributions from our Classroom Program student editors, we may see a decrease in the overall number of students in the program. By targeting smaller classes that produce better content, we believe will have a deeper impact on Wikipedia, but this impact may come from fewer student editors. We believe the amount of content added and the quality improvement is more important than the number of students as a metric of our success.


Visiting Scholars Program

Connection to the draft Wikimedia Movement Strategic Direction

The Visiting Scholars program exemplifies the knowledge equity desire to break down barriers preventing people from accessing free knowledge. To build Wikipedia, editors need access to high-quality sources. There's more free content available than ever before, but a great deal of what editors need is locked behind paywalls, in expensive books, or only accessible through a particular institution. Through the Visiting Scholars program, we use our networks and connections to put these resources into the hands of people who cannot otherwise access them. Everything our Visiting Scholars add to Wikipedia is content that would not have been there without access to sources behind paywalls, which provide a literal barrier to free knowledge. By adding the content to Wikipedia, Visiting Scholars promote knowledge equity.

In our Visiting Scholars program, the Wiki Education Foundation facilitates relationships between existing Wikipedia editors and educational institutions to provide access to sources for the existing Wikipedian. Our theory of change is that by providing access to sources otherwise behind paywalls, existing editors will be able to edit more and with higher quality contributions because we provide them with the reliable sources they need.

Background of program

 
The Wikipedia Library's Jake Orlowitz and the Wiki Education Foundation's Jami Mathewson recruit Visiting Scholars host institutions at the American Library Association conference in 2015.

The Visiting Scholars program originated from the Wikipedia Library team at the Wikimedia Foundation. In the program, the Scholar (an experienced Wikipedian) is granted remote access to library holdings, given an official university email address, and connected with library staff/faculty/centers. In return, the Scholar writes or improves Wikipedia articles in a subject area that may be specified by the university (typically related to focus areas and special collections specific to the library). It is an unpaid, remote, officially credentialed position.

The Wikipedia Library team at WMF piloted the program in 2014-15 with logins from four universities: Montana State University, the University of California at Riverside, George Mason University, and Rutgers University. In 2015, Jake Orlowitz and Alex Stinson, the Wikipedia Library staff at WMF, recognized the potential of the program, but also recognized they did not have the time to devote to the project. The Wiki Education Foundation, with our connections to universities in the United States and Canada, was a natural home for the program; Jake and Alex asked us to take over the program, and the transition happened in the middle of 2015.

In the two years since we’ve been running the program, we’ve made a few minor changes, but the key idea remains the same. We’ve learned that this program is an impactful way to make a major difference to a small number of Wikipedia editors, but it’s a more boutique program focused on individual connections. We’ve learned that sponsor institutions often have challenges technically providing the login; dealing with the bureaucracy within the host organization can quickly derail an otherwise promising scholar relationship, sometimes indefinitely. As an example of this, it took the Smithsonian Libraries, where User:Czar is a Visiting Scholar, 14 months from the point where our contact there had selected Czar as the Visiting Scholar until the point where they were able to actually give him access.

Another challenge is on the Scholar side; sometimes the host institution’s desire for a Scholar simply doesn't match an existing editor who needs access to sources. As an example of this, we’ve had an open call for a Visiting Scholar position at the University of North Carolina that’s been unfilled since July 2016. In this instance, they’re seeking an experienced Wikipedian to improve the quality of articles about clinical psychological science, with an emphasis on evidence-based assessment articles. We’ve been unable to find a candidate who wants access to sources to improve Wikipedia articles in this topic area.

Finding sponsors, finding Scholars, and pairing the appropriate sponsor and Scholar remains a challenging process, but its impact when it works is incredible.

A bidirectional approach to education

While our Classroom Program brings Wikipedia to academia, the Visiting Scholars program brings academia to Wikipedia. We believe both of these are strategically important, and this bidirectional work is one of the key roles our organization plays in the Wikimedia movement. We bridge the worlds of Wikipedia and academia, and the Visiting Scholars program shows why the connections we facilitate are so important.

The one key change we’ve made to the program since taking it over from the Wikipedia Library is indicative of this strategy. Instead of solely creating positions from sponsors and seeking Wikipedia editors to fill them, we also accept open applications from experienced editors who are interested in gaining access to sources in a particular topic area. We then use our extensive academic networks to find a sponsor for that Scholar. An example of how this worked is User:Gen Quon, a longtime editor who wanted to improve English Wikipedia articles on classical studies topics. Wiki Education Foundation staff reached out to Classics departments and libraries across the country, eventually connecting with the University of Pennsylvania, who were able to provide the login for Gen Quon.

In addition to customizing topic areas for editors, we also customize what specifically they need the sources for. Different Wikipedians like to do different tasks on Wikipedia, and the program is flexible enough to support a variety of different content development approaches. This can be seen in our existing Visiting Scholars: User:Wehwalt focuses on bringing articles up to Featured or Good Article status; User:M2454 creates extensive timeline articles and adds images; User:Rosiestep focuses on establishing notability for new biography articles; several others use their access to sources to do a variety of general content development.

By making it a program centered on the individual editor’s needs, we support a much less scalable program. Nevertheless, we firmly believe supporting the existing editor community by providing access to sources through the Visiting Scholars program is critically important. Experienced editors can tackle a wide range of articles that would be too challenging for a new editor to approach, but that are very important articles that get a lot of page views. For example, User:Bfpage significantly expanded the English Wikipedia articles on rape (which receives 4,000 page views a day) and MRSA (which averages 2,000 page views a day) with high-quality medical sources she got access to through her Visiting Scholar position.

How it works

 
Visiting Scholar Jackie Koerner spoke about how access to sources through the program improved her editing capabilities during Wikimania 2017.

Each Visiting Scholar position begins in one of two ways: we identify a qualified sponsor or a qualified Scholar.

When the host institution initiates a position, the program works like this: Our Outreach Manager and Educational Partnerships Manager are constantly seeking new sponsors when they attend academic conferences; we will often meet librarians, who are natural hosts for Visiting Scholar positions. We also ask our Classroom Program faculty on the instructor survey at the end of the term if they are interested in hosting a Visiting Scholar in their department; we then follow up with those who indicate interest. Once we have qualified the interest and the belief that it’s technically possible to host a Visiting Scholar, the Community Engagement Manager works with the sponsor to craft an open call for Wikipedians to fill the spot. The Community Engagement Manager spreads the word through relevant WikiProjects, discussion spaces, and occasionally watchlist notices; once we have candidate(s) for the position, we evaluate their on-wiki work. We want to ensure that the Scholar has a history of positive contributions to Wikipedia. We recommend qualified candidate(s) to sponsor; the sponsor makes the final selection.

When a prospective Scholar initiates a position, it works like this: An experienced Wikipedian submits an application through our open call, indicating his or her interest in editing a particular type and topic of articles. Our Community Engagement Manager runs the same check to ensure the editor has a history of productive contributions to Wikipedia. If the Wikipedian passes this qualification check, our Outreach Manager and Educational Partnerships Manager seek someone in our network who could potentially provide the Scholar the sources he or she needs. If we can’t find someone in our existing network, we do cold outreach to institutions until we find someone who can help the Scholar. Once we locate a sponsor, we recommend the Scholar to the sponsor.

This work we do to facilitate a Visiting Scholars position is the majority of the Wiki Education Foundation’s role in the program. Once a Scholar is placed, we step back and let them improve Wikipedia thanks to the access to their sources. We ask that they help contribute to our blog with a post showcasing how the access to the sources provided by the sponsor have helped them improve their own editing, which is our goal for the program. At the official end of a scholarship, we reach out to both the sponsor and Scholar to see if they’re both interested in renewing, and help facilitate that if possible.

Defining quality content

Our Classroom Program's section described the challenges in automated quality metrics; in light of those, we also use the ORES structural completeness metric to measure quality for the Visiting Scholars program. In the case of this program, we have a high confidence level that the sources these editors are using are high quality, since providing those sources is the explicit point of the program.

In the 2016-17 fiscal year, Visiting Scholars improved 569 articles as part of their position at the institution. For this program, we only measure the articles that Wikipedian edits using the sources; normal edits they make in the course of their volunteer work on Wikipedia are not counted. At the 1,000-byte threshold, the improvement isn't that impressive in graph form:

 

The graph above shows some of the challenges in ORES: Existing editors are more likely to grapple with a poor quality section that's cited to not particularly reliable sources in an existing article, rewriting it to add citations to reliable sources. While a human can quickly identify that the article has improved immensely, ORES struggles to see it as a major improvement.

When you look at the 6,000 byte threshold, however, you really can see the impact of the Visiting Scholars program. They added more than 6,000 bytes to a total of 55 existing articles, and the article improvement is dramatic:

 

The average article started as rated at 34 before the Visiting Scholars started editing and 62 after they finished editing. They average an improvement of 28 points to articles where they added more than 6,000 bytes.

Logic model

 

Activities in 2018

In 2018, we hope to extend our existing Visiting Scholar positions for another year, as well as adding more positions into the mix. We will continue:

  • Actively recruiting new host institutions (sponsors)
  • Actively recruiting existing Wikipedians who need access to sources (Scholars)
  • Facilitating the continuation of existing scholar–Sponsor relationships when both parties desire it
  • Blogging about the impact of the program

We would also like to start some new activities as part of the program:

  • Survey Scholars in the beginning and end of the term asking questions about access to sources, how access to sources has affected their editing, the quality of sources they use, their enthusiasm for the subject, and other editing habits; compare the beginning/end in order to demonstrate the varied impact the program has on its participants.
  • Create Visiting Scholars profiles, regularly updating their contributions (e.g. how they’re contributing to the Future of Facts, impact on Wikipedia), adding praise to the Scholars’ work, and attracting more attention to the project and host alike.
  • Pilot the use of small honoraria to create additional motivation for Scholars, testing whether offering a small financial award at the completion of a Scholarship will help fill longstanding openings.

Quantitative targets in 2018

Measure of success What it measures Baseline (2016–17) Goal (2018)
Number of Scholars Output (participants) 11 15
Number of blog posts about Scholar work Output (activity) 1 15
Find sponsors for qualified Scholar applicants within 6 months Output (activity) 50% 60%
Number of articles edited Output (direct product) 569 680
Amount of content added Impact (short term) 676,000 words 810,000 words
Number of articles edited with at least 10 ORES point improvement Impact (short term) 364 435

Risks

  • Because the Visiting Scholars Program is focused on a small number of editors who edit in different ways, it doesn't scale directly with the addition of more editors. Some editors edit more than others, some do extensive work on a small number of articles, while others do small work on a large number of articles. Half of our current Visiting Scholars account for 90% of the content added by the program; were we to lose one of our Visiting Scholars who adds a lot of content, our overall numbers would likely drop.
  • Another challenge in the program is mismatching between Scholars and sponsors. If we're unable to find relevant sponsors for qualified Scholar candidates, and unable to find qualified Scholars for interested sponsors, then we won't be able to expand the program next year.



Wikipedia Fellows Pilot

Connection to the draft Wikimedia Movement Strategic Direction

The Wikipedia Fellows Pilot will also address knowledge equity: By empowering subject matter experts to contribute to Wikipedia, we will expand the high quality information available in the open knowledge format. Our movement has struggled to recruit subject matter experts in the past, but we believe that our support infrastructure is well-positioned for a new program to serve this group as they add free knowledge to Wikipedia.

In 2018, we are undertaking a new pilot program: Wikipedia Fellows. In the Wikipedia Fellows Pilot, we work with academic association partners to have them select 2–3 of their members, who are professors or other subject matter experts in their field, to be a “Wikipedia Fellow” for six months. The Wiki Education Foundation will provide training on how to edit Wikipedia, and the Fellow will contribute content to Wikipedia in their area of subject matter expertise. Our theory of change is that by providing subject matter experts a formal role from their academic association to edit Wikipedia in a structured term, we will improve the quality and quantity of information on Wikipedia. Very early indicators suggest that new editors recruited through programs like this may have higher retention rates than other new editor programs. In this pilot, we will test whether our theory of change is accurate.

Background of program

In Fall 2016, the Global Ageing group at Cochrane kicked off a pilot program modeled off the Visiting Scholars program that focused on recruiting subject matter experts to improve articles related to ageing on Wikipedia. Cochrane put out a call to their network that resulted in six subject matter experts volunteering for the program. User:Bluerasberry offered a four-week class to these six editors on how to contribute content to Wikipedia. The results were impressive: more than 100 articles edited, with more than 16,000 words added. And the content was important to readers, too: the articles were viewed more than 9.35 million times by the completion of the project in January 2017.

Even more impressive is the ongoing engagement in Wikipedia from the contributors brought in through this project. One of the editors, User:JenOttawa, took to Wikipedia editing and has continued to edit, including coming to Wikimania 2017 to talk about her experiences. Two other editors have continued to edit sporadically since the conclusion of the formal project, meaning that half of the editors who came to Wikipedia through the project have continued to edit. This sample size is to small to draw to draw firm conclusions, but it is intriguing enough that the Wiki Education Foundation wants to see if we can replicate it.

Wiki Education Foundation staff had conversations about the program with User:Bluerasberry throughout the Cochrane pilot, and we’re eager to take his learnings and put them into a pilot for our own program. We’ve called this pilot “Wikipedia Fellows”.

Education outside the university

Unlike our other two programs, the Wikipedia Fellows project isn’t centered around a university. Rather than following the formal structures of a classroom setting or jumping through the hoops of acquiring a university login, the program is open to subject matter experts in specific academic subject areas. While we expect most of these people will be from academia, they may also be those using their academic expertise in other professional work. The benefit of this program is that we’re reaching out through disciplinary associations rather than through the structure of the higher education system, thereby bringing in expertise in particular academic subject areas. It has the potential to open the program to those who wouldn’t otherwise be a natural fit for one of our other programs.

How it works

 
The Wiki Education Foundation's Executive Director Frank Schulenburg and former American Sociological Association Executive Director Sally Hillsman sign the partnership memorandum of understanding.

A key strategic priority for all our programs is our work to recruit organizational partners. Our Educational Partnership Manager’s job is to connect with academic associations, signing formal memoranda of understanding to have them support our programs. In the pilot phase, we already have verbal commitments from three association partners to participate: the American Sociological Association, the Midwest Political Science Association, and the National Women’s Studies Association. Each of these disciplines are key to our Future of Facts initiative.

Each of these associations will put out a call to their members this fall, seeking to have 2–3 positions at each association. The Wiki Education Foundation’s Community Engagement Manager and association staff will jointly screen and select the candidates to be Wikipedia Fellows.

Wikipedia Fellows are asked to commit three hours per week to the program between January and June 2018. At first, most of the time will be spent on learning how to contribute, including scheduled small group meetings. Structured training sessions will become increasingly infrequent over the course of the pilot as more time is dedicated to making contributions to Wikipedia. Fellows will be asked to sign up for Slack, a cloud-based chat/collaboration service which will be used to ask questions, share experiences, share resources, and collaborate on articles. By the end of 6 months, all participants should have made a significant improvement to at least two articles. Our Wikipedia Content Experts will be available to provide support for Fellows. Toward the end of the pilot, participants will be asked to write a short reflective blog post about their experience and respond to a short survey.

Defining quality content

Since Wikipedia Fellows is a pilot program, we don't have a baseline to measure it against. We define this program as a success if each Fellow improves at least two articles by 10 ORES points, since that indicates a significant improvement of an article. We will re-evaluate this at the stage-gate.

Logic model

 

Activities in 2018

The first half of 2018 will be the pilot phase for Wikipedia Fellows. Wiki Education Foundation staff will:

  • Hold videoconference training sessions for the Wikipedia Fellows
  • Monitor contributions from Wikipedia Fellows
  • Answer questions about editing Wikipedia as needed.

In the second half of 2018, we will evaluate the impact of the program at our stage-gate. We will determine if the theory of change that we tested is valid or not. If we think the program has potential, we will determine what changes, if any, need to be made to our program design, then enact a second round of Wikipedia Fellows (post-stage-gate in the chart below), bringing on additional association partners. If we determine the pilot program did not fulfill its promise to add high quality content to Wikipedia, we will discontinue the program.

Quantitative targets in 2018

Measure of success Type Pilot goal

(January–June)

Post-stage-gate goal

(July–December)

Total Goal (2018)
Number of academic associations signed on to participate Output (participants) 3 Retain initial 3, add 2 new 5
Number of Fellows Output (participants) 6 14 20
Number of articles edited Output (direct product) 24 56 80
Amount of content added Impact (short term) 7,200 words 16,800 words 24,000 words
Number of articles edited with at least 10 ORES point improvement Impact (short term) 12 28 40

Risks

  • It's possible we won't attract the interest of enough people in the associations; we think it's unlikely based on past conversations, but we acknowledge it is a risk.
  • Fellows have busy academic schedules, making it challenging to schedule synchronous learning experiences, which is important for the success of the program. It also may mean that other commitments overtake the Fellows commitment, and Fellows leave the program mid-term or otherwise become inactive.
  • Because this program targets subject matter experts, people who are professionals but not academics in the field may lack access to research resources. We hope to work with the association partners to resolve these issues if they arise.
  • If any of these risks come to pass, we will stop the pilot at the stage-gate and not continue it.



Staff and contractors: upcoming year's annual planEdit

1. Please describe your organization's staffing plan or strategy here, and provide a link to your organization's staffing plan or organogram if you have one.

The Wiki Education Foundation has been operating with a small number of staff since 2014. By outsourcing some of our needs in the areas of finance and HR, we've been able to save costs (see notes below). Our general approach is to maximize our impact, while keeping our staffing at the same level: with somewhere between 10 and 12 employees, we've more than tripled our impact between 2014 and today. We made this happen by continuously tracking our progress, increasing our effectiveness, developing tools that allow us to scale our programmatic outcomes, and by focusing heavily on learning and innovation. At this point, we are committed to keeping our FTE numbers at the same level for the foreseeable future. We will only add any additional resources on a temporary project-by-project basis, if we're also able to raise funds for these new projects.

2. List of staff by department or function.

Table 4

FTEs
Department or function Current End of funding period Explanation of changes
General Operations & Fundraising [1] [2] 2.5 2.5
Core Programs 5.0 5.0
Programs Support 3.0 3.0
Digital Infrastructure 1.0 1.0
Total 11.5 11.5

Table 4 notes or explanation of significant changes:
[1] Since 2014, we've been working with TriNet, a professional employer organization (PEO), which provides payroll and other services (like health benefits, 401(k), basic HR) to us. Our contract with TriNet allows us to be more cost efficient in the area of General Operations.
[2] In September 2017, we started working with an outside service that provides us with support in the area of finance (bookkeeping, accounting, etc.) on an hourly basis. This change allowed us to further reduce our costs (until then, we had a dedicated position for all finance-related tasks on staff).

3. How much does your organization plan to spend on staff by the end of the current funding period, in currency requested and US dollars?

Our total spendings on staff for calendar year 2017 are projected to be $1,226,017 (note: our Simple APG does not cover staff costs).

4. How much does your organization plan to spend on staff by the end of the upcoming funding period, in currency requested and US dollars?

Our total spendings on staff for calendar year 2018 are projected to be $1,413,476.


Financials: upcoming yearEdit

Detailed budget: upcoming yearEdit

Please link to your organization's detailed budget showing planned revenues and expenses for the upcoming funding period (e.g. 1 January 2017 to 31 December 2017). This may be a document included on this Wiki (Meta) or a publicly available spreadsheet.

Google Spreadsheet: Wiki Education Foundation Projected Budget 2018

Revenues: upcoming yearEdit

Please use this table to list your organization's anticipated revenues (income, or the amount your organization is bringing in) by revenue source (where the revenue is coming from) in the upcoming funding period (e.g. 1 January 2017 to 31 December 2017).

  • Use the status column to show if this funding is already guaranteed, if you are in the process of requesting funding, or if you are planning to request funding at a later time.
  • Please include in-kind donations and resources in this table, as applicable, and use the status column to show that they are in-kind resources.
  • Do not include money you plan to draw from your reserves during the upcoming funding period.

Table 5

Anticipated revenues for the upcoming funding period
Revenue source Currency requested US dollars Status (e.g. guaranteed, application)


Individuals $30,000 $30,000 Board giving is pledged; the rest will be requested at a later time.
Foundations: FDC Annual Plan Grant $750,000 $750,000 Application in progress.
Foundations: Stanton Foundation $400,000 $400,000 Guaranteed match of up to $400,000 of total received through the FDC process in 2018.
Foundations: Other $1,945,000 $1,945,000 Some in LOI state; the rest to be requested at a later time.


Total revenues (should equal the sum of the rows): $3,125,000 $3,125,000 -


Table 5 notes: If your organization has significant funding other than FDC funds, please note how those funds will be used. As can be seen on the detailed budget, our FDC funds will cover 40% of our Classroom Program spending, 74% of our Visiting Scholars Program spending, and 56% of the Wikipedia Fellows Pilot. The individual gifts, Stanton Foundation match, and other foundation money will go toward the remaining programmatic and operational expenses of our organization.


Operating reserves: current and upcoming yearsEdit

Please note that there is a policy that places restrictions on how much FDC funding your organization can use to build its operating reserves. If you would like to use FDC funding to for your organization's reserves, you must note that here. You will not be able to decide to allocate FDC funding from this grant to your reserves at a later date.

1. What is your plan for maintaining, building, or spending your reserves in the current year and the upcoming funding period? Please use the table below to show the amounts in your reserves at the beginning, year-to-date, and end of your current year, and the amount you plan to have in your reserves by the end of the upcoming funding period.

We currently don't have an operating reserve.

Table 6

Year Year start Year start (US) Year-to-date Year-to-date (US) Year end Year end (US)
Current year (e.g. 2017) $0 $0 $0 $0 $0 $0
Upcoming year (e.g. 2018) - - - - $0 $0
2. How much FDC funding is your organization requesting to add to your reserves in the upcoming funding period? If you are not requesting any FDC funding to add to your reserves in the upcoming funding period, you can write zero.

None of the FDC funding will go toward a reserve.


Expenses: upcoming year's annual planEdit

1. Expenses by program (excludes staff and operations).
Program expenses are the costs associated specifically with your organization's programs, and do not include operating expenses or staff salaries, which will be described in separate tables. Program expenses may be the costs of an event, the costs of outreach materials specific to a program, budgets for microgrants and reimbursements, or technical costs associated with specific programs, for example. The programs listed in this table should correspond to the programs you have listed in the programs section of this proposal form.


Table 7

Program Currency requested US dollars
Classroom Program $226,910 $226,910
Visiting Scholars Program $40,095 $40,095
Wikipedia Fellows Pilot $29,895 $29,895
Guided Editing $583,050 $583,050
Total program expenses (should equal the sum of the rows) $879,950 $879,950

Table 7 notes: If your organization has significant funding designated for specific programs (e.g. a restricted grant), please make a note of that here. The $583,050 in expected spending for the Guided Editing program will be funded through a separate grant designated for it, and thus is not included in our FDC application.


2. Total expenses. Please use this table to summarize your organization's total expenses overall.
These are divided into three categories: (1) staff expenses from Table 4 (including expenses for staff working on both programs and operations), (2) expenses for programs from Table 7 (does not include staff expenses or operations expenses), and (3) expenses for operations (does not include staff expenses or program expenses). Be sure to check the totals in this table to make sure they are consistent with the totals in the other tables you have submitted with this form. For example, your total program expenses excluding staff will be equal to the total in Table 7, while your total staff expenses will be equal to the total in Table 4 and your total expenses will be equal to the total in Table 1.


Table 8

Expense type Currency requested US dollars
Program expenses (total from Table 7, excludes staff) $879,950 $879,950
Operations (excludes staff and programs) $473,581 $473,581
Upcoming staff total expenses (from Table 4) $1,413,476 $1,413,476
Total expenses (should equal the sum of the rows) $2,767,007 $2,767,007



Verification and signatureEdit

Please enter "yes" or "no" for the verification below.

The term “political or legislative activities” includes any activities relating to political campaigns or candidates (including the contribution of funds and the publication of position statements relating to political campaigns or candidates); voter registration activities; meetings with or submissions and petitions to government executives, ministers, officers or agencies on political or policy issues; and any other activities seeking government intervention or policy implementation (like “lobbying”), whether directed toward the government or the community or public at large. General operating support through the FDC may not be used to cover political and legislative activities, although you may make a separate grant agreement with the WMF for these purposes.
I verify that no funds from the Wikimedia Foundation will be used
for political or legislative activities except as permitted by a grant agreement
Yes


Please sign below to complete this proposal form.

IMPORTANT. Please do not make any changes to this proposal form after the proposal submission deadline for this round. If a change that is essential to an understanding of your organization's proposal is needed, please request the change on the discussion page of this form so it may be reviewed by FDC staff. Once submitted, complete and valid proposal forms submitted on time by eligible organizations will be considered unless an organization withdraws its application in writing or fails to remain eligible for the duration of the FDC process.
Please sign here once this proposal form is complete, using four tildes. --LiAnna (Wiki Ed) (talk) 01:58, 30 September 2017 (UTC)