Learning and Evaluation/Grants Impact/Instructions

A Grants Impact report summarizes the collective outcomes of grants awarded through the Wikimedia Foundation (WMF) grants programs within a specific fiscal year, or awarded as part of an Inspire campaign.

Before you start

  • Look through a few grants from each grant program, to orient yourself on how the proposal and report forms differ between the grant programs.
  • Look through previous impact reports to understand their purpose and structure.
  • Familiarize yourself with the basic terminology and formatting (below)
  • Read through the instructions for collecting qualitative and quantitative data (below), familiarizing yourself with the relevant spreadsheets / documents.

Terminology

A larger glossary of terms is available, but here are a few for those unfamiliar with the terminology used with the various grants programs.

  • Goals
  • Objectives or Measures of Success. Note: PEG refers to "Measures of Success," while IEG refers to "Objectives," and APG refers to targets.
  • Outcomes

Formatting

  • If a cell is the sum of multiple numbers, perform the calculation in the cell. Every number in the calculation should have a source.
    • Leave the source in a note on the cell (e.g. WLM - 10K images; WLE - 5K images)
  • If a number seems funky (e.g. 400K articles?!), color the cell yellow and leave a google comment and tag Jaime or Sati.
  • If a number seems to have been under-reported, color the cell light red.
    • "Under-reported" means that an event should have reported this number, but didn't or did so vaguely (e.g. "We had so many partners for this event!")

Collecting QUANTITATIVE Data

Global Metrics

  • For all grant programs, Global Metrics should be entered into this google spreadsheet, under the relevant grant program tab. Metrics that are Global metrics should only be pulled from the Global Metrics tables.
  • For APG grantees: For each organization, enter metrics by "Program," where these programs correspond to those identified in their report (typically there will be about 2-6, but WMRS can have up to 10-15). Each row should be a program.
    • Example: In Amical's 2014-15 progress report, you will see 4 programs: Community care, GLAM, Education, and Community innovation/partnerships. Each of these programs should be a separate row in the database.


Other quantitative data

  • For non-APG grantees: In the same google spreadsheet that Global Metrics is collected, in the PEG or IEG tabs, scroll right to see the additional columns under "Additional participation", "Additional text content", "Additional bytes", and "Media". This is the full list of additional metrics to be entered when they appear.
  • For APG grantees: Each organization should have a file within this Organizational profiles folder; you'll need to create one if they don't have one yet.
    • Enter data into the blue "Data" tabs: Participation, Text Content, Bytes Added, Media.


Watchouts

  • The numerical value of a single metric may not always match between those reported in the text vs. those reported in the Global Metrics tables. Use the #s reported in the Global Metrics table as the "most-correct"; leave a note in the google spreadsheet when there might be a different number reported.
  • We have separated the Global Metrics "#4 - # of New images/media added to Wikimedia articles/pages" into the following two metrics. Be sure to read the text to verify which is actually being reported. The easiest way to spot a mis-reported number is the by magnitude of the data - "images uploaded" (4a) will typically be in the 1000 to 10,000 range, while "images in use" (4b) will typically be in the 100s range.
    • 4a - # Unique images/media added to Wikimedia articles/pages
    • 4b - # Images/media uploaded to Commons

Collecting QUALITATIVE Data

"Qualitative data" will be quotes / excerpts from the grant report about any outcomes the grantee reports either in the report or talk page. For each grant, in a text document (MS Word, Google doc, etc.) copy-paste any outcomes reported. These text documents should be kept in this shared google folder.


Tips

  • Err on the side of capturing more information than less.
  • Outcomes can be anything, but it's helpful to think about outcomes as:
  • Results in terms of Learning, e.g.
Event organizers learned that providing a casual social atmosphere makes it easier for new contributors to learn and makes it more likely they will return.
  • Results in terms of Actions, e.g.
Advocating within the government lead to the officials promising to release educational content under a Creative Common license.
  • Results in term of changes to Conditions, e.g.
GLAM Institutions who were previous Wikimedia partners are now advocating (independently) for other institutions to release their content as well.
  • Some statements will be a mix of qualitative and quantitative outcomes (e.g. "We had 400 participants attend our contest, and 4 of them were inactive editors who came back for the event."). Be sure to capture the quantitative data in the correct spreadsheet, as well as the whole quote.


Examples of outcomes
Project grants typically report outcomes in the following areas:

Conference and Travel support grants report a very broad range of outcomes, but you can find examples of outcomes from the Wikimania consultation and the 2014 Wikimania scholarships evaluation.

Additional notes and resources

General tips
For PEG reports:

  • All grant reports can be found by writing "/Report" as the end of the grant proposal URL
  • The "Goals" and "Project Plan" and "Measures of Success" sections will help quickly orient you to the specifics of the grant, and the expected outcomes to be included in the report.

For APG reports:

  • Dates can become very confusing. Use this calendar and its archive to keep everything straight. As an example, the "2014-15 Round 1 organizations" indicate that:
    • They were funded in Fall 2014 (thus in the 2014-15 funding cycle)
    • They are in the process of doing activities in 2015
    • They were previously funded in Fall 2013, to do activities in 2014, and have submitted an end of year "Impact report" for those 2014 activities.
    • Lastly, all these years may be different from an organization's fiscal year.


Additional resources

  • (need to complete)