Learning and Evaluation/Evaluation reports/2015/Limitations

Wikimedia Programs Evaluation 2015

Summary

Important definitions

Evaluation initiative

Goals

Limitations

Data access

Reading guide

Programs

How deep do the data go?
This report systematically measures a specific set of inputs, outputs, and outcomes across Wikimedia programs in order to learn about evaluation and reporting capacity as well as programs impact. Importantly, it is not a comprehensive review of all program activity or all potential impact measures, but of those program events for which data was accessible through voluntary and grants reporting.
Read this page to understand what the data tells you about the program and what it does not.

Data and analysis	We had many questions to ask program leaders! We have a lot of questions the data reported has helped us answer and others that need further investigation What do these programs costs in terms of dollars and hours invested and what potential costs are managed through donated resources? What is the reach of these programs in terms of accessing new and existing users? How much content do programs produce in terms of bytes, pages, or media added? What are the costs in terms of dollars and hours input per unit of content (text or media pages added) or per participant (for workshops which produce no content)? To what extent can we determine program outputs increase the quality of Wikimedia projects? To what extent does program participation produces new active editors/contributors, or retain active editors, at 3- and 6- months retention points? To what extent does the program have examples for easy sharing and replication?
Priority goal setting	Since 2013 the Program Evaluation and Design team has worked with the community to discover the most commonly seen goals for programs. Since the June pilot workshop activities online and at conference workshops - such as the Metrics Dialogue - have served as a way for our team to identify common goals across programs, discovered through conversations, break out sessions, and logic modeling. Most often these general goals have been to: Build and Engage Community Increase Reader Satisfaction Increase Access to Wikimedia Increase Accuracy and/or Detail of information Increase Awareness of Wikimedia Projects Increase Buy-in to the Open Knowledge / Free Content Movement Increase Contributions Increase Diversity of Information Coverage Increase Diversity of Participants Increase Participation of Content Experts Increase Perceptions of Credibility and/or Quality Increase Skills for Editing/Contributing Increase Use and Access Increase Usefulness, Usability, and Use of Contributions Increase Volunteer Motivation and Commitment Make Contributing Easier Make Contributing Fun Recruit and/or convert New Users Retain and Activate Existing Editors Notes