Wikimedia Conference 2014/Documentation/19

19 Program evaluation conference session

Speaker: Jaime Anstee (WMF)

tl;dr: Wikimetrics can be helpful.

Notes

Pilot Beta Reports

Initial metrics that are important for many of the programs (do not apply to all programs)
still developing metrics for other programs
Mapping out the Logic of the Program to determine whether the metrics are appropriate for a specific program.
Process and Tracking Tools > Tracking and Reporting Tool Kit > Forms and Templates, includes initial reporting template

Output

participation and level/amount of content added during the event/program
Total # of participants. number of new user accounts created. This could change what your impact looks like
Want to be able to find your users, or might have a file on your computer (identifying where you stored usernames)
Wikimedia Projects: entering the names of the Wikimedia projects your cohort contributed to as part of this program
Categories: Please enter any category names that were created and used for content generated as part of the program implementation. Need to know what categories to search. important to be correct.
Wikipedia Education Programs: # of classes and # of instruction included in writing.

Content Added

How many Bytes were added to the Wikimedia project pages by your user cohort as a part of this progarm implementation? Use wikimetrics to find the total number of Bytes (Positive Bytes Added) by your participants during your program implementation.
How many "pages created" in Wikimedia by you user cohort during the program implementation? Use Wikimetrics to find the total number of "pages created" by your participants during your program implementation.
How many photos/media were even uploaded to commons as part of the program?
What is the total number of new articles that were created by your presentation as part of the program? if you tracked the articles that participants created, please share the count.
What is the total number of Wikimedia pages that were improved by your participation as part of this event if you tracked the page that participants worked on, please share the count.

Replicating and Learning

Knowing if there are leaders or informative blosg posts are very helpful--May insert links
Reportng on replication and shared learning that relates to your program/event. Please indiciate which of the ofllowing are true about this program impleentation (mark all that apply).
Initial outcomes (Short-term outcomes)-- This is an area where you can share outcomes and processes--such as observations on gender and diversity from your program.

Wikimetrics Opt-in Consent

Providing option to partipants of events to be included in Wikimetrics.
Important to protect identity of participants

using Wikimetrics

Metrics:
- Bytes, Edits, and Pages Created: Bytes added are more useful than a count of edits.
- Treshold and Survival are more important for new users-- When a new account was created, seeing how many edits were made from that time. Based on the number of hours from when they created their account
Consent: When collecting usernames at in-person events, be sure to get their consent to track your cohort via Wikimetrics (See Opt-in form: https://docs.google.com/forms/d/1JjhwWjEU-S3t0I7hPyGQbVlaqPE3lk7gZKN7lIGYNrQ/edit )
Wikimetrics site:
- will store your cohorts forever, and store your reports for 30 days.
- Example: en:Wikipedia:Meetup/NYC/NYU_Law
- Easy way of adding all attendees to Wikimetrics: Copy and Paste Attendees on a blank text file. > Make Plain Text. > "Replace all" with Nothing and you can get a perfect list. > Go to Wikimetrics > Create a Cohort > Create a name for the Cohort > Add Description (optional) > indicate default project > Upload CSV file that was created with plain text usernames > submit
- Creating the report: Click "Create Report" > Pick Metrics, including date and specific time range (UTC) > Configure report, check off individual and average > Get Report > Will give you a report for each metric you set. Will get data > Open CSV file in any spreadsheet program > Establish that it is a comma separated value file and that every comma indicates separate data and every line break indicates a separate line.
- Wikimetrics Mailing list for bugs: Bug Report in Bugzilla or report in other channels that can be found on the Wikimetrics site under "Support"

Future

graphics
debugging

Links

PE&D
Tracking and Reporting Toolkit
https://metrics.wmflabs.org/ (page to create reports)