Wikimedia Conference 2014/Documentation/18

18 Programmatic evaluation: How to measure blood, sweat and tears?


tl;dr:

WMUK

edit

Presentation slides here (on Slideshare, contains non free images).

  • Strategic priorities
→ Goal
→ Subgoals ('objectives')
→ TARGETS
  • Linking (coherently) activities to goals
  • Identify responsible people
  • Clear expectations for what to measure
  • 1 Activity → ALL goals? = NOT CLEAR
How well can measure?
  • Picking the BEST thing
  • Not everything you CAN but BEST
Tricky to get the data
  • ease burden with systems
  • sense making - from abstract to clear definitions + measures = measure the same NOT different

Long term is hard to see & measure → identifying short + intermediate PROXY indicators

Reflection Questions
  • Which evaluation targets have you been able to identify and measure against and which have you not?
  • Which goals are you able to identify clearly and set target objectives for?
  • Which goals have you been challenged to identify clear targets or measurements?
  • What, if any, challenges have you faced in actually measuring progress toward identified targets? (i.e., accessing data, information, or participant feedback, the data collection and/or analysis process, etc.)
  • To what extent have you been able to identify suitable short-term and intermediate measures to your long-term impact goals?
  • What, if any, challenges have you run into?
  • What, if any, strategies have you found successful in meeting those challenges?
  • What next steps have you planned?

WMDE

edit
 
Slides (pdf) by Manuel Merz
  • Don't make your goals depend on metrics. The metrics it self don't matter so much.
  • Tools that you would not develop yourself.
What challenges do we face?
  • lots of activities, goals and strategies
  • finding a consensus between our Community, Supervisory Board, the Executive Director and all employees (ongoing challenge for us)
  • organization "in progress"
  • collaborative working culture (everyone wants to involve, it is a good thing to have a participatory innovation, which means the people who do the work[], but it also means it take a lot of times and efforts. Depends on how much you want to achieve and accomplish, and do you take action on it.
Theory of Change Approach
  • Problems are bigger starting with bigger organizations
  • Growth → more difficulty to see interlinking of means to shards goals
  • Most obvious ≠ most useful or important
  • Measurement without purpose can do harm
  • Careful not to promote poor proxies
  • Know your story first
  • Disadvantage = delay to actual data
  • Advantage = actual hypotheses → INDICATORS
  • Easy to go for low haning fruit. DON'T STOP THERE!
  • METRICS BASED ON GOALS NOT GOALS BASED ON METRICS
  • Challenge of large organisations: multiple strategies
  • Difficult to explain long-term importance - ID preconditions/subgoals toward long-term aka 'objectives'
  • Collaboration → everyone wants to participate = good but lots of time + effort (top down would be easier but goals and approach may not be embraced)