Wikimedia Conference 2014/Documentation/18
18 Programmatic evaluation: How to measure blood, sweat and tears?
- Speakers: Jaime Anstee (WMF), Anasuya Sengupta (WMF), Manuel Merz (WMDE), Daria Cybulska (WMUK)
tl;dr:
WMUK edit
Presentation slides here (on Slideshare, contains non free images).
- Strategic priorities
- → Goal
- → Subgoals ('objectives')
- → TARGETS
- → Subgoals ('objectives')
- → Goal
- Linking (coherently) activities to goals
- Identify responsible people
- Clear expectations for what to measure
- 1 Activity → ALL goals? = NOT CLEAR
- How well can measure?
- Picking the BEST thing
- Not everything you CAN but BEST
- Tricky to get the data
- ease burden with systems
- sense making - from abstract to clear definitions + measures = measure the same NOT different
Long term is hard to see & measure → identifying short + intermediate PROXY indicators
- Reflection Questions
- Which evaluation targets have you been able to identify and measure against and which have you not?
- Which goals are you able to identify clearly and set target objectives for?
- Which goals have you been challenged to identify clear targets or measurements?
- What, if any, challenges have you faced in actually measuring progress toward identified targets? (i.e., accessing data, information, or participant feedback, the data collection and/or analysis process, etc.)
- To what extent have you been able to identify suitable short-term and intermediate measures to your long-term impact goals?
- What, if any, challenges have you run into?
- What, if any, strategies have you found successful in meeting those challenges?
- What next steps have you planned?
WMDE edit
- Don't make your goals depend on metrics. The metrics it self don't matter so much.
- Tools that you would not develop yourself.
- What challenges do we face?
- lots of activities, goals and strategies
- finding a consensus between our Community, Supervisory Board, the Executive Director and all employees (ongoing challenge for us)
- organization "in progress"
- collaborative working culture (everyone wants to involve, it is a good thing to have a participatory innovation, which means the people who do the work[], but it also means it take a lot of times and efforts. Depends on how much you want to achieve and accomplish, and do you take action on it.
- Theory of Change Approach
- Problems are bigger starting with bigger organizations
- Growth → more difficulty to see interlinking of means to shards goals
- Most obvious ≠ most useful or important
- Measurement without purpose can do harm
- Careful not to promote poor proxies
- Know your story first
- Disadvantage = delay to actual data
- Advantage = actual hypotheses → INDICATORS
- Easy to go for low haning fruit. DON'T STOP THERE!
- METRICS BASED ON GOALS NOT GOALS BASED ON METRICS
- Challenge of large organisations: multiple strategies
- Difficult to explain long-term importance - ID preconditions/subgoals toward long-term aka 'objectives'
- Collaboration → everyone wants to participate = good but lots of time + effort (top down would be easier but goals and approach may not be embraced)