Wikimedia Conference 2015/Programme/19

19: WMF Grantmaking’s Impact Focus

How to move forward

Four of the five panelists

What was this session about?

The panel initiated a discussion around the definition and measurement of “impact” in the Wikimedia Movement.

What are the next steps to be taken?

Learning and Evaluation at WMF will be conducting an “Impact Analysis” of the FY2013-14 PEG and IEG grants, exploring new qualitative areas of impact beyond Global Metrics. This will feed into future conversations about impact, potentially even at Wikimedia Conference 2016.

Who is the person to reach out to?

Reach out to Feel free to reach out to Learning & Evaluation at WMF or Cornelius, the Program and Engagement Coordinator for the Wikimedia Conference, if you’re interested in the topic and would like to further work on it.

Photos

see the Commons category

This session documentation was approved by two of the speakers.

Session Documentation

Original Description: WMF grantmaking has increasingly been emphasizing the need for impact. What do program officers and committee members mean by impact? What kinds of metrics are useful to consider and how should they be incorporated? Are "global metrics" changing the way that proposals are reviewed? We’ll share some examples of what we’ve learned so far and ask you to share your thinking as well. There will be time dedicated to an open Q&A.
Desired Outcome: Presenting learnings and experiences identifying common challenges. Participants get useful advice from peers about problems they face.
Audience: All organisations seeking WMF grant funding (e.g. FDC or PEG) and anyone interested in the WMF’s approach to grantmaking
Session Format: Panel discussion; 90 min
Speaker: Siko Bouterse (WMF, Director of Community Resources), Alex Wang (WMF, PEG Program Officer), Sati Houston (WMF, Learning & Evaluation), Ido Ivry (WMIL, Grant Advisory Committee member), Kaarel Vaidla (WMEE)

Summary of the session

“Even small things may have high impact.”

Siko introduced and facilitated the panel discussion about the WMF’s Grantmaking Impact Focus. Her first question was why the movement should care about impact. Several panelists said “impact” is what Wikimedians were striving for, in the sense of doing something meaningful, changing/influencing local communities and societies and not wasting time while doing this. One panelist added that even small things (like having good articles in a rare language) may have high impact and, furthermore, but that impact is derived from each organisation’s strategy. However, for APG grantees, those messages couldn’t be delivered via the grant proposal, at the moment, s/he said, but as organisations we need to do a better job of translating our strategic thoughts and long-term hopes into the yearly projects and goals we want to achieve.

“Global Metrics are a start. But context is important too.”

One person of the audience voices her/his difficulties in defining the word “impact”, s/he pled for talking about Global Metrics instead of impact, as it were more specific. Siko picked the thread up and asked the panelists for what Global Metrics were useful for and what kind of limitations did they have. Three panelists agreed that Global Metrics or data in general were a starting point, but a lot of strengths identified by organisations (like partnerships, community support) and the context were not covered. They asked for more context to be provided around metrics, to better understand the numbers. pled for giving the opportunity to provide more context around the metrics. Siko agreed, she said as the movement is so diverse it is a problem for metrics to cover this diversity, particularly how Global Metrics could apply to different types of events and their specific goals (e.g. photo competitions versus social events versus experimental tool building); she asked to think about whether Global Metrics could apply to the specific goals of the chapters.

Siko then asked what kind of information decision-makers on grants (like GAC members) needed from a grant proposer to think about that and decide it. One panelist said that the community likes to do what they like to do and it takes real humility to read a grant proposal that someone put their effort into; so the look for real effort and a real attempt to achieve something unique. S/he said the idea of grantmaking should to be to enable people to be more innovative, which sometimes created conflicts with the idea of having impact. Another panelist pointed out that the Wikimedia grant system was that unique, as you could still improve your grant proposal during the proposal phase, not as in many other grant processes in the world.

“Think more about long-term impact and how to measure it.”

Some panellists mentioned that many important things are very hard to measure. One participant of the audience, staff member of WMDE, expressed his pleasure about the opportunity to discuss “impact”, but noticed that the panel moved straight to the discussion of metrics instead. Although this may seem as the easier thing to discuss, discussing measurement (quantitatively or qualitatively) should only be the second step. S/he said that one should use this opportunity to discuss the impacts the movement tries to achieve. S/he said that e.g. WMDE’s “impact” focus would include planning for sustainable long-term “impacts”, working not alone but in partnerships, and being driven by innovation.

“Reductionism is a danger.”

One participant pled for more work for evaluation, as the movement were still beginning to scratch the surface of evaluation of work. There were resources to address this, but a common will was needed for this. S/he claimed the biggest danger to be reductionism, as measuring only the easy things and leaving out the harder ones, like community health, quality, community changing a core policy, etc. However, s/he said, for the moment there weren’t good ideas how to evaluate these aspects.

Siko asked then about the aspects of time. One panelist said that 60 days (limit for publishing the grant report for a Project & Events Grant (PEG)) were too short to measure impact for some grants, as it may take more time to develop or outcomes to be realized. S/he said that longer-term relationship between granters and grantees helped, so that the Movement, grantee and grants officer learns with them along the way and, much beyond the time of the short grant. Another panelist said, that it depended really from the project, and even legislative work might have impact within the 60 days, though they tend to operate on a much longer time scale.

“Ask your volunteers the right questions.”

Siko asked also the representatives of smaller organisations and groups to participate in the discussion, as most comments were from bigger organisations. The WMDE representative said that the WMF Learning & Evaluation team is building capacity in movement organisations and providing support. S/he referred the audience to the L&E Portal on Meta where a huge base of information and resources is to be found on how to plan, evaluate and report with a focus on outcomes. While this information were suitable for movement organisations, s/he said it could be an overkill for volunteers that apply for small grants. Therefore, s/he said, the WMDE grantmaking team saw this “burden” on itself (as the grant-giving organisation) and not on the volunteers, especially if they were dealing with individuals or small(er) groups. Instead, s/he said, WMDE tried to help volunteer projects by asking the right questions (about impact) and providing backbone functions.

One participant from a user group told a story about how they ran WLM but couldn’t measure impact of those contests until those monuments were gone from an earthquake. S/he suggested “lowering the barrier” for smaller groups to make things easier for them. Siko asked what further suggestions participants of the session might have to lower this barrier, and this participant said that, in general, the Wikimedia Conference and the Wikimania helped a lot, as well as peer-review and peer-to-peer-support would help as well.

At the end, Siko said that the session was intended just to be start of a conversation around “impact” and asked all to find ways to keep having these conversations.