Wikimedia Deutschland/Evaluation/Thoughts about Global Metrics

Thoughts about Global Metrics, submitted by WMDE August 28, 2014.

The Wikimedia movement, as it grows in complexity, scope and size, seeks to better understand and maximize its social impact. At the same time, there is growing pressure to increase the accountability of grantees, and find a common way to measure the impact of movement organizations. Finally, shared learning, innovation and capacity building are crucial components as the movement grows.

Global Metrics have been introduced as a way to address the issues of ensuring accountability and demonstrating impact. While maximizing social impact is not the same as maximizing profit, there is an agreed-upon need to compare and assure the efficiency of different projects, programs and movement entities in reaching social impact. Global Metrics were suggested as a means to consolidate and compare the inputs, outputs and outcomes of funded projects, programs and movement entities.

Metrics as a tool in philanthropy (and for that matter, in the business community) have been tested and heatedly discussed for over a decade now (see literature below). There is much knowledge today of what works and what is counterproductive, so there is no need for WMF to reinvent this wheel. The literature and the discussion call for a cautionary, iterative approach.

While WMDE acknowledges that inevitably metrics will be part of the WMF-grantee relationship, we would like to caution WMF decision makers about the risks of overemphasizing this component over the qualitative evaluation methods (“telling the story”), and the other backbone components and movement values: innovation, shared learning and capacity building. These could be negatively affected by placing too much weight on quantitative metrics.

In the following we will briefly touch on the general meaningfulness of global metrics, their possible risks, their suitability for the Wikimedia Foundation’s capacity-building and multi-year general grants approaches, and the problems of a limited grantmaking scope with the diversity of the Wikimedia movement. Finally, we address the political implications for the movement as a whole and the relationships within the movement.

Doubtful rationale

The concept of global metrics by itself is no meaningful solution to the need for accountability. Quantitative metrics alone cannot measure what is important and impactful about an entity or program. “Global” metrics by definition will focus on the "easily measurable" and by design are based on simplistic direct cause and effect models. The model assumes that the most important outcomes achieved by movement entities could be simply aggregated using a set of numbers. The model will fail to measure the incredible diversity of organizations and programs using the same static set of metrics. This simplistic model and its metrics are not useful tools to guide decisions in an environment that requires highly complex systemic strategies. Global metrics alone cannot provide sufficient information to make meaningful comparisons between programs and organizations.

High risk of unintended consequences

The introduction of global metrics to grantmaking is a common intermediary step in the evolution of many grantmakers. Grantmakers start out with a philanthropic motivation, and then tend to become more suspicious of the return on their investment. The next impulse is then to try to quantify social impact, or “define success”. The nonprofit community (including grant makers) continues to discuss the implications of this step and its potentially negative consequences on social impact: Experience shows that often, quantitative metrics constrict the scope for grantees. While this may be intended at first, it often comes with unintended consequences: a focus on the superficial, quantifiable and short-term. What usually falls short is creating sustainable partnerships, encouraging innovation, listening to the community, and working towards game-changing long-term outcomes. Once standardized metrics are used as a basis of decision-making about grant awards, they may start to displace the true goals of the movement. We fear that global metrics might have unintended negative consequences for ambition, foresight and innovation within the Wikimedia movement.

Undermining capacity building and organizational effectiveness

Our shared goal is to build the Wikimedia movement's knowledge about how to sustainably maximize its social impact. We support the Wikimedia Foundation’s thought leadership and efforts towards developing movement organizations through capacity building and multi-year general grants. However, multi-year general grants will require different sets of metrics than project-specific grants. Metrics in this context would focus on organizational capacities, contribution to movement learning, and other long-term social impacts. Experience in the grantmaking community suggests that methods such as site visits, face-to-face conversation, cohort capacity building, personal feedback on reports and evaluation all support mutual trust and understanding, help to build a shared vision and create communities of practice and learning. Global metrics on the other hand work towards a traditional funder/grantee power structure. Systemic impact and learning from failures become lesser priorities in this type of relationship, in which money is traded for numbers.

The political dimension

Global metrics do not only measure outputs, they actually end up defining the goals. As a result, defining global metrics opens up a highly political dimension for our movement. Global metrics would become an accepted interpretation of the practical boundaries of the Wikimedia movement’s objectives. However, Wikimedia organizations, as they evolve over time are likely to develop goals beyond those easily measurable ones, goals that are specific to their geographic, linguistic, economic and cultural context. Organizations will become more independent players in the movement, thus bringing up fundamental questions of power and relationships. As a result, we believe the political component of the discussion cannot be addressed through methodological solutions alone.

While the Wikimedia Foundation emphasizes its role as a partner within the Wikimedia movement, the Chapters’ Dialogue has shown that differing perceptions exist among movement organizations. Top-down decisions on goals and metrics could feed into some of the more negative perceptions of WMF’s role, and in fact turn it more into a traditional grantmaker, rather than a movement partner. The methodological discussion would benefit from being accompanied by a political, movement-wide thought process and dialogue about the Wikimedia movement’s core objectives, a joint vision for collective impact, the role of WMF, and, within all this, the scope and role of grantmaking.

Conclusion and Recommendations

WMDE would strongly caution WMF against relying on global metrics for grantmaking decisions. Standardized quantitative metrics can be useful as one component of evaluation, if designed and thought through well. But they should not be the only way for WMF/FDC to assess the impact of funded programs or organizations. Grantmaking decisions based solely on global metrics, or even worse, a dollar/quantitative output ratio would prevent us from reaching the goals that are core for the movement strategy. At the same time global metrics come with a high risk of unintended consequences for the Wikimedia Foundation’s long-term capacity building efforts and for the Wikimedia movement’s long-term ambitions, innovative spirit and creative energy.

We therefore recommend that standardized metrics, if implemented, are tested in an iterative pilot fashion, with feedback loops, and opportunities to adjust metric design and check the validity of the gathered data. One common way to make standardized metrics more flexible would be to allow grantees to choose from a menu of metrics, so they would be able to report on those that are most meaningful to their situation, organizational stage, and most likely to show the impact of their work. This would also serve as an internal feedback loop on which metrics contain meaning for movement organizations and which do not.

It will be important to identify ways to put numbers in relation to context. This requires that we know about context - by asking questions about organizational capacity (not just governance), about the culture and state of the movement communities served by the organization and about any other factors influencing the validity of the data collected.

There are many areas of movement work which standardized quantitative metrics will be unable to capture. We therefore need to find complementary ways to tell our story. Qualitative metrics, capturing promising innovation and anecdotal success also can add up to a powerful demonstration of impact.

And finally, a related point we have also recently made in the context of reforming the FDC process: Creating a learning community, or a community of practice of movement organizations to maximize the movement’s impact, remains a crucial task, requiring leadership from the Foundation and all of us ‘bigger players’. This task may be harder to achieve when tied too closely to the grantmaking process, or worse, to standardized metrics. Learning from mistakes can be an open and honest process, when it is separated from one’s financial survival. WMDE is looking forward to working with WMF on these and other tasks.

WMF does not have to repeat the mistakes of the ‘metrics mania’ other grantmakers have gone through in the last 10 years. We can apply a more measured, evidence-based approach and still increase grantee accountability and effectiveness.

We believe that our evolving movement is at a crucial point in time, and decisions made hastily now in the name of accountability and “data-driven” decision-making could set WMF-Chapter relationships on a course of mutual distrust, power through control, and top-down goal setting. Our movement values of freedom, shared power and independence should be honored when making these decisions.

Manuel Merz and Nikki Zeuner are the principal authors. Portions of this text were drafted by Nicole Ebber and Christof Pins.

Literature

Bartczak, Lori (2014): The Role of Grantmakers in Collective Impact. URL: http://www.ssireview.org/articles/entry/the_role_of_grantmakers_in_collective_impact

Buteau, Ellie and Timothy Chu (2011): Grantees Report Back. Helpful Reporting and Evaluation Processes. URL: http://www.effectivephilanthropy.org/wp-content/uploads/2014/01/HelpfulReportingEvaluation.pdf

Bernstein, Alison R. (2013): Metrics Mania. The Growing Corporatization of U.S. Philanthropy. URL: http://www.nea.org/home/50022.htm

Fram, Eugene H. and Jerry L. Talley (2012): Using Imperfect Metrics Well. Tracking Progress and Driving Change. URL: https://nonprofitquarterly.org/management/20701-using-imperfect-metrics-well-tracking-progress-and-driving-change.html

Grantmakers for Effective Organizations: Smarter Grantmaking. Learn for Improvement. URL: http://www.geofunders.org/smarter-grantmaking/learn-for-improvement

NCRP (2012): Criteria for Philanthropy at Its Best. Benchmarks to Assess and Enhance Grantmaker Impact. URL: http://www.ncrp.org/files/PAIB-Effectiveness_lowres.pdf

Neary, Dave (2014): When Metrics Go Wrong. URL: http://community.redhat.com/blog/2014/07/when-metrics-go-wrong/

Project Streamline / Jessica Bearman (2008): Drowning in Paperwork, Distracted from Purpose. Challenges and Opportunities in Grant Application and Reporting. URL: http://www.projectstreamline.org/projectstreamline.org/documents/PDF_Report_final.pdf

Schambra, William (2012): Mesmerized by Metrics. Is Philanthropy Engaging in Magical Thinking? URL: https://nonprofitquarterly.org/philanthropy/19421-mesmerized-by-metrics-is-philanthropy-engaging-in-magical-thinking.html

Schambra, William (2013): The Tyranny of Success. Nonprofits and Metrics. URL: https://nonprofitquarterly.org/management/23455-the-tyranny-of-success-nonprofits-and-metrics.html

Urban Institute / Francie Ostrower (2004): Attitudes and Practices Concerning Effective Philanthropy. URL: http://www.urban.org/publications/310986.html