Grants:APG/Funds Dissemination Committee/Advisory Group/Recommendations/2014/ED Response/zh
As the Grantmaking team does this, it has begun evaluating grants and program impact. We know this is preliminary analysis because movement organizations have only just begun to systematically measure their outcomes. However, this impact analysis has provided a critical framework for my responses, and the questions that guide me and my team as we think together with the community on next steps. The analysis found that organizations were doing many different kinds of offline activities that could be translated into online impact, but there was no direct correlation between the size of grants and their impact: large grants did not necessarily translate into significant impact. The analysis also highlighted considerable investment in organizational structures, systems, and processes, but did not show that these investments produced quantifiable online impact, particularly in terms of expanding (quality) content and growing the contributor base.
Finally, our analysis shows that currently, the proportion of our funds going towards Global North organizations is 92.55%. This is in a context where the rate of growth of readership and contributorship in the Global North is either stable or declining. The active editor rate has declined at 2.3% over the last year, while in the Global South (or our emerging regions and language communities), the active editor growth rate over the same period is 21.7%. Overall editor decline in the Global North was 1.4% while overall editor growth in the Global South was 13.1%. Similarly, gender gap research estimates only a current 8-16% of our contributors identify as female, with major gaps in female-focused content across all Wikipedias and Wikimedia projects.
- The FDC has been a major improvement in our funds dissemination process: it has served as a critical peer-review mechanism of large annual plan grants and led movement-wide conversations around the strategy and impact of Wikimedia organizations.
- Impact is not currently proportionate to the size of grant. In comparing Annual Plan Grants (APG) with Project and Event Grants (PEG) going to Wikimedia organizations, we found that per dollar annual plan grants perform worse than project grants related to content. While this may be due to lack of reporting, the overall sizes of grants are too large to ignore the lack of reportable outcomes.
- Wikimedia organizations have potential for high impact content partnerships or policy change (with national level GLAM and education institutions, for instance) but are not always able to leverage this potential at scale.
- Wikimedia organizations are not yet as well developed in terms of competencies around program design, implementation and evaluation as they would like to be, and as the movement might need them to be. Although significant strides have been made over the last two years, organizations are not yet very effective in tracking data, measuring outcomes and evaluating impact.
- Few organizations have been building and exercising the capability of raising funds outside of the WMF. Most FDC-supported organizations are currently in the Global North where multiple avenues for resource mobilization exist, and building partnerships with local funders is advantageous to our mission.
- The current proportion of Annual Plan Grants going to emerging Wikimedia regions and language communities is 7.45%. In contrast, six Wikimedia organizations funded through the FDC (WMDE, WMFR, WMUK, WMNL, WMCH, WMSE) receive over 57% of the entire WMF grants budget (for individuals to established organizations). The first four chapters receive 60% of all funding to Wikimedia organizations.
- How do we increase the impact of WMF/FDC-funded Wikimedia programs?
- What non-monetary support is needed by Wikimedia organizations to drive impact? How should we structure grants to focus on impact?
- How can Wikimedia organizations leverage local partnerships in ways that significantly improve content on the online Wikimedia projects? How can Wikimedia organizations develop stronger competencies in program design, implementation, and evaluation?
- How can Wikimedia organizations diversify their funding and other resources, reduce risk, and grow their partner networks through funding from local private and charitable organizations?
- How do we increase our support to and impact on emerging regions, languages, and contributors? How do we make our communities and our content more diverse?
- How can the FDC be instrumental in further improving our funding effectiveness?
- We need to distribute funds towards programs with the best performance outcomes – whether in proven areas or experiments – and proportionately lower costs in overheads. In the case of annual plan grants, a key focus should be on supporting organizations and projects that are successful at bringing significantly more content online onto our projects, particularly around critical content gaps in different language Wikipedias or thematic projects.
- Wikimedia organizations should spend a significant proportion of their program budget, time and other resources on programs and projects that are proven to be effective, either with successful pilots or strong track records. They should also be budgeting for experiments, but with lower-cost programs and pilots. I believe innovation occurs both in generating new ideas and improving current processes.
- Training, support in program design and evaluation, as well as other non-monetary support, is needed to improve the effectiveness of programs. The WMF needs to give more focus to supporting and enabling effective and impactful programs. I recognize that volunteer motivation is a significant foundation to all our work and requires programs that support social and group interactions, idea sharing and offline brainstorming. I also recognize that this is some of the work that Wikimedia organizations do in their local context.
- We need to focus on objectives that are best achieved at the local level by a Wikimedia local organization. With focus and support, Wikimedia organizations could be poised to support content partnerships (e.g., with GLAM institutions or governments) as well as policy change (e.g., in enabling Wikipedia to be part of national education policies, or supporting open data and licensing of educational materials).
- Wikimedia organizations should diversify funding through other organizations within their respective regions, especially in the Global North, and track the in-kind resources they receive.
- Diversity in content and contributors are key factors in deepening the quality of our content, and Wikimedia organizations need to be critical drivers in challenging diversity gaps across our movement (including gender and language). We need to balance funds towards diversity and emerging Wikimedia regions and language communities.
Impactful, effective projects and programs should and will be funded and resourced to the best of the movement’s ability. However we have to continually select where and how our resources are best used. These are the questions I have put in front of the WMF. And these are the questions that we need to think through together with respect to funding programs led by volunteers and organizations. As we learn more about what works and what doesn’t in particular contexts, the FDC too can focus more on the productivity and effectiveness of specific programs.
I believe that great ideas come from volunteers that imagine and experiment: I am excited by the incredible passion and motivation I see from individual contributors to informal groups, from emerging to established organizations. I would like to see our grantmaking strategy continue to focus, as our team has been doing, on structured resourcing (not just funding) of different phases of development of ideas and initiatives: from seeding good ideas, to growing and expanding effective ones and phasing out ineffective practices.
I will also commit to offering my thoughts as ED next year at the beginning of fiscal year 2015–16, to review the progress of this past year, and to be clear about our continued and evolving role in supporting and resourcing the movement.
I am responding to the Advisory Group with my plans for how WMF might move these recommendations forward. I outline what we are implementing right away, what we will do a little further down the road, and what we need to discuss with our communities as part of the longer term strategy.
The Advisory Group discussed nine main topics:
- Simplified renewal or continuing funding for applicants
- Segmenting proposals or ‘bands of funding’
- Flexibility in the process
- Metrics and evaluation of the impact
- Grantmaking diversity across regions
- Diversification of funding sources for Wikimedia organizations
- Capacity building of applicants
- Eligibility for FDC membership
- WMF’s involvement as a fund-seeker in the FDC process
The table below summarises my responses to the recommendations.
|FDC AG recommendation||Action||Comments|
Simplified renewal or continuing funding for applicants
Segmenting proposals or ‘bands of funding’
Grants should be supported according to impact, not size of organization
Flexibility in the process (‘guardrails’ to be removed)
Guardrails are being removed from analysis, but any rate of growth should be directly correlated to outcomes on specific programs
|=== 影响的衡量与评估 ===||是|
Global metrics and staff guidance from WMF
Grantmaking diversity across regions
Overall strategy needs to increase resources and focus on diversity to support community and organizational readiness for pilot programs to scale effectively
Diversification of funding sources for Wikimedia organizations (without diverting from program focus)
Reports will begin measuring in-kind resources as well as funds; increase learning and guidance on partnerships
Capacity building of applicants
WMF, chapters and external experts to work on both program and organisational effectiveness
Eligibility for FDC membership (bar staff of Wikimedia organisations, ensure diversity)
No staff on FDC; continue 2 year limits; ensure diversity particularly through Board selection process
WMF’s involvement as a fund-seeker in the FDC process
The WMF will increase the rigour of its own quarterly reporting, ensure public review of annual plan but not be part of the FDC process
Proposal form simplified
In place for October 2014 round of the FDC proposals; reporting format will also focus on key lessons for the movement
Two reports, with financial updates; learning in groups facilitated by WMF
Community consultation on next steps for APG grants and FDC process
End of community consultation by August 2015 with implementation due July 2016
Simplified renewal or continuing funding for applicants
The Advisory Group recommended a simplified renewal funding application process to make it easier for returning grantees to seek funds again.
What we’re doing
Simplifying proposal requirements, and piloting two-year, results-based grants for the APG process.
In the past, Annual Plan Grants have had terms of 12 months, which allowed us to check progress and re-evaluate the appropriateness of the size of the grant as well as the ability of the organization to deliver on intended impact. However, 12 month grants can also limit the ability of grantee organizations to plan beyond one-year horizons on longer projects. A longer-term grant could also reduce the burden on both the grant applicant and the FDC. However, since the APG/FDC grant sizes are already large, the pilot will not include a growth in grant size for the second year and will focus on a smaller, well-defined grant.
Segmenting proposals or ‘bands of funding’
The Advisory Group recommended segmenting proposals into groups by size of request, age of organization, or any other criteria (e.g. by how much of their funding they receive through the FDC).
What we’re doing
We are continuing to develop on our current strategy of supporting different phases of development of movement ideas and initiatives, across the different grants programs that we offer, not only the FDC’s Annual Plan Grants. This would allow us to incubate and grow successful programs. We will be sharing more details of our current thinking on this as part of an overall strategy on supporting effective programs and projects through both funds and non-monetary resources. However, we do not believe that the Annual Plan Grants need specific segmentation because we do not currently see size of grant (or of organization) as being correlated to impact. We will continue to monitor the different groups within the FDC process to see if segmentation at the proposal stage is needed.
I believe that the most important criterion for judging a proposal is current and potential impact. Today, a smaller grant issued by the WMF appears to be more effective than a larger grant. That is counter to well-established successful program metrics. Guidelines and parameters around funding should be encouraging ideas and innovation at a smaller scale, and focusing on optimizing impact in larger scale programs. I would like to continue to urge the FDC and all our other grants committees to judge a proposal on its effectiveness and impact criteria, in its own context and across the portfolio.
Flexibility in the process
The FDC Advisory Group considered whether the -20% to 20% “guardrail system”, as a general guidance for annual increases, was still appropriate in today’s context. They recommended instead having an organization be considered in its own context. Further, they proposed the creation of a "Grantmaking Advisory Group" that can help support an integrated strategy.
What we’re doing
We are no longer using the guardrails in the FDC process. During the pilot period, with very little data on impact, the guardrails served as a guideline and were not meant to be absolute. In fact, the FDC has made several recommendations that have gone above the 20% guideline.
Optimally, increases or decreases in funding should follow successes and failures of the programs they support and in a well designed system, the guardrails would not be necessary. However, given our current limitations around availability of data, we will continue our research on the current suggested maximum or minimum increases in funding. Aligning growth or decreases in funds with successes and failures of corresponding initiatives is one way to ensure that organizations are funded appropriately to their level of impact. This will be considered as part of our larger WMF strategic planning process rather than decided here, and as part of this process, our teams will work to secure data on what kinds of growth are appropriate and sustainable in our movement.
In addition, we’ll be thinking about whether a formal Grantmaking Advisory Group can serve us well and help us integrate our various grantmaking programs. We already informally reach out to a number of experts in different fields to help support our work.
When the FDC process was created, the "guardrails" were devised as a tool to guide applicants in their funding requests and the FDC in their decisions. These funding guidelines outlined a suggested maximum increase or decrease of up to 20% to the previous year’s funding. This +/- 20% tool has been criticized as arbitrary and not meaningful. I am concerned that 20% itself can be a high level of growth for an organization that may be unprepared to manage it well. I would like the FDC to focus on results-based programs rather than staff as justification for budget.
It is worth noting as well that the “guardrails” approach may tend to reinforce and continue funding disparity between the Global North and Global South despite a general understanding that both needs and opportunities may be greater in the South.
The FDC should continue to review each proposal in its own context and consider the “value for money” question: to what extent will the proposed plan create impact, and is that impact commensurate with the funds requested?
We’ll continue to review this going forward.
Metrics and evaluation of the impact
The Advisory Group recommended that WMF define a set of global metrics and indicators for all proposals and reports, while encouraging applicants to self-define other quantitative and qualitative metrics pertinent to their context and strategies. They also noted that grantees should strive to measure how their offline activities impact online work.
What we’re doing
Global metrics have now been created and will build data to allow us to understand impact. The data we collect from global metrics will also give us a bird’s eye view of impact across the movement and allow us to compare the work of APG grantees to the work of others in different grantmaking programs. Other metrics, both qualitative and quantitative, that are relevant to grantees should continue to be proposed. WMF staff, particularly the Learning and Evaluation team, are prepared to offer support to grantees and applicants to help them identify baselines and set targets for metrics.
Further, I endorse the AG’s consideration around alignment with the Wikimedia mission and strategic goals. Offline activities should impact online work, or when they do not directly, this work should be closely linked to the strategic goals.
Metrics and measures of success are critical to knowing whether we’re making a difference. APG grantees have made significant improvements in tracking and reporting progress against targets, but we all still have significant work to do, including WMF. The global metrics are intended to give us a portfolio or aggregate understanding of our impact as a movement, or as a set of grantees.
Grantmaking diversity across regions
The Advisory Group recommended that movement resources be distributed across diverse geographies, including to the Global South. However, they believe that organizational readiness, rather than geographic location, should remain the primary criterion for the FDC’s funding recommendations.
What we’re doing
The WMF has several other initiatives, including grants programs, available to support diverse movement constituents. The WMF Grantmaking team will be trialing a few campaigns to proactively increase diversity in the coming year.
The Advisory Group identified one of the challenges the FDC faces, which is funding diverse geographies. I agree that one criterion is organizational readiness, but even more critical is the success of pilot programs led by diverse communities, that can then scale effectively. Within an organization’s geographic or thematic focus, I think more can be done towards bridging content and contributor gaps related to diversity. For example, some organizations have begun working with diaspora communities (for e.g. Wikimedia Sweden’s recent experiment in this area), and could focus more content partnerships towards this as well. While Wikimedia organizations should continue to support each other across the movement, our key focus should be on supporting the growth of diverse local communities and their leadership.
Diversification of funding sources for Wikimedia organizations
The Advisory Group discussed the diversification of funding for grantees, and encouraged organizations to consider resources beyond financial resources. While they believe there are benefits to increasing diversity of funds supporting our movement, they caution against spending significant time and resources on fundraising. They offered a number of specific guidances.
What we’re doing
Partnership building should be a prerequisite for all Wikimedia organizations requesting funding through the APG process. For those organizations that currently are not ready or are not active in building partnerships, the WMF should provide advice in building out that capacity before providing significant monetary support through the FDC.
If possible and appropriate in their contexts, APG grantees and applicants should be encouraged to develop other sources of partnerships and support (not just financial) beyond Wikimedia that are consistent with our mission. This kind of support and help increases local interest and commitment to Wikimedia projects.
It is essential for larger organizations to have diversified resources, including funding. This provides a healthy sanity check that we are attending to work that is meaningful to others and the community at large.
We know that most movement organizations are heavily reliant on WMF as a source of funds. External partnerships can help promote financial sustainability as well as cultivate meaningful resources and relationships beyond money. These partnerships may actually help our grantees generate more impact, interest and ownership in their local contexts.
Increasing capacity building for applicants
The Advisory Group recommended increasing capacity building for grantees.
What we’re doing
While the FDC remains focused on its mandate to evaluate proposals to disseminate movement funds towards effective programs, WMF staff will be looking more at organizational effectiveness through research this year. However, in a global and distributed organization like ours, we advise that organizations also seek local training and information appropriate for their region.
Our — and the FDC’s — focus should be on increasing on program effectiveness. Organizational capability and stability is an area of focus for WMF grantmaking staff and we have been developing this over time. Work is underway to contextualize organizational effectiveness for the Wikimedia movement, and that body of research will help AffCom and WMF staff to guide and support movement organizations.
As part of the strategic planning process, we will be looking at how our teams support learning and facilitate knowledge sharing on program level activities, including facilitation of programs required for this.
Eligibility for FDC membership
The Advisory Group made several recommendations about the composition of the FDC itself as a committee, including:
- disallowing staff members of FDC-funded organizations from joining, as the strength of the FDC is that it is wholly a volunteer committee,
- considering what the ideal amount of committee turnover is each year, and,
- considering diversity in all ways in FDC membership.
What we’re doing
- Staff of Wikimedia organizations will not be members of the committee.
- Two-year terms will remain constant until we do another significant review of the overall grantmaking strategy, most likely in 2016.
- When the Board is appointing FDC membership, diversity of all kinds will be prioritized.
I believe that one of the strengths of this participatory grantmaking program is that the committee members are volunteers.
In terms of turnover, it is too early to know whether roughly 50% turnover annually is a desirable level as it takes time to come up to speed on issues and best practices around grant programs. Since there are currently no term limits, members may be selected or elected multiple times. I know that the FDC staff will work to ensure a strong orientation program and integration of new members. This issue may need to be addressed again in the future.
Finally, I agree that diversity on the FDC is critical. I am pleased that the current membership is diverse in areas such as gender, region, age, language, and offline and online experience, as well as experience on Wikimedia projects. We will have to ensure that a diverse group of candidates continues to serve the movement on the FDC.
WMF’s involvement as a fundseeker in the FDC process
The Advisory Group recommended a public review of the WMF’s annual plan prior to its approval by the Board, but did not agree on whether the WMF’s plan should go through the FDC process.
What we’re doing
The WMF greatly appreciated the FDC’s review of our plan, which helped shape some of our implementation this past year. However, we will not go through the FDC process again, as it was not easily suited to either the WMF planning process or the FDC deliberations process. At the same time, I take public review and transparency seriously and commit to having a form of public review of WMF’s annual plan. In addition, I would like community review of our work to be more iterative and periodic than only on an annual basis. Since WMF operational practices are being updated, we will share these as updates become available. As we move to the quarterly system, we plan to release our quarterly score cards as well.
The WMF is committed to community review of our plan. As the lead movement infrastructure and support organization, the WMF has different goals and demands on its planning. WMF will integrate an open process into its yearly, quarterly and strategic planning. This is a broader process than the one that can be effectively managed by the FDC process.
The Advisory Group discussed other issues, and I wanted to address a few where we are making changes.
What we’re doing
To keep the process simple, we will not create different proposal forms for different types of applicant organizations within a single grantmaking program like the Annual Plan Grants program. The Grantmaking team has significantly revised the proposal form to make it lighter, shorter and to incorporate global metrics.
After each of the four previous rounds of funding, FDC staff have made minor revisions to the APG proposal form to improve data collection and minimize the back-and-forth needed with applicants following proposal submission. They work to keep the proposal form tight and light, collecting only data needed for analysis by the FDC. However there is often a balance needed with the amount of data to collect for the FDC’s decision-making process, given the FDC makes recommendations on up to $6,000,000. For the sake of analysis, along with due diligence and transparency to our donors, we must work hard to get the right balance.
I would ask the WMF grantmaking team to take on the burden of collecting additional information required for due diligence on larger grants. Increasing review and analysis of effectiveness of larger grants is necessary to ensure we are applying our funds wisely. This work will help us overcome some of the burden currently put on grantees and applicants, as well as address translation and language issues.
We have been considering whether four reports (3 progress and 1 impact) are excessive. The FDC AG also discussed this.
What we’re doing
I would like to reduce the reporting requirements from four to two: a mid-term report and a final impact report. Organizations should document their progress in programs and spending but will not be required to share it with the FDC (unless requested when necessary). Learning patterns and specific metrics are also being encouraged. Long-term we foresee creating different reporting requirements for different sizes of grants to further reduce burden on smaller grants.
We need to get better at capturing data and telling stories. The staff will need to revise and simplify the reporting forms to help enhance this and to include the global metrics, and we’ll be looking at how we can support grantees in this area going forward.
Currently, all APG grantees are required to submit both narrative and financial reports on a quarterly basis: progress reports in Q1, Q2, and Q3, followed by a final impact report including Q4, and totaling four reports. This are heavy requirements when our goal is to spend time on results, not administration. Some have shared that they appreciate the rigor and discipline of reporting, and have shaped their internal reporting systems to align with these requirements. These reports, including the financial reports, have helped inform FDC decisions. However, other applicants feel the reporting is cumbersome and detracts from mission-aligned work. Few members of the community seem to review these reports. In addition, KPMG audited WMF’s reporting requirements and found them to be higher than what was required by law and good practice.
I encourage organizations to submit learning patterns during the “off quarters” in which there is no reporting as a way to continue to ensure documentation of experience and knowledge, and to facilitate learning between movement constituents. A culture of learning is a critical component of the APG process, yet does not need to be done through traditional grant reporting. In place of the progress reports, we will put a greater burden on staff to review existing reports and communication by grantees, as well as to host learning conversations and events.
As we go into this next year of the FDC process, supported by the Advisory Group’s faith in the work of the FDC members and staff over the past two years, I would like to see an open community consultation on impact through the Annual Plan Grants, over the ways in which to further strengthen the mission of the FDC: to review and recommend proposals of high current or potential value to the movement.
In order to do this, I suggest that:
By March 2015: the FDC Board representatives and staff have discussed with the FDC members and current grantees their suggestions for ensuring significant programmatic impact. I recommend discussing different options around grant sizes, and most importantly, working to understand the kind of non-monetary support that our organizations need to do better.
By July 2015: we would have heard from different community members (particularly core contributors) on the suggestions created in partnership with the FDC and Wikimedia organizations.
By August 2015: we would have a decision on any future structure and scope of the FDC to be implemented by July 2016. This would ensure that Wikimedia organizations in this process are consulted and have worked through their strategies for sustainable impact well before any potential changes are launched.
I would like to thank everyone who has put their time and effort in this in the last two years. You have been instrumental in improving our programs as we move towards high-impact, distributed support towards Wikimedia projects.
Members of the FDC Advisory Group
- Richard Ames, FDC Advisory Group
- Thomas d'Souza Buckup, FDC Advisory Group
- Ting Chen, FDC Advisory Group
- Peter Ekman, FDC Advisory Group
- Crystal Hayling, FDC Advisory Group
- Christophe Henner, FDC Advisory Group
- Kathy Reich, FDC Advisory Group
- Pavel Richter, FDC Advisory Group
Members of the Funds Dissemination Committee, current and past:
- Arjuna Rao Chavala (former)
- Anne Clin (current)
- Cristian Consonni (current)
- Dariusz Jemielniak (current; Chair)
- Ali Haidar Khan (current; Vice-Chair; former FDC Advisory Group member)
- Delphine Ménard (current)
- Matanya Moses (current; Secretary)
- Dumisani Ndubane (current)
- Mike Peel (former; former Secretary)
- Yuri Perohanych (former)
- Sydney Poore (current; former FDC Advisory Group member)
- Osmar Valdebenito (current; former FDC Advisory Group member)
- Anders Wennersten (former; former Secretary)
WMF Board Members working with the FDC:
- Frieda Brioschi, Wikimedia Foundation Board of Trustees (current Board representative to the FDC)
- Bishakha Datta, Wikimedia Foundation Board of Trustees (current Board representative to the FDC)
- Patricio Lorente, Wikimedia Foundation Board of Trustees (former Board representative to the FDC, current Vice-Chair)
- María Sefidari, Wikimedia Foundation Board of Trustees (future Board representative to the FDC)
- Jan-Bart de Vreede, Wikimedia Foundation Board of Trustees (former Board representative to the FDC, member of the FDC Advisory Group, current Chair)
- Stu West, Wikimedia Foundation Board of Trustees (member of the FDC Advisory Group, former Chair of the Audit Committee)
And all the Wikimedia organizations and volunteers partnering with us.
感谢您！ Lila Tretikov, WMF