Grants:IdeaLab/Reimagining WMF grants/Evaluation 2017

Reimagining WMF grants

Evaluation 2017

Introduction

This report includes feedback from staff and applicants during the first year of the restructured grant programs. In February 2017, we sent an evaluation survey to anyone that had applied for a Rapid Grant, Project Grant, or Conference Grant since the programs launched in April, June, and July 2016, respectively. The first year of the new Simple Annual Plan Grants program was evaluated in September 2016. You can find that report here.

This report includes:

• An assessment of how the programs are meeting the needs of grantees, including a summary of results through February 2017.
• An analysis of what we learned about the different application processes from applicants and staff.
• Next steps for improving the grant programs.

Overall, applicants are satisfied with their experience, and the restructure appears to be meeting the needs it was designed to address. 97% of respondents rated their overall experience with Wikimedia Foundation grants as "good" or "excellent". We have identified some specific areas for improvement in the grant process, highlighted by program below. Many thanks to all applicants who provided feedback for the purposes of this evaluation.

Note: We did not survey members of the Project and Conference grant committees and this was an oversight. Some committee members were also grant applicants, but were not asked specifically about their experience as a committee member. We welcome their feedback on the talk page and will be sure to include them in the next evaluation.

Survey

By February 1, 2017 we had reviewed 98 Rapid Grants, 33 Project Grants, and 7 Conference Grants.

• The survey was sent to 130 Rapid, Project, and Conference grant applicants (some grants proposals have more than one applicant).
• 58 people completed the survey.
• The majority of survey respondents applied as individuals (23) or Wikimedia User Groups (18), which is predictable given the nature of the Rapid and Project Grants programs.

Rapid Grants

• 
  Wikivacaciones, Colombia
• 
  Wiki Loves Monuments, Tunisia
• 
  Wiki Loves Butterfly, India
• 
  Neuroscience and Mathematics Editathon, Brazil

Program overview as of February 1, 2017:

• 98 grant applications received
• 89 grants approved
• $105,177 USD approved
• 13 days average from submission of application to decision
• 53 (60%) grants to emerging communities (e.g. Brazil, India, Turkey, Tunisia, Sri Lanka, Morocco, and more)
• 18 (20%) grants to support gender diversity-related activities

In creating the Rapid Grants program, our goal was to meet the need for speed, simplicity, and flexibility with grants that are relatively low risk. We aimed to offer quick funding decisions for requests or experiments that don't need extensive community deliberation throughout the year. The feedback we received from this survey strongly indicates that we are meeting this goal (see details below). However, in order to understand the value of the program based on grant outcomes and longer-term impact, we will be conducting an evaluation of the first year of grants in FY 17/18. The exact timing of the evaluation will be dependent on staff capacity.

Applicant experience

Applicants were asked a number of questions about their experience, including if it was clear which program to apply to, if planning resources and information were comprehensive, if they understood where they were in the process at all stages, if they knew where to ask questions, and if they received useful feedback. For these questions, more than 90% of applicants either strongly or somewhat agreed the process was easy. 85% agreed that they received timely feedback on their application.

Applicants indicated that the hardest parts of the process are receiving money, collecting metrics, and reporting.

Of the 16 Rapid Grant applicants who received an Individual Engagement Grant or Project & Events Grant in the past, 72% said their experience applying for a Rapid Grant was somewhat or much better.

What's working well

• Easy application
• Quick decisions
• Results focused
• Transparent process
• Less time spent reporting
• Much more approachable program for new grantees

What should we improved

• Language is a barrier -- application forms should be translated into other languages
• The process should be even faster
• It would be helpful to include case studies for each kind of project, not just sample applications
• There should be opportunities to expand the budget (beyond the $2,000 limit) or combine grants (e.g. one Rapid Grant for a small conference and 5 Rapid Grants for individual speakers)
• Guidelines should include a requirement that user groups need to ensure all members of the group have been notified of and support the grant application process beyond the process of existing Granted benefits and consideration for the New grantees to procure funding easier for the non astute computer Using grant applications

Staff reflections

Rapid Grants have been an amazing way to support new and diverse communities. While we could have predicted the program would be popular, we did not realize how many applications we would receive in the first year! As this was a pilot, we were not staffed sufficiently to be able to review and process so many grants. As a short-term solution, we enlisted all Program Officers on the Community Resources team to rotate reviewing applications every month, as well as additional grants administration support to cover the increase. The downside is that knowledge about specific projects or communities is hard to share between Program Officers and there was less consistency in review. To address this issue, we have asked to hire a junior Program Officer to manage Rapid Grants and an additional Grants Administrator in the 2017/2018 Annual Plan. Additionally, because we designed the program to be rapid and lightweight, there is less time to provide more hands-on mentorship or capacity building as with our other grants programs. We hope that as individuals or groups apply for subsequent grants, a longer-term relationship can be developed.

Next steps

Based on the feedback provided, we will work on the following:

• Include more specific guidelines around community notification, completed
• Include more information up-front about bank account requirements and the process for receiving funds, June 2017
• Include links to real grant example for each type of project, June 2017
• Translate application and report forms into the most common languages, August 2017
• Increase staff capacity to review and manage Rapid Grants, timing to be decided (TBD)
• Better understand challenges with metrics and reporting in an effort to make those processes easier, ongoing

Project Grants

• 
  Offline Wikipedia Mongolia project
• 
  Wikidata module
• 
  Whose Knowledge? with Okvir and One World Platform, Sarajevo
• 
  Fountain tool for online writing contests

Program overview as of February 2017:

• 37 grant applications received
• 17 grants approved
• $328,839 USD approved
• 5 (30%) grants to emerging communities
• 3 (17%) grants to support gender diversity-related activities

Applicant experience

Applicants were asked a number of questions about their experience, including if it was clear which program to apply to, if they knew where to ask questions, and if they received timely feedback from staff and committee. For these questions, more than 80% of applicants either strongly or somewhat agreed the process was easy or they were satisfied. 75% agreed that they received useful feedback about their application.

Applicants indicated that the hardest parts of the process are the review process in general and receiving money.

Of the 9 Project Grant applicants who received an Individual Engagement Grant or Project & Events Grant in the past, 56% said their experience applying for a Project Grant was somewhat or much better. 34% said the experience was about the same.

What's working well

• Committee structure is very supportive and there is great interaction between grant applicants, committee members, and WMF staff
• The process supports learning and iteration through an open and collaborative process

What should we improve

• Some respondents indicated that two rounds a year is too limiting (others said there are enough)
• Structural bias toward applicants with more skill in proposal-writing (tends to favor applicants from the United States and Europe)
• Reporting requirements can be burdensome

Staff reflections

By basing the new Project Grants program on our successful experience with Individual Engagement Grants (IEG), we have been able to preserve a number of elements that work well. We have an active, effective committee with a wide diversity of skills and expertise. Through their written comments on the proposals, it is possible to give each applicant much more extensive and pluralistic feedback than would be possible with a single program officer.

We increased the maximum request amount from $30,000 USD to $100,000 USD and this has made risk management harder. Given the diversity of both projects and applicants, it is challenging to adequately review larger-cost (and often larger-risk) projects and still keep the timeline for the round reasonably short. However, the diversity of perspectives from the committee contribute to much better risk management, since each person has skills and knowledge that the program officer does not.

We can attribute a significant amount of the committee's effectiveness to the round schedule, which allows committee members to plan ahead for focused bursts of activity, resulting in high levels of participation and more clarity about decisionmaking. We launched the program offering four rounds per year, but quickly realized this was too much for the program officer to manage and switched back to the two round model. Even so, since two grantmaking programs were combined into one and the volume of grants has risen, it is still challenging to provide as much hands-on support during the grant period as we would like.

Next steps

• Review reporting requirements and consider ways to improve and possibly streamline
  • By September 2017: Streamline metrics requirements for Final Report. Review qualitative reporting requirements and identify changes to make them more useful and less burdensome for grantees.
• Investigate additional ways to overcome structural bias in the program
  • Completed: video tutorials for each section of the grant application in order to make the application process easier for new applicants.
  • By end of 2017 Round 2: Experiment with offering additional opportunities for mentorship and support to applicants from emerging Wikimedia communities

Conference Grants

• 
  GLAM Coordinators Meeting 2017, Paris, France
• 
  WikiConference North America 2016, San Diego, USA
• 
  WikiIndaba 2017, Accra, Ghana
• 
  Wikimedia Conference 2017, Berlin, Germany

Program overview as of February 2017:

• 9 grant applications received
• 8 grants approved
• $202,631 USD approved
• The annual Wikimedia Conference is also managed under the Conference Grants program with a budget of $261,000 USD.

Applicant experience

Applicants were asked a number of questions about their experience, including if it was clear which program to apply to, if they could access all the information they needed to apply for a grant, if they understood where they were in the process at all stages, and if they received timely feedback on their requests. For these questions, more than 75% of applicants either somewhat or strongly agreed the process was relatively easy.

Applicants indicated that the hardest part of the process is receiving useful feedback on the grant proposal from WMF staff/committee, collecting metrics, and reporting on the success of the event.

Of the 5 Conference Grant applicants who received an Individual Engagement Grant or Project & Events Grant in the past, 60% said their experience applying for a Conference Grant was somewhat better.

What's working well

• Support for engaging communities around needs and goals, surveying, and program development
• Assistance with scholarship process and review
• Support with navigating complex vendor and other payment issues

What should we improve

• Clarifying what events are eligible and what are WMF funding priorities
• Getting timely feedback from WMF

Staff reflections

Creating a dedicated program to support conference grantees had enabled us to provide more focused guidance throughout the conference planning period, especially in the areas of soliciting community engagement, scholarship review, and program planning. In the case of WikiIndaba 2017, this new process worked especially well. However, as a pilot program there are still a number of challenges to work through in order to provide the most effective and timely support to applicants. As a new program, the budget was built around experience from previous years and designed to meet the needs of what we expected people to ask for. However, plans change, new conference come up during the year, and other ones are postponed. This has made it very difficult to both manage the budget and manage staff capacity throughout the year. We are thinking about different options to better understand people's plans at the beginning of each fiscal year. We also need to understand each grantee's financial situation in terms of available fiscal sponsorships or payment plans for grantees who are not registered NGOs and/or do not have the capacity to manage a large grant. This issue slowed down the review for a number of grants and was an unexpected challenge to the process.

Beyond grant reports, we are looking to create a more robust evaluation process to understand the outcomes and longer-term impacts of conferences. Conferences require a large amount of human and financial resources. They are also impossible to measure quantitatively, so we need to develop a better process for understanding their impact on individuals, communities, and the movement as a whole. If you have ideas about how to do this, we welcome your suggestions on the talk page!

Next steps

• Improve the quality and timeliness of feedback from WMF. In the second round of conference grant applications in 2017, WMF will send applicants a status update email each Friday during the review period to update them on the status of the grant review.
• Make it easier to find useful information to prepare the grant request. In May and June 2017, conference planning resources will be moved to more prominent locations on the conference grant portal.
• Provide support to grantees in report preparation. In May 2017 the program officer began working with grantees to prepare their reports.
• Begin making introductions to others (committee members and past grantees) with relevant experience. During the second conference grant cycle of 2017, the program officer will ask grant committee members and past grantees to answer questions or provide support to current grantees on areas where they have experience or expertise, such as scholarship review or planning a conference program.

Additional feedback

We also received feedback not specific to one individual grant program. We have summarized the most useful points below.

What should we improve

• Create a map that identifies grants that were given in geographical areas so applicants can understand where they can go to guidance and resources (brochures, banners, etc.)
• Provide clearer guidance on what long-term support for a project might look like
• Provide clearer and easier connections between the different materials (e.g. on the grant application form, include links to useful materials)
• Establish a communication and collaboration channel like Facebook, which is easier and quicker, especially for non-native English speakers
• Provide feedback on a clearer timeline (most likely for Rapid and Conference Grants)
• Provide more feedback on grant reports (most likely for Rapid Grants)

Next steps

As we bring on more staff capacity towards the end of 2017, we will revisit the points listed above and hope to make improvements in all areas. In terms of a clearer timeline for feedback on Rapid and Conference grants, we will clarify expectations on the relevant portals. We also hope implementing rounds for Conference Grants moving forward will make the process more clear.

Thank you

Many thanks to all of the grant applicants that participated in the survey! We really appreciate your feedback. If you have more to share, please post on the discussion page, participate in future surveys, and/or email us directly.

Rapid Grants: rapidgrants"at"wikimedia.org

Conference Grants: conferencegrants"at"wikimedia.org

Project Grants: projectgrants"at"wikimedia.org