Grants:APG/Proposals/2016-2017 round 1/Wikimedia CH/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or other relevant funding	$298,993	$335,537	+12.24%
Budget	$696,574	$869,881	+24.88%
Staff (FTE)	3.1	3.4	+9%

Overview of strengths and concerns

This section summarizes the strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

• WMCH has a new strategic plan in place
• WMCH is developing a fundraising strategy
• WMCH is developing a strong GLAM network within Switzerland.

Concerns

• Past results and potential for impact at scale are not commensurate with funding
• Metrics are not focused on outcomes, and impact will be difficult to measure
• Plan does not give clear sense of priority or make clear connections between activities and results.

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

• WMCH’s multicultural and multilingual environment is an important factor in the way WMCH sees its role and designs its programs.
• WMCH is making an effort in 2017 to initiative more collaborative GLAM work through its Europeana focus, and plan to participate actively in the GLAM coordinators network in order to increase international collaboration in this area.
• An organizational crisis at the board level has made it more difficult for WMCH staff and board to interact with their communities. This appears to have had a negative impact on community relations in the past year.
• The Art+Feminism project in Switzerland has catalyzed opportunities for growth in WMCH’s gender-focused work, as they are working with a growing community of women active in the French language.
• Costs of operating in Switzerland continue to be high. At the same time, WMCH has not yet been successful in generating significant income outside of the Wikimedia movement. In 2015, WMCH did not raise funds from institutional grants, although they have set an external fundraising target of $51,000 for 2017.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

• We continue to be concerned that WMCH does not generate results that are commensurate with funding. WMCH has been reluctant to set relevant outcomes-focused targets for its programs, but in areas where they have done so they are often underachieving against their own targets. Even in areas where they are achieving well against their targets, overall results are low with respect to funding received.
• WMCH met and exceeded its global metrics targets for 2015, although targets were low to begin with. Some targets for 2016 were ambitious (e.g. 1,505 active editors and 58,000 images), but WMCH is not on track to meet those targets (e.g. 205 active editors, 18,443 images). WMCH’s achievement in articles (3,158) puts them on track to achieve their 2016 target of 7,200.
• WMCH’s participation in Art+Feminism yielded 86 articles and involved 88 participants, and WMCH continues to get good results by supporting prolific contributors in areas like photography and press accreditation, although these successes may be highly contingent on the work of individual volunteers. WMCH also achieved success in their organization of opendata.ch general assembly at which they could promote Wikidata.
• In 2016, WMCH grew their network of GLAM partners to 11 institutions. WMCH’s institutions are organized in a way where they can interact regularly and share learning through the “GLAM Wiki Collaboration Group”. This group includes prestigious national-level GLAMs like the Swiss Federal Archives and Swiss National Library, and new 2016 arrivals include SBB Historic and Diplomatic Documents of Switzerland (DDOS).
• In 2016, Wikimedia CH submitted two written statements on copyright revision and net neutrality that were presented to the Swiss Parliament. As a result of interventions, the Swiss Federal Council published a detailed statement affirming that Federal employees could continue to edit Wikipedia.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

• WMCH faced organizational challenges last year, including questions about a potential conflict of interest in the board. Following up on these issues WMCH organized a governance workshop that reasserted principles of good governance (conflict of interest management, effective communication, division of tasks and roles). WMCH is undergoing a governance review.
• WMCH’s board is composed of trustees with diverse linguistic backgrounds, including French, German, and Italian. WMCH currently has no women on its board.
• Since mid-February 2016, WMCH has a new Executive director, who has been instrumental in working on the new strategic plan. WMCH’s new ED is qualified and the transition appears to be going well so far, as is evident in the development of WMCH’s new strategy document.
• WMCH’s staffing plan is organized around language communities. While this structure may make sense in the cultural context of Switzerland, it is not clear how it fits in the new strategic plan and support strategic program development.
• WMCH has provided an interesting visualization of allocation of resources per project, which is an interesting way to track what resources are used.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan?

• The improved strategic plan identifies direction, resources and potential for the organization. This plan includes an analysis of risks and opportunities and focuses on three program areas: Education, GLAM and Community. WMCH identifies three enables to the plan: Communication, Diversity, and Technology/Know-how. The intersection of program areas and enablers is illustrated using a house metaphor.
• The strategic plan does not include long term quantitative targets, although qualitative goals are outlined in each program area, with detailed charts showing how specific programs and activities will lead to long term results. It seems likely that WMCH will be able to track their progress against this plan during the three-year strategic period, and they have explicitly stated that the plan is dynamic, and may undergo revisions during the strategic period.
• The strategic plan’s program areas and enablers are well-reflected in the structure of the annual plan, which is an improvement over previous years, although the area of public relations and partnerships seems the least well-aligned.
• WMCH states in their strategic plan that they want to ensure that program outcomes are commensurate with costs, and this strategy is a move in this direction. While this a commendable goal, it is not clear how this has been implemented in the current annual plan, in which a significant increase is requested before we have evidence of improved outcomes.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

• For an organization of WMCH’s size, with a detailed strategy, this program plan sometimes lacks definition and a clear sense of prioritization. While the plan sometimes reads like a list of activities, at other times specific details that could explain the link between WMCH’s activities and their expected results are missing. For each program, WMCH offers background, goals, metrics, and strategic alignment. What is often missing is an explanation of how the proposed activities will lead to WMCH’s goals and a sense of prioritization among the list of projects included in this plan.
• WMCH’s national-level network of GLAM partners positions them to release large amounts of culturally relevant content and achieve changes in the attitudes and working approaches of GLAM nationwide. In 2017, WMCH is proposing a focus on small GLAMs with an eye to adding culturally relevant content. WMCH does not yet measure the quality or use of media contributed through this GLAM work, which makes it more difficult to appreciate their potential for impact.
• WMCH is measuring the “number of new partnerships” through one of its grantee-defined metrics; while this may be an indicator of influence, it does not capture information about the depth of their partnerships work. For example, even within WMCH’s own sphere of activity, partnerships can vary from collaborations with national-level institutions like the Swiss Federal Archives to work with small GLAMs with limited connections.
• One of WMCH’s grantee-defined metrics is the number of people reached in Switzerland through WMCH’s actions and projects. This metric will be difficult to measure without specific tools that could provide reliable results, and is unlikely to offer much useful information about the impact of WMCH’s work.
• In the area of education, WMCH is planning to adopt practices of other groups by adapting WMDE’s The Open Science Fellows Program and through collaboration with Wiki2Learn. It is good to see that WMCH is plugging into existing opportunities by bringing Wikimedia into broader programs that focus on digital tools in education, since this may enable their education program to have more impact at scale. While we appreciate these thoughtful approaches, we do not see substantial outcomes focused on online impact included in the targets for this program.
• While WMCH has a long history of being close to technological development, notably with its support for Kiwix. Their Content monitoring Tool is also an interesting idea that could enable more targeted community support. It is worrying, however, not to see strong evidence in this plan that technical endeavours are being developed with the wider Wikimedia community. We appreciate that the GLAM statistics tool is being developed in response to the expressed needs of WMCH’s local GLAM network, but we urge WMCH to ensure that this effort builds on existing tools and experiences in the international movement. WMCH’s emphasis on Natural Language Processing with a focus on Wikisource might be impactful, but we did not see evidence that WMCH is connecting with WMSE, who is working on the WikiSpeech tool, or the larger MediaWiki community. Especially in the area of technology, coordination with the international community will be important for the success of any initiative.
• WMCH’s focus on bringing content to Wikisource has good potential for online impact, but in places where funders and institutional partners have significant resources, such digitization projects should be funded by external supporters or partners, rather than by Wikimedia affiliates.
• We encourage WMCH to deepen their work on on the gender gap, and to begin to measure the results of this work. We are encouraged to see momentum in this area.

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

• Despite receiving large grants through the FDC process each year, WMCH has not been delivering results that are commensurate with funding.
• WMCH has a history of underspending and failing to meet revenue targets, yet are planning to increase their budget to $870,000 in 2017. They are also requesting to increase their grant from $299,000 for 2016 to $336,000 for 2017.
• WMCH still relies on APG for 42% of funding in 2016 and plans to decrease reliance to 40% in 2017. WMCH’s non-FDC revenue consists mainly of donations, but they are exploring opportunities for institutional fundraising in 2017.
• The structure by which program expenses are allocated has changed significantly in 2017, and so it is difficult to track the potential impact of this grant increase. Based on percentages of staff time allocated per program, we estimate that WMCH’s resources are concentrated in the Community program (including staff and non-staff expenses).
• WMCH does track the value of in-kind donations and expects $46,000 in 2017.
• WMCH is holding large reserves, about $416,000 or close to a year’s worth of staffing expenses.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

• Major strength
• Strength
• Neither a strength nor a concern
• Concern
• Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Neither	The organization has a new strategic plan in place, which is reflective and high quality. Programs in the plan are aligned with this strategy, which in turn is aligned with potential online impact. However, some programs seem more like a list of ideal or potential activities rather than activities designed to maximize impact within the strategic framework.
P2. Potential for impact at scale	Neither	WMCH continues its emphasis on tools, and some of this work may have potential to achieve impact at scale. Finally, WMCH's network of GLAM partners at the national level is impressive, and may enable national-level scale in GLAM. On the other hand, some programs seem driven by the interests or contributions of individual contributors, and are unlikely to scale and we have not yet seen WMCH generate results at a scale that is commensurate with funding.
P3. Objectives and evaluation methods	Concern	WMCH proposal does not set relevant targets in most areas, and it is not clear how these programs are addressing needs. For example, WMCH does not measure the quality or use of images generated through its GLAM work and many tools lack relevant metrics that can tell the story of their impact.
P4. Diversity	Neither	WMCH has a program focused specifically on gender diversity, which could provide a good foundation for future work. WMCH takes a local lens on a diversity, looking at less-reached areas of Switzerland and working with a variety of languages; however, this is not likely to have a large impact on the wider movement and there is not a clear link to online results.
Organizational effectiveness
O1. Past results	Concern	WMCH is not on track to meet its own targets in several areas, and past achievements have been low across the board. WMCH is achieving progress in growing its network of GLAM partners, although we are not yet able to understand the results in terms of use or quality.
O2. Learning	Strength	WMCH's progress report includes a thoughtful approach to learning, listing how they plan to adapt their programs according to findings. This kind of reflection is also present in their strategy document. WMCH is adapting the ideas of other chapters and groups to strengthen its program plan.
O3. Improving movement practices	Neither	WMCH has shared learning patterns which will be useful for other organizations, and are participating in the FKAGEU. They are actively working to improve coordination on the international level (e.g. GLAM). While they are looking at tools that could have impact across the movement, there is no evidence of their consulting with other movement entities to achieve synergy.
O4. Community engagement	Neither	WMCH is becoming more effective at engaging the community in the creation of their plans, and are continuing their community survey. A number of active contributors are highly engaged in specific projects. Yet, the results of WMCH's community support work are unclear, since metrics around engagement are not included. The organization faces an ongoing challenge in working with several large language communities, including German, Italian and French Wikipedias.
O5. Capacity	Neither	WMCH went through a difficult governance challenge last year. While things have settled down, it took staff time and effort to keep their course through that challenge. WMCH has had a new ED for eight months and the transition seems to be going well. The activities included in WMCH's plan are diverse, and some projects seem to rely on individual contributors. It is not always clear what WMCH is adding to these projects.
Budget
B1. Past budgeting and spending	Major concern	WMCH has been receiving large grants from the FDC for several years, without generating commensurate impact. Past budgeting has not been consistent, with a pattern of underspending that continues in the current year.
B2. Budget is focused on impact	Concern	WMCH is making progress on strategy and showing a nice level of reflection in its reports and plans; however, the lack of relevant targets related to online outcomes is a major concern for a budget and grant amount of this size. While we can imagine that some programs will lead to impact, the link between this budget and program results is not clear.

This staff proposal assessment is the work of FDC staff and is submitted by: Delphine (WMF) (talk) 21:36, 8 November 2016 (UTC)

Staff proposal assessment framework

• Major strength. This is something the organization does very well, and this is a strong indicator of future success.
• Strength. This is something that the organization does well, and this could indicate future success.
• Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
• Concern. This is something that the organization does not do well, and this could make funding the application a risk.
• Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O2. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O3. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O4. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O5. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.