Grants:APG/Proposals/2015-2016 round 2/Wikimedia Armenia/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or PEG funding	$120,354.75	$193,768.51	61%
Budget	$134,196.18	273,768.51	103%
Staff	1	4	300%

Overview of strengths and concerns

This section summarizes the strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

• Wikimedia Armenia is generating large numbers of articles on Armenian Wikipedia and Armenia Wiktionary through its work with youth.
• The education program is well-designed and supporting programs (wikicamp, wikiclubs) that create a pipeline to impact.
• Wikimedia Armenia has been effective in bringing in new resources to support its work, including external grants and in-kind donations.

Concerns

• International work and increase in grant amount and staff may strain organizational capacity, and may not lead to commensurate impact.
• Wikimedia Armenia’s proposal is not focused enough; lack of strategy and program structure.
• Wikimedia Armenia’s with youth lacks a lens for quality and retention.

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

• As we noted last year, Wikimedia Armenia adapts programs and approaches to its context, and it has tailored them to achieve more impact by leveraging its volunteer community. As these programs grow, we would like to see Wikimedia Armenia become more sophisticated in its approach to evaluating and improving these programs. (Specific suggestions are noted in the Programs section below.)
• We noted last year that Wikimedia Armenia has cultivated an engaged community of offline volunteers and new online contributors, who have maintained an impressive level of engagement with Wikimedia Armenia’s activities over time. Through its work with youth, Wikimedia Armenia may be developing a pipeline of new contributors.
• Last year we noted that Armenia is well placed to address global content gaps through its work on Wikisource and digitization, and we hope to see these content gaps better defined.
• In the current year, Wikimedia Armenia has expanded its involvement with Western Armenian diaspora communities, and plans to continue that work in the coming year, assisted by external funding. At the same time, this new work area creates new challenges in terms of Wikimedia Armenia’s working environment, as they are pursuing projects located outside of Armenia.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

Wikimedia Armenia has shown impressive results from its work with youth, in terms of content generated.

In articles, midpoint achievements exceed those of most other APG organizations reported (with the exception of WMDE and WMSE, who include Wikidata entries in this metric) at 60,458 articles created or improved at the midpoint of the current grant, although article quality is not taken into account in this metric. This is exceeding the current year target of 40,700 articles. Adding a Wiktionary track to the wikicamp has resulted in significant content creation, with 19,476 Wikipedia articles and 40,982 Wiktionary articles at the midpoint.

Programs like WikiCamp have performed well in terms of generating article content and engaging large numbers of participants. We encourage Wikimedia Armenia to build on evaluating these programs, by tracking the online engagement of participants over time and developing ways to assess the quality of articles contributed through this work. This will help us better understand how Wikimedia Armenia’s programs with youth are laying the groundwork for future contributions and contributing to the long term health and quality of the Armenian language projects.

Compared to other APG organizations, Wikimedia Armenia performs less well in the area of images used, with only 3% or 885 out of 26,686 images uploaded used in articles (generated through Wiki Loves Monuments), but exceeding its target of 300 images in use. For individuals involved they have achieved only 600 at the midpoint of over 2,000 targeted. Wikimedia Armenia has already exceeded two of its targets for participation: active editors (target: 418, midpoint: 500) and new editors (target: 389, midpoint: 470). Spring and early summer activities may further drive up these participation metrics.

At the midpoint the current year, Wikimedia Armenia is currently on track to exceed its targets in most areas, and so may need to improve at setting appropriate targets. While Wikimedia Armenia’s past performance indicates that it may meet or exceed its proposed targets, the proposed targets seem too low. 31,850 articles are targeted, which is similar to last year’s target, which Wikimedia Armenia has already exceeded at the midpoint. On the other hand, a target of 4,030 images used seems unrealistic when measured against a midpoint achievement of 855. Targets for participation of 205 active editors, 470 new editors, and 960 total individuals involved are also lower than what Wikimedia Armenia has already achieved at the midpoint of the current year. With the exception of images in use, past performance makes meeting or exceeding the new targets likely, but these low targets do not clearly articulate how the increase in funding will lead to increased impact.

At the midpoint, Wikimedia Armenia had not made significant progress on its goals for the digitization program. While they had 47 active editors participating out of its target of 55, they had only digitized 9 of 90 books, proofread 2 out of its target of 20, and had 4 books used in other programs out of its target of 30.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

As Wikimedia Armenia plans to expand its budget, grant amount, and staff significantly with this proposal, standards for organizational effectiveness increase accordingly. We are concerned that this rapid expansion in budget and staff may strain the organization’s capacity.

Wikimedia Armenia has been able to secure significant in-kind donations (office space) as well as external funding for its Western Armenian work, an important achievement. At the same time, the focus on Western Armenian brings some concerns. Work on Western Armenian requires international coordination. While funding for this program is secured independently, it will likely have a significant organizational cost, and is unlikely to show significant impact at scale based on current results.

As the organization expands its staff, careful attention should be paid to increasing volunteer engagement at similar levels, to ensure that programs are not becoming overly reliant on staff at the expense of volunteer leadership.

We also encourage Wikimedia Armenia to improve its evaluation methods to better understand the longer term impact of its work and develop more sophisticated program approaches.

Wikimedia Armenia has been active in sharing its work with the broader movement through regular blog posts and including its stories in the education newsletter. Beyond this, they have been active in the CEE regional group and have engaged in cross-group collaborations, such as the Armenian-Georgian collaboration that produced 1,500 articles and for which winners received the prize of a trip to visit each other’s groups. Wikimenia Armenia has also shared its work at Wikimania and the WikiConference Costroma. In the current year, they are proposing a collaboration with Wikimedia Norge.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan? Wikimedia Armenia has a five-year strategic plan covering five areas: promote education, increase content quality, increase reach, promote volunteerism, and collaboration within the movement and with the open knowledge movement more generally. It is not clearly articulated how each program in the proposal aligns with Wikimedia Armenia’s strategic plan. At the same time, given the emphasis on content throughout the plan, connections with online impact are clear.

At the level of funding requested, we would expect a more developed strategy and better structured proposal, including a way to measure long-term progress and more detailed descriptions about how and why strategic priorities were determined and will be implemented. The program plan appears unfocused and dispersed, even though there are clear points of connection between the different programs.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

While WikiCamp is a program that yields significant results, we would like to see Wikimedia Armenia evaluate and adapt the program. This is particularly important as many in the movement are watching this program and learning from it. After several instances of WikiCamp, it would be good to track participant engagement over time and understand whether there is longer term retention. It would also be good to attempt to measure the quality of contributions from camp participants, as well. We look forward to what Wikimedia Armenia learns from running the camp with different age groups (University Students).

We appreciate the way the Wikiclubs, Wikicamp, and the Education Program seem to create a pipeline for student engagement, extending from primary schools to the university level. It would be nice if the structure of Wikimedia Armenia’s programs were better connected with obvious links, in order to create better integration between programs and a more focused plan overall. Wikimedia Armenia may be laying a foundation for future success through its work with youth now, which may be difficult to measure in the short term.

We would like to see work with teachers more clearly defined with metrics that reflect the program’s theory of change. We don’t know whether measuring on wiki engagement will be an effective way of predicting how teachers use Wikipedia in the classroom, as actual editing activity may be very different from teaching others to edit. Also, it is not clear what skills will be targeted through these training workshops. It would be good to target and measure progress on teachers gaining specific skills.

We appreciate the active participation of the Wikisource community in the digitization program. Although this program seems worthwhile and has minimal costs, it would be good to better understand how the content being digitized will be used in order to articulate the value of this program. We appreciate that Wikimedia Armenia has set specific goals for use of this content and is interested in how this works in the current year.

Some programs lack specific or relevant objectives:

• In particular, we are concerned about the microgrants program. While microgrants to contributors in Armenia may be a good idea in some respects, this program lacks goals. Giving grants in accordance with the procedures outlined with the APG grant agreement will require the organization’s attention. Furthermore, this does not seem significant enough to be a standalone program and might be best integrated with other community support work.
• The goals of international involvement are not clear, and this work is a significant portion of Wikimedia Armenia’s overall budget. Beyond being present at international events, we encourage Wikimedia Armenia to make the most of opportunities to learn and share with the international movement in online contexts.
• While the community support program is designed to help sustain the community, the only SMART objectives for this program are around content, and so it will be difficult to know if Wikimedia Armenia is achieving the intended outcomes.

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

Wikimedia Armenia is proposing to significantly increase its overall budget from about $135,000 projected spending in the current year to roughly $274,000, increase staff from 1 to 4 FTE, and increase its grant amount from about $120,000 to $194,000. This is a 61% increase from its current grant.

Wikimedia Armenia is receiving significant external support in the form of funding for its Western Armenian work. They also receive a donated office and community space. We appreciate the detailed budget that includes expenses by program and line item.

Wikimedia Armenia is increasing its investment in travel, food, and accommodation for participants in its WikiCamp activities, and now in its Wiki Loves Science edit-a-thons. These activities have been very successful, but it is not clear that this significant investment is always needed to achieve success. Furthermore, this cost-intensive approach will not be infinitely scalable or easily adapted to other contexts where costs are higher. At nearly $80K, these expenses make up 30% of Wikimedia Armenia’s total budget.

Staff expenses are more than tripling from $15K in the current year to $50K. Besides the increase in WikiCamp expenses, costs for community support are tripling from $10K to $32K to cover office renovations. Education costs are increasing from a few hundred dollars to $13K.

It is not clear how increases in budget are linked to increased impact. At the same time, the budget is heavily weighted toward programs like WikiCamp that have been effective in the past, with the exception of the large budget for the international program, which may lead to limited increase in impact.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

• Major strength
• Strength
• Neither a strength nor a concern
• Concern
• Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Neither	WMAM is proposing a significant increase in its overall budget and APG funding this year, and along with this, standards for strategic planning are raised. WMAM has a strategic plan in place, but the lack of focus in the structure of its proposed programs raises concerns about its strategic direction. For example, programs like WikiClubs, WikiCamp, and the Education Program seem to be closely related through a similar theory of change, and yet are not integrated approaches. Small programs like Microgrants are presented as standalone programs, detracting from the focus of the plan overall. The plan, however, is still well aligned with online impact.
P2. Potential for impact at scale	Neither	Programs could lead to significant online impact at scale, since WMAM is doing work to lay a strong foundation of future success through the engagement and retention of young contributors. On the other hand, results may not be commensurate with the funding requested as program costs for WikiCamp increase significantly without clear benefits in terms of program results.
P3. Objectives and evaluation methods	Neither	Some programs include a plan for measuring results and ensuring learning, while SMART objectives are missing for a few programs (e.g. WikiClubs, Microgrants). In a few cases, such as the work focusing on teacher training, WMAM may not be purusing nuanced enough metrics to target the right outcomes (e.g. teacher activity on wiki may not be a good indicator of whether teachers will engage others with wikipedia). After several iterations of WikiCamp, we would like to see more detailed metrics tracking retention over time, in order to understand the longer term effects of these programs.
P4. Diversity	Strength	Programs will expand the participation in and reach of the Wikimedia movement, including work in the CEE region and with a minority language globally (Western Armenian). While not specifically targeting the gender gap, programs and the organization's leadership have a good gender balance.
Organizational effectiveness
O1. Past results	Major strength	WMAM is achieving significant results in this important area, with successful programs that are generating large amounts of content and may be leading to the retention of new contrinbutors. WMAM has also done a good job of documenting its specific approaches. Finally, the programs in the proposal are based on past programs in which WMAM has had success. For instance, in the first six months of the year, WMAM has improved almost 20K articles on WP, and nearly 41K pages on Wiktionary. More information about quality and retention will be needed to maintain WMAM's momentum in this area.
O2. Learning	Strength	WMAM is addressing risks and challenges effectively and is learning from and documenting its experiences. WMAM can improve in applying this learning to its programs to refine them over time, and in showing in its proposals and reports how it is doing so. Standards in this area are rising now that WMAM is entering its second year of the FDC process, and requesting a significant grant and budget increase.
O3. Improving movement practices	Strength	WMAM's performance in this area has remained consistent. This organization engages effectively within the Central & Eastern European group as well as the broader movement, and is exploring more cross-chapter collaborations, like its work with WMNO.
O4. Community engagement	Strength	Volunteers and communities are engaged with the planning and implementation of WMAM's programs. We will want to carefully monitor this area if WMAM expands its staff significantly in the coming year, to ensure that community and volunteer engagement grows alongside staff capacity.
O5. Capacity	Neither	While WMAM has volunteer resources and expertise to do the plan proposed, we are concerned that international work (Western Armenian), though externally funded, will put a strain on the organization overall. We also have concerns that staff growth may constrain the organization's capacity, though we appreciate part of that work is externally funded. Given the ambitious program plans, we are concerned about the organization's ability to manage such rapid growth in staff and budget effectively.
Budget
B1. Past budgeting and spending	Strength	WMAM has a history of budgeting realistically and managing funds effectively in the past, allocating funds toward programmatic impact. While performance has remained consistent, standards in this area have increased as WMAM is entering its second year in the FDC process and is requesting a significant grant increase.
B2. Budget is focused on impact	Neither	Funds are allocated to programs and activities with strong potential for programmatic impact (e.g. WikiCamp), and yet it is not clear whether the increases in funding and staff proposed will lead to commensurate impact. WMAM anticipates spending $82k for Wikicamps ($62k last year), and $58k for Western Armenian ($15k last year), and $32k for commuinty support (nearly $10k last year). Staff expenses are increasing from $15k to $50k. These are significant increases all around for the organization.

This staff proposal assessment is the work of FDC staff and is submitted by: KLove (WMF) (talk) 23:35, 9 May 2016 (UTC)

Staff proposal assessment framework

• Major strength. This is something the organization does very well, and this is a strong indicator of future success.
• Strength. This is something that the organization does well, and this could indicate future success.
• Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
• Concern. This is something that the organization does not do well, and this could make funding the application a risk.
• Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O3. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O4. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O5. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O6. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.