Grants:APG/Proposals/2016-2017 round 2/Wikimedia Armenia/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or PEG funding	$176,000	$302,415	+71.83%
Budget	$254,431	$405,415	+59.34%
Staff	3.5	4.25	+21%

Overview of strengths and concerns

This section summarizes the strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

WMAM excels at movement-building, nurtures vibrant communities of contributors, and succeeds at integrating Wikimedia into Armenian society at local, national, and international levels.
WMAM has implemented governance improvements, and is poised to successfully transition to a model with an Executive Director and a governance board.
Although working in a small country, WMAM has unusual opportunities to achieve impact at scale, especially through work with the global Armenian diaspora.

Concerns

WMAM has inadequate financial systems, and needs to be more prepared to monitor and adjust their budget to accommodate changes (such as the hiring of their Executive Director).
WMAM’s proposal for the upcoming year mixes programs with a long list of activities, as well as operational work such as fundraising.This lack of clarity may make this plan difficult to implement and measure effectively.
WMAM’s ability to implement successful programs surpasses their ability to evaluate their results, and WMAM must build capacity in this area.

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

There are important opportunities in WMAM’s environment that may enable its impact to extend throughout Armenia, and beyond its borders. The Armenian diaspora offers a unique opportunity for influence, since there are more Armenians scattered throughout the world than there are in Armenia, and there is a well-connected network of organizations linking these diaspora communities with Yerevan. WMAM seems to be part of a network of organizations that are poised to have a positive impact on civil society in Armenia.

WMAM takes advantage of opportunities in their local environment, as demonstrated by the success of their WikiClubs approach, which brings Wikimedia to rural environments throughout the country. Their WikiCamps model also capitalizes on some of the unique characteristics of the Armenian Wikimedia communities as well as youth culture in Armenia. This ability to create culturally appropriate program approaches may be one of the keys to WMAM’s success.

Armenia is a multilingual society, and participants in Wikimedia Armenia’s programs edit in multiple languages. Many Armenians have competence in at least two languages (Eastern Armenian, Russian), and Armenians in the diaspora community also bring more linguistic diversity to the table (Western Armenian, and at least one other language).

The Armenian Wikimedia projects have vibrant and growing communities with close ties to WMAM, which this enables WMAM to cater closely to their needs. Armenian Wikimedians also enjoy opportunities to meet and collaborate together in person, which makes a chapter that can facilitate offline activities a valued asset.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

At the 2015-16 midpoint WMAM achieved 34,163 articles on Wiktionary and the Armenian Wikipedias, and has already involved 505 participants in its programs (including 196 active editors). WMAM now supports 7 Wikiclubs, which extends the reach of Wikimedia projects throughout the country. Wikimedia Armenia supports an active community of 40 editors on Armenian Wikisource, who are proofreading the 40 books digitized. At the midpoint of the current year, 8 books have already been completely proofread, an accomplishment made especially challenging because of the unreliability of Armenian OCR techniques.

Beyond bringing more content and more new editors, WMAM is extending its reach beyond Armenia through international WikiCamps, and by developing partnerships with organizations that work with the Armenian diaspora. WMAM has cultivated a partnership with the Armenian General Benevolent Union, an internationally renowned organization with a 110 year history of working with Armenian communities around the world. WMAM has also succeeded in including Wikipedia in the national curriculum for teacher training, which means that teachers who are trained to use Wikipedia in the classroom are paid higher salaries.

WMAM’s proposal for the upcoming year includes programs that build on these past successes, so it seems likely they will be able to achieve what they are planning. At this stage, WMAM’s ability to achieve results may surpass their ability to effectively measure results, as WMAM has been focused on the successful implementation of their flagship activities designed to bring in new editors and contribute large amounts of content. We encourage WMAM to implement more refined metrics that capture the depth of their work, and to continually reflect on how their evaluation of this work will influence the planning and implementation of their programs.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

WMAM is an organization in transition. They have recently made improvements to their governance structures. This includes separation between staff and board duties, a newly elected board with a diversity of skills and connections that will benefit the organization in the long term, and the establishment of board committees to help board members focus their work. They are also planning to involve more active wikimedians in their governance structure, especially in the development of their strategy. WMAM is poised to make a successful transition from a working board to a governance board.

WMAM does not have adequate financial systems in place, and will need to improve them in the immediate future. WMAM also made an unusual decision to hire an Executive Director during the current grant period, and reformulated their budget in order to enable this additional hire. This may have made sense for the organization, but it also may have been more effective to plan this transition in advance, or to have a clear idea of how this would impact their budget in other areas before this decision was made. WMAM needs better systems for monitoring their spending against their budget throughout the year so that they are prepared to handle changes when they are necessary.

WMAM has a highly competent staff team with strong roots in the Armenian Wikipedia communities, and has board members with strong community ties. Yet, Wikimedia Armenia has taken on many programs and projects that require staff attention, and at times it may be difficult for them to balance these needs. For example, Wikimedia Armenia plans to expand its support of Wikiclubs in the coming year and it is not clear how staff support will be implemented. The hiring of a new Executive Director is likely to improve WMAM’s ability to effectively direct staff capacity and plan timelines for program implementation. Staff would be able to report to the executive director directly rather than have their activities managed by the board directly. In turn, this would allow the board to focus on high level strategic concerns, and WMAM can develop into a more strategic and effective organization.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan? Wikimedia Armenia has reviewed its strategic plan and adopted a new one for 2017-2020. While this new strategic plan does not include a detailed assessment of risks and opportunities, it describes strategic goals and strategies for implementation.

At this level of funding, it is important for Wikimedia Armenia to clearly articulate their annual plan’s strategic vision. Yet this annual plan includes a long list of programs and the relationship between this long list of programs and their strategic goals is difficult to track. While WMAM envisions a system of interrelated programs that complement one another and work together to achieve a larger vision, the structure of this proposal does not show this adequately because activities like Wikicamps and Wikiclubs are included as separate programs rather than related activities that are part of one program. It appears that the programs presented in this proposal are given equal weight, even when they do not take up similar amounts of organizational resources or focus. A good example of this is the digitization program, which is a relatively minor area of focus for the organization but is featured as a program on par with WMAM’s WikiClubs or WikiCamps, or the inclusion of fundraising at the program level, when it should be listed as an operational measure.

Wikimedia Armenia has been effective in assessing opportunities in their environment (for example, in taking advantage of timely opportunities for collaborations with NGOs in the diaspora community and taking advantage of existing infrastructure and cultural context to leverage their WikiClubs model), yet this vision is not fully articulated in their strategic plan. Nor are risks associated with their operating environment.

Throughout the proposal, Wikimedia Armenia lists many thoughtful strategic observations which they hopefully can better articulate in their next proposal.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

Through WikiClubs, WMAM has succeeded in extending its reach well beyond Yerevan and this network of WikiClubs is likely to grow in the upcoming year. The WikiClubs model is a good example of how WMAM is leveraging local infrastructure and taking advantage of specific opportunities in their culture and context. This model redirects the attention and time of young people in rural areas of Armenia who are otherwise lacking adequate activities to Wikimedia, leading to results that benefit both the Wikimedia movement and the participants. Young people share this experience with teachers and family members, extending the reach of these clubs in their communities. WikiClubs are generating new content and new editors, but are also creating a national network of Armenian people who identify with the Wikimedia movement.

WMAM is expanding its WikiCamp model to include camps for Western Armenian contributors outside of Armenia. This is likely to lead to more content in more languages, and to extend Wikimedia Armenia’s reach. This extended reach can also be achieved in partnership with NGOs that work with Armenian diaspora communities.

WMAM’s programs are effective in drawing in new contributors and generating content, but generating content in the most relevant areas has sometimes been a challenge, especially since many of these contributors are young people. Through programs like Wiki Loves Science, we are seeing WMAM effectively tackle the challenge they have identified in getting more content in scientific topics on Wikipedia. Wiki Loves Science capitalizes on the benefits of facilitating interactions between experienced and inexperienced Wikimedia contributors, who may have different levels of expertise to offer in specific content areas.

WMAM's GLAM work is still in a nascent phase and is not yet producing results at scale. This area may have potential in the future, if WMAM is able to leverage some of the relationships it has been building into effective partnerships. Digitization is a worthwhile activity that engages volunteers, yet seems too minor to be included as its own program.

The choice of editor retention as a grantee-defined metric is both bold and relevant, as this is an area where Wikimedia Armenia is uniquely poised to have impact. Yet, WMAM has not yet developed more sophisticated ways to show the value of their successful work with new editors, for example they are not yet assessing quality, identifying content gaps, or understanding how involvement in Wikimedia is benefiting individuals and Armenian society at large (since they have identified that this is an important way of understanding the value of their work).

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

Wikimedia Armenia is requesting an increase of ~ US$123,000 in their grant amount, most of which will be used to hire a high-level Executive Director. They believe this approach will lead to significant results in the long term. This means that about one third of WMAM’s total grant is dedicated to funding this Executive Director’s salary. While this investment may lead to the impact WMAM hopes for, there is also a possibility that this increased investment will not lead to sufficient impact in the short term.

Other than this, WMAM continues to invest at least $79,700 or 35% of their program expenses in WikiCamps, which is a significant portion of an organization's budget to dedicate to one type of activity. Although these camps achieve impressive results, they require continued investment, so WMAM has plans to make the funding for these camps more sustainable by charging some participants for their participation. It is important that this strategy succeeds if WMAM continues to want to expand the WikiCamp model by including more camps in more locations.

WMAM is also pursuing a model to make its local Wikiclubs more sustainable by involving local anonymous sponsors who can provide funding for materials like snacks and computers, without having a role in influencing the operations of the club. It would be useful for WMAM to more consistently track and quantify the value of in kind donations in its budget.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

Major strength
Strength
Neither a strength nor a concern
Concern
Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Strength	WMAM has a new long-term strategic plan in place, which has sound goals. Some programs seem to be closely related as part of a system, and yet this needs to be more clearly articulated in their strategy and program design.
P2. Potential for impact at scale	Strength	While WMAM is operating in a relatively small country, their potential for impact extends beyond their borders. WMAM is planning to operate programs on a national and international scale. WMAM is achieving deep penetration of Wikimedia projects in Armenian society throughout the country, which has the potential to lead to long term results. Finally, WMAM's programs are unusual and innovative.
P3. Objectives and evaluation methods	Neither	Editor retention is a bold and relevant grantee-defined metric. We encourage WMAM to grow in this area by developing more metrics that reflect the deeper value of their work. Standards in this area will increase along with funding.
P4. Diversity	Strength	WMAM is one of few chapters in their geographic region. Their work contributes to language diversity in Eastern Armenian, Western Armenian, Russian, English, and a number of other languages connected with the diaspora. While WMAM does not have programs that focus on the gender gap, their staff, active volunteers, and contributor base include a notable ratio of female contributors.
Organizational effectiveness
O1. Past results	Major strength	WMAM's results go beyond their high achievements in participation metrics and numbers of articles. WMAM has established long term partnerships that are likely to result in impact at scale, including integrating Wikipedia into the national program for teacher training, and initiating a partnership with the Armenian General Benevolent Union. Wikimedia Armenia is successfully building a movement of Wikimedia contributors on a national scale.
O2. Learning	Strength	While their implementation could have been more structured, WMAM has been willing to adopt the suggestions for improvement made by the FDC regarding their organizational development. They are also adapting and refining their activities as they gather more experience.
O3. Improving movement practices	Strength	Wikimedia Armenia continues to make significant contributions to movement practices for an organization of their size. They have participated in editing contests involving a number of other chapters, and are also contributing actively to the CEE regional group as well as international gatherings of program leaders. Furthermore, their innovative program work has the potential to contribute to the body of movement learning. Standards in this area will grow along with funding.
O4. Community engagement	Major strength	Wikimedia Armenia effectively engages active contributors and volunteers, and is also successful in activating new contributors through its programs. Even while WMAM has expanded their staff, they in turn have very close ties to the community and include volunteers into all aspects of their work.
O5. Capacity	Neither	WMAM has a highly engaged board and a dedicated and skilled staff team. They have also recently taken steps to improve their governance structure. However, WMAM is aiming to achieve a lot that capacity may be stretched thin to implement the long list of programs in this proposal. Hiring an Executive Director has the potential to increase the organization's capacity, and we hope this approach ends up being successful.
Budget
B1. Past budgeting and spending	Neither	Past funding has led to commensurate impact. Yet, WMAM hired an ED during the middle of the current fiscal year and grant period without much planning, requiring them to adjust their budget after the fact. Spending has been directed at program activities, but should be more consistently monitored throughout the year. This may be accomplished through improved financial systems and practices.
B2. Budget is focused on impact	Neither	WMAM's large budget increase in the upcoming year enables them to hire a high level ED. WMAM believes they need to take this step in order to take advantage of timely opportunities in their context. If this approach is successful, there could be a major payoff in terms of results. On the other hand, if this approach fails, this investment might not result in commensurate impact.

This staff proposal assessment is the work of FDC staff and is submitted by: Morgan Jue (WMF) (talk) 18:55, 8 May 2017 (UTC)

Staff proposal assessment framework

Major strength. This is something the organization does very well, and this is a strong indicator of future success.
Strength. This is something that the organization does well, and this could indicate future success.
Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
Concern. This is something that the organization does not do well, and this could make funding the application a risk.
Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O2. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O3. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O4. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O5. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.