Grants:APG/Proposals/2015-2016 round 2/Wikimedia Norge/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or PEG funding	$141,351.60	$212,602.00	50%
Budget	$149,396.00	$225,473.04	51%
Staff	1.3	2	54%

Overview of strengths and concerns

This section summarizes the strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

• Partnerships with prestigious GLAM institutions
• Devoting program focus and resources specifically to gender gap work
• Thoughtful approach to online competitions, adapting activities from Wikimedia Armenia

Concerns

• Wikimedia Norge still does not supply relevant and required targets for content achievements, including images in use and articles created or improved
• Increase from $149,396 projected spending to $225,473 proposed, with 94% reliance on APG funding not likely to result in commensurate impact
• While progress is being made, link between offline activities and online impact is still weak

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

• Last year we noted that Wikimedia Norge is cultivating opportunities in its local context to engage with GLAM institutions and encourage content donations. WMNO continues to nurture connections with prestigious GLAM institutions (e.g. Arts Council of Norway, National Library, National Archive, Museum of Nordland, and Nobel Prize Center). GLAM partners have offered prizes for online competitions and have also released content, like the Nanset collection from the National Library Wikimedia Norge seems well-placed to deepen GLAM partnerships in the years to come.
• Last year we were concerned how unclear Wikimedia Norge is engaging with online contributors or how it connects these offline activities with online work. We have noticed improvements in this area with more efforts to engage with communities online, and we encourage Wikimedia Norge to strengthen its connections with online communities.
• Last year we noted that Wikimedia Norge works with several small language communities, which could be a barrier for achieving progress in increasing participation. In the most recent impact report, Wikimedia Norge expressed that these communities are declining or remaining stagnant. This is an ongoing concern.
• Last year we appreciated that Wikimedia Norge is taking advantage of opportunities to increase female participation in its projects. We recognize that this work has deepened during the current year, although without a clear link between offline activities and online impact.
• Wikimedia Norge has opportunities for fundraising in its local context. We congratulate WMNO on the progress they have made, yet these have not yet been able to sustain Wikimedia Norge’s maintenance or growth.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

While generally on track to meet or exceed last years targets in all areas where targets were provided, last year’s targets are very low across the board. Several important targets were missing altogether (including images in use) or missing for particular programs (Academia). Last year, we noted that exceedingly low targets for content, missing targets for Academia, and Gender Gap program targets focused solely on offline participation would make it difficult to assess Wikimedia Norge’s performance in the current year against its plan. Indeed, this has made the organization’s progress more difficult to assess relative to its own goals.

Wikimedia Norge has developed impressive partnerships with prestigious GLAM institutions like the Nobel Center, Nordslandmuseet, and Arts Council of Norway, but the results of these partnerships are still unclear. Wikimedia Norge has not included images in use (a required global metric) and so it is difficult to understand if content generated through its programs is relevant or useful, and there is no targeted assessment of content gaps that are being addressed through this work.

Wikimedia Norge has been able to reach a very limited number of people (150 at the midpoint of the current year, with 300 total individuals targeted throughout all of the current year) through its focus on costly (in terms of budget allocations as well as staff time) offline events like workshops, edit-a-thons, and train the trainers events. This strategy should only be continued if we begin to see a commensurate increase in content, retention, or targeted and measurable skill building. WMNO has just started work on a branch of the Wikipedia Library, which involves community members and staff, and will focus on supporting editors and enable the creation of quality content. We believe this is a good approach for WMNO.

Based on current and past performance, the participation targets in the current year’s proposal do not seem feasible. Wikimedia Norge plans to more than double the number of individuals involved in its work from 303 to 690 (156 achieved at the midpoint), triple the number of active editors involved from 42 to 150 (46 achieved at the midpoint), and quadruple the number of new editors involved from 75 to 325 (60 achieved at the midpoint). Even by taking into account this year’s overperformance on participation targets, these numbers will be difficult to achieve. Targets for content are missing, with a target for number of articles included only for its Gender Gap work and targets for images used missing across all programs. Without revisions to these targets, it will not be possible to assess the organization progress against its goals in the coming year either.

Wikimedia Norge has set a target of retaining 5 editors of the 30 new editors participating in its Gender Gap program, which is a very high retention rate. We have not seen any retention results yet reported for this program, so it is not clear if this is feasible.

Wikimedia Norge’s participation metrics are particularly low when compared with other APG organizations. Wikimedia Norge’s midpoint achievement of 156 individuals involved is the lowest of any APG organization and their achievement of 60 new editors involved is also the lowest of any APG organization. Wikimedia Norge’s midpoint achievement of 46 active editors involved is the lowest of any APG organization except WMUK. Articles created or improved at 1,294 are low when compared with many APG organizations, although a handful of organizations (WMIL, WMFR, WMUK, WMCH) had lower achievements in articles at their midpoints.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

Events of the past year have created concerns about the organization’s ability to manage its finances, cash flow, and staff, although the organization is working to address these issues and appears to be stabilizing. In the past year, Wikimedia Norge has made a number of improvements to its financial systems and processes,which will need to be maintained to ensure the effectiveness of the organization. The organization now has stable staff and Board in place, which is an important foundation for programmatic success. Last year, we lauded the improvements Wikimedia Norge made to make its board more diverse. The board has continued to maintain healthy gender and skills diversity in the current year, with 3 of 7 board members female.

Last year we also expressed concerns about Wikimedia Norge’s proposed staff growth. We urged Wikimedia Norge to look at increasing volunteer engagement in its activities and rely less on staff, which is a significant expense in Norway. In addition to raising external funds and leveraging partner contributions, this would create a more robust ecosystem of resources which could enable the maintenance or expansion of its programs without diverting excessive resources to this region. This will be key to Wikimedia Norge’s future effectiveness, since expanding staff in Norway is cost prohibitive.

The organization is learning from its programs and adapting accordingly, shifting from a focus on face-to-face events and working through online forums and programs. In the progress report, Wikimedia Norge noted that they started out organizing many small events for new editors, but learned that few would continue to edit. This year, Wikimedia Norge has begun to shift from offline events to supporting online editing contests, using project pages, and designing new homepages to increase communication with community members. Wikimedia Norge has worked toward a more focused plan at the advice of FDC staff, and these improvements are evident.

Wikimedia Norge is still not producing detailed monthly or annual activity reports, although we have seen great improvements in its APG reporting. We encourage the organization to continue to document its learning and to share how it is adapting its programs based on that experience.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan? Wikimedia Norge has produced a new strategic plan for the period of 2016-2020, which is significantly different from its earlier strategic plan from 2011-2015. The new plan focuses on quality, diversity, access and the free knowledge environment. Unlike the previous strategy, this plan is focused on program impact rather than on institutional concerns. However, the plan does not include a way to track the organization’s progress against long term goals. It also appears to focus on offline approaches, but the link to online impact is not clear. We appreciate that Wikimedia Norge is exploring opportunities to connect with online communities through this plan, and also that they are conducting a survey to better understand its online communities, and encourage them to move in this strategic direction.

Last year, we offered feedback that the Academia program did not seem strongly aligned with WMF’s strategic focus areas, and this still appears to be the case. The link between this model and online outcomes is still not evident, so we are unclear on the rationale for continuing to build that program, especially with significant staff investment.

In the current proposal, Wikimedia Norge improved significantly in articulating the logic behind its programs, especially by including a description of the desired change it is seeking and providing some detailed logic models for select programs.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

There are significant GLAM opportunities in Norway, and Wikimedia Norge is building strong relationships with prestigious institutions. This could open the gateway for large amounts of quality content. At the same time, Wikimedia Norge does not set targets for images in use or articles created or improved, or measure quality, and so it is not possible to understand the value of this work. If a GLAM and content focus is indeed a cornerstone of the chapter, we expect to see more sophisticated targets and metrics in this area.

We appreciate that Wikimedia Norge is devoting a program specifically for addressing the gender gap. Wikimedia Norge has focused on female editing workshops and has led seminars about the gender gap; however they have yet to build a strong connection between this work and online impact or demonstrate that this program is leading to the type of retention they are targeting.

The benefits of the Academia program are not clear, and in other contexts, similar models have not been successful in the past. One major challenge encountered by others in the Wikimedia movement is that people in Academia tend not to have the time available to contribute to the projects, despite their valuable expertise. It’s not clear from the presented logic models how Wikimedia Norge will expect to achieve better results with its program.

Within Community Support, we appreciate that Wikimedia Norge is integrating more approaches that include reaching out to contributors in online spaces and organizing online contests and activities. This will be important to increasing community engagement in its activities and for exploring lower cost models that may lead to impact. Yet, this program does not offer relevant global metrics targets or SMART objectives that would articulate its potential impact. While this program is designed to offer useful support for community members in their work, there are no relevant targets around content or retention. Given the significant investment of funds and staff time in community support work, this relevant lack of targets and SMART objectives is problematic.

Last year we suggested that Wikimedia Norge take time to reflect on and refine its strategies around editing competitions, which have been a successful strategy for them in the past. We appreciate the detailed logic model explaining improvements made to the Wiki Loves Monuments competition, and encourage more detailed reflection in this area.

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

Last year, FDC staff expressed concerns about Wikimedia Norge’s lack of success with external fundraising, given the high costs of staff, and opportunities in its local context to draw in additional funds. We recommended that Wikimedia Norge not consider expanding its staff until more success in external fundraising has been achieved. We acknowledge that WMNO has attempted to raise significant resources outside of the movement, and has had some success in doing so, although these external funds do not yet make up a significant enough portion of their budget to justify growth since they are 95% reliant on APG funding.

Despite these recommendations, Wikimedia Norge has moved forward with a requested staff increase that is heavily reliant on APG funding. As external funding has been challenging for Wikimedia Norge to raise, we are concerned about the large reliance on APG funding. Both last year and this year, APG funding is likely to comprise 95% of their organizational budget.

Wikimedia Norge has overcome serious financial and governance challenges in the past two years, and the organization appears stable and in good financial health; however, this also does not lay a strong foundation for rapid expansion proposed. Last year, we urged Wikimedia Norge to provide more details about its planned spending. We appreciate receiving a more detailed budget after FDC staff requested one. In the future, more granularity around program expenses and staff allocations may be helpful in understanding Wikimedia Norge’s spending, but we acknowledge the steps that are being taken in this direction.

The budget is weighted toward staff and operating expenses, with minimal program expenses. 44% of total program expenses go toward events (line items not specified), 27% to merchandise (line items not specified) and 18% to volunteer travel (line items not specified). The bulk of total expenses go toward Wikimedia Norge’s requested 2.0 FTE. The Community Support program receives the largest FTE allocation with a 0.5 FTE community manager, while the Academia program receives the second largest FTE allocation at 3.0 FTE. In terms of program allocations, community support at $19K makes up the bulk of program expenses at nearly 54% of the the $36K program total and Academia around 18% of the program expenses at $6,500.

More strategically aligned programs like GLAM and the Gender Gap work receive a smaller allocation of FTEs and a smaller portion of the program budget, while Community Support and Academia appear to be receiving more of the organization’s resources.

We have serious concerns that the proposed increase in funding will not lead to increased impact, based on the results achieved by Wikimedia Norge in 2014-2015 and progress achieved so far in the current year (see Past Results section of this assessment for more details). Furthermore, at this level of funding, we would expect some impact to scale beyond Norway, and we see no evidence that this is likely.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

• Major strength
• Strength
• Neither a strength nor a concern
• Concern
• Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Strength	WMNO has a five year strategic plan, and is adopting a new strategic and more focused approach. Programs are moving to emphasize online impact rather than face-to-face interactions. The strategic alignment of Academic work is not clear.
P2. Potential for impact at scale	Concern	Programs are unlikely to lead to significant online impact at scale, and corresponding to the amount of funds requested. WMNO focuses on small language wikipedias that will have limited impact outside of Norway and the Nordic region. Although there is potential for productive GLAM work with prestigious institutions in Norway, the participation results targeted are not likely to be achieved based on current performance, and the content results targeted are very low. For example, WMNO has targeted 690 individuals involved in the proposed year, while in the current year 156 individuals have been involved so far out of 303 targeted for the current year. On the other hand, WMNO is targeting only 150 articles improved or created in the proposed year (because article targets are missing for most programs) and has not provided a target for images in use.
P3. Objectives and evaluation methods	Concern	While programs include a basic plan for measuring results and ensuring learning, with SMART objectives and logic models, critical targets are missing from WMNO's evaluation plan. While WMNO notes they will look at quality of images by some partners, WMNO has not set targets for images in use (or other reasonable quality targets), despite its emphasis on GLAM and quality. In particular, for the Academic program, it is unclear how results related to online impact will be achieved.
P4. Diversity	Neither	Programs will expand the participation of women in the Wikimedia movement, although programs are focused on a part of the world that is currently well-served by the Wikimedia sites, and so this work is unlikely to expand reach to less represented groups.
Organizational effectiveness
O1. Past results	Concern	We are concerned about the amount that is being spent on this work, without commensurate results. Low performance in participation and content has occured year over year. While WMNO is on track to exceed many of its current targets, these targets set a very low bar for achievement. For example, WMNO reported just 21 invidiuals involved in its activities in its impact report for FY 2014-2015; while they are projecting 303 individuals involved in the current year this consists of just 42 active editors and represents a very high cost per participant based on a projected spend of nearly $160K.
O2. Learning	Strength	WMNO has been addressing risks and challenges effectively, especially internal challenges around finance and governance, and is improving in learning from and documenting its experiences, and is applying learning to improve its programs. Reports on their APG have improved significantly. WMNO has noted learning from its experiences caused them to focus more on online activities (e.g. contests and work with the Wikipedia Library).
O3. Improving movement practices	Neither	This organization is improving at sharing learning about its work with the broader movement and beyond, and helping others in the movement achieve more impact, through engagement in cross-chapter collaborations and within the Nordic chapter group. At the requested level of funding, standards in this area are high, and we would like to see more impact emerge out of these collaborations.
O4. Community engagement	Neither	We are encourage by the progress WMNO is making in effectively engaging communities and volunteers in the planning and implementation of its work. In the coming year, WMNO is planning a survey of its editors and is working on expanding its reach online through use of online spaces like village pumps. We hope to see these strategies pay off in the coming year, as increasing volunteer engagement will be successful to WMNO's ability to sustain its work without relying heavily on staff.
O5. Capacity	Neither	To some extent, this organization has the resources and ability to do the plan proposed. We appreciate that WMNO has successfully emerged from a difficult period of board and organization transition, overcoming challenges in governance and finance. At the same time, we are concerned by the ongoing lack of volunteer engagement and reliance on staff which raises questions about the organizations current and future capacity. Finally, based on current performance it seems unlikely that WMNO can achieve the targets outlined in this plan.
Budget
B1. Past budgeting and spending	Concern	This organization has a poor track record with budgeting realistically and managing funds effectively in the past. In particular, budgets offered to the FDC have lacked sufficient detail to understand how funds are used, although we appreciated that WMNO provided more details in a timely way when asked this round. We are concerned with past cash flow issues that had serious ramifications for the organization and threatened its stability, and with poor management of finances in the past. Finally, we have not seen significant grants result in proportionate impact in past years.
B2. Budget is focused on impact	Major concern	This budget is focused on staff costs and programs which have not been leading to results commensurate with funds requested in the past. We do not believe the significant increase proposed (from a grant of about $141,352 to $212,602) will lead to the impact proposed or to significant impact overall. External funding would be needed to support any staff growth at this stage, and increased volunteer engagement should accompany that growth.

This staff proposal assessment is the work of FDC staff and is submitted by: Winifred Olliff (WMF Program Officer) talk 23:31, 9 May 2016 (UTC)

Staff proposal assessment framework

• Major strength. This is something the organization does very well, and this is a strong indicator of future success.
• Strength. This is something that the organization does well, and this could indicate future success.
• Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
• Concern. This is something that the organization does not do well, and this could make funding the application a risk.
• Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O3. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O4. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O5. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O6. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.