Grants:APG/Proposals/2015-2016 round1/Wikimedia Deutschland e.V./Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) An overview about all applications in the current round; (2) A narrative assessment; (3) An assessment that ranks each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview for all assessments

High quality of proposals and annual plans

Proposals in this round have significantly improved in quality. Proposals are accompanied by detailed strategic plans, and some strategic plans include ways to monitor long term progress over time. With a few exceptions, proposals have been more focused and organized into 2-3 programs with common objectives. Most proposals have been more clearly focused on program work and have included less of an emphasis on operational details. Organizations have included logic models and other clear explanations (e.g. needs assessments and community surveys) to demonstrate why their work is needed and how proposed activities are likely to lead to results.

We appreciate the significant effort that has gone into these proposals, and also believe that the high quality of these proposals will aid the Funds Dissemination Committee in their understanding of each organization’s planned work. Furthermore, quality proposals with clear objectives will provide a strong foundation for each organization’s evaluation of their results in the coming year.

Better targets and effective storytelling

The inclusion of global metrics targets for each proposal has enabled a better understanding of the entire portfolio of Annual Plan Grant proposals in Round 1, and should help show the impact of Annual Plan Grant organizations as a group in a few key areas.

We know global metrics don’t tell the entire story of an organization’s impact. Organizations have included program-specific metrics that measure results that are specific to their work. We see organizations thinking about ways to understand the quality of the content contributed through their work, as well as ways the content is used by contributors (e.g. use rates for media), or readers (e.g. pageviews). Organizations are doing important work that is not yet reflected in global metrics, such as work in the areas of policy advocacy, building the MediaWiki community, and attempts to address community health. Organizations are striving to find good ways to show their achievements in these areas, and to link these achievements to online impact. We appreciate the different ways organizations are working to show the impact of their programs through metrics that are specific to their contexts, and we know this work will continue to be important moving forward.

Beyond the metrics, organizations are improving in their abilities to effectively tell stories and document lessons learned, both in terms of organizational learning and program learning. We have seen significant improvement from last year, with several organizations becoming systematic in the way they document and share their stories. We appreciate where organizations have highlighted important aspects of their work through effective storytelling.

Community health

This year, several organizations are using the lens of community health to structure their work with online contributors. Wikimedia Nederland is doing innovative work around training Arbitration Committee members and administrators. Amical Wikimedia, Wikimedia Nederland, and Wikimedia Israel are addressing online conflicts in their communities. We encourage them to share what they learn about this developing approach with the broader movement and Wikimedia Foundation, as we all stand to gain from these experiences.

Beyond this work, we have seen many organizations implement community surveys to gain a better understanding of the communities they work with and their specific characteristics and challenges. We see movement organizations as clear leaders in this area, since most have direct relationships with their communities and are well-equipped to address challenges that are specific to the geographies and languages they work with. We appreciate that learning how to address challenges in specific Wikimedia communities may be used to improve our understanding of community health in the broader Wikimedia context, which is one reason we see this work as so important.

Many organizations are looking beyond Wikipedia and Commons communities, to focus on other projects. Wikimedia Sverige and Wikimedia CH are both focusing on Wikidata this year, and Wikimedia Israel and Wikimedia Serbia are continuing their strong work on Wiktionary. A number of organizations are focusing on other projects like Wikisource and Wikivoyage, which may offer significant opportunities for improving content and increasing participation and reach.

Policy and legal work

Organizations are focusing on policy areas like copyright reform in a consistent and coordinated way. Wikimedia organizations already have a track record of participating in policy discussions. Some organizations, like Wikimedia Sverige, are seen by others as policy experts in their regions.

However, organizations do not yet have a strong track record with influencing policy, as it remains a nascent area of work. Plans focus on processes and mechanics of policy works rather than identifying specific and achievable policy goals.

Outside of policy work, organizations are also working on supporting contributors with legal expertise (e.g. Wikimedia CH and its work on copyright guidance for users on Commons).

In general, we see this policy work as well-aligned with Wikimedia Foundation’s strategies in these areas. We encourage organizations to continue to articulate how these policy goals benefit Wikimedia projects. We acknowledge that it is challenging to measure the outcomes of this work, and encourage further discussion.

Technology work

Organizations are continuing to request funding for technical work, with a focus on technology tools. In general, we have seen an improvement in plans for technology work and have seen these projects proposed on a smaller scale. Some organizations already have expertise in these areas, while others are working to build more expertise. Beyond Wikimedia Deutschland’s software development plans, Wikimedia Sverige is working with its community to address bug reporting, and Wikimedia Sverige and Wikimedia CH are building relevant tools to expand the use of Wikipedia (text-to-speech, Kiwix and Wikimini). Wikimedia UK is also continuing work on VLE (Virtual Learning Environment) and QRpedia. Wikimedia Israel is working on ways to support technical contributors to MediaWiki, and Wikimedia Ukraine is planning to support the development of gadgets relevant to its community.

Coordination among organizations

Organizations are continuing to coordinate with one another to improve movement practices and accomplish shared goals. Wikimedia Nederland, Wikimedia Österreich, Wikimedia Ukraine, and Amical stand out as high performers in documenting learning and improving movement practices, and Wikimedia Israel, Wikimedia Argentina, and Wikimedia Sverige, are making important contributions too.

Wikimedia Argentina and Wikimedia Ukraine are taking leadership roles in regional collaborations like Iberocoop and the CEE regional group. Organizations are collaborating across languages (e.g. Amical and Wikimedia Sverige), as well as within languages (e.g. DACH). A number of organizations are supporting international program groups and convenings, including Wikimedia Ukraine (Wiki Loves Earth), Wikimedia Nederland (GLAM conference), Wikimedia Österreich (Wikisource conference), Wikimedia Israel (Hackathon). Organizations in Europe are coordinating around EU advocacy issues.

Beyond technology tools, organizations like Wikimedia Österreich, Wikimedia Nederland, Wikimedia Sverige and Wikimedia Israel are working on tools for advocacy work and education work that can benefit the broader movement. Organizations are also engaging actively with the Community Engagement team at Wikimedia Foundation to coordinate with Wikimedia Foundation on movement learning.

Quality of budgets

Budgets in this round varied in quality. Some organizations are still struggling to include an appropriate amount of detail in their budget. We believe more conversation is necessary to agree on the level of detail needed in these budgets, and to establish shared norms for good practices in budgeting and financial reporting. WMF plans to provide more in-depth guidance for budgets in the future.

Wikimedia Sverige and Wikimedia Nederland have performed consistently in the area of making quality budgets and offering transparent financial reporting, and we’ve also seen significant improvement from smaller organizations like Amical and Wikimedia Serbia. Several organizations included staff time and costs for each program. This is a good practice, which can help organizations demonstrate how staff are contributing to different program areas, and tracking this closely will also help organizations understand how their resources are being allocated.

In some cases, however, we received budgets that included line large line items without sufficient detail for understanding the costs. This means that we could not draw strong conclusions about how budgets were weighted toward impact in many cases.

Finally, we noticed large amounts allocated to travel expenses, printing and promotional materials, and it is sometimes difficult to link these significant expenses to commensurate impact. Where expenses are significant, we would like to see clearer links to impact.

Pilot: one restricted grant request

The FDC makes general support grants to support programmatic and operational costs. For the first time, the FDC has received a request for a restricted grant, from Wikimedia Deutschland. We are piloting this approach to specifically support Wikimedia Deutschland’s software development focus. In a restricted grant, funds are allocated to specific line items in an organization’s budget.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or PEG funding	$942,304	$1,683,000 restricted grant
Budget	$1,525,634 (Software Development budget, not including admin)	$1,783,953 (Software Development budget, not including admin)	17%
Staff	19.75	24.25	23%

Overview of strengths and concerns

This section summarizes the strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

Wikidata is strategically aligned with both Wikimedia Foundation goals and movement strategic priorities
Wikidata has grown in recent years, and has potential to have impact on the Wikimedia projects at scale
Strong and capable software development team

Concerns

Budget includes several items that are not likely to lead to significant impact

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

Berlin, where Wikimedia Deutschland is located, is a hub for software development.
Wikimedia Deutschland’s software development team has close working relationships with relevant Wikimedia Foundation teams.
There may be external funding opportunities for Wikidata.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

Wikidata has an increasing number of editors. By mid-2015, numbers of editors, active editors, and very active editors have all increased by 12-14% since the end of 2014.
From the end of 2014 to mid-2015, pages have increased to over 18,000,000 (+8%), items have increased to over 17,800,000 (+7%), items with referenced statements are up by nearly 20% to over 10,000,000, statements were at nearly 66,000,000 (+24%), edits were up by 25% to over 227,700,000, and statements referencing other sources were up by 46% to over 11,600,000. The baseline in September 2015 for the number of data uses from Wikidata in other Wikimedia projects is 153 million.
The software development team has had some delays in delivering in the past (e.g. automated list generation, which was scheduled for 2014).

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

Wikimedia Deutschland is a capable organization, and has recently successfully transitioned Executive Directors. The software engineering team has strong leadership under Abraham Taherivand.
Wikimedia Deutschland has led the movement’s software development on Wikidata and has the expertise needed to do this work.
Wikimedia Deutschland has committed to closer alignment with the product team at Wikimedia Foundation by reporting on the shared product page, and they have been sharing goals and progress over the past two quarters, which will be important to the success of this project.
Wikimedia Deutschland uses common product and engineering development standards, and executes well in both areas. The team has has a high level of technical competency.
Wikimedia Deutschland excels in securing students for school credit, providing mentorship, and getting volunteers involved. Wikimedia Deutschland has included a metric around tracking mentee involvement to help them see how effective their mentorship program is at retaining these mentees.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan?

Wikimedia Deutschland has a strong one-year strategic plan (KOMPASS) that is tightly focused and designed with community input. Wikimedia Deutschland’s KOMPASS is well aligned with their annual plan.
Work on Wikidata work is strongly aligned with movement strategic priorities, while other areas (community centered development and work to build FOSS community) are less aligned, although these programs may yield meaningful results for the German community.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

This request for a restricted grant will focus on software development, which is comprised of Wikidata and other areas of work related to MediaWiki and community building.

Software development program has three goals: 1) Grow reach of Wikidata into the Wikimedia projects and beyond; 2) Build trust in Wikidata through improving data quality and building community of FOSS developers; and 3) Increase innovation potential through community centered software development.
Goal #1, with its large focus on Wikidata, appears to be aligned to movement goals and is likely to be the most impactful. Work under this goal aims to improve Wikidata user interface, support new article creation through article placeholder extension, and secure donations to Wikidata.
Wikimedia Deutschland has identified concerns around quality and intake of data for Wikidata. Wikimedia Deutschland’s product planning for 2016 has aligned to address these concerns. Wikimedia Deutschland has also identified the needs for stronger data analysis and the need to be able to tie that data back to the overall product goals. The organization is planning to move to a more structured Product Management model and their FDC request for operational support, product support and analytics support reflect their needs to mature the product at scale.
Goal #2 appears to have some work that is well-aligned with movement priorities and has good potential for impact. Relevant work in this program includes monitoring and addressing data quality through the Wikidata dashboard and the improvement of data types. Areas of work that are less aligned with movement strategic priorities include supporting the FOSS community, which includes plans to create podcasts, mentoring young developers, and supporting volunteer developers to engage in movements with the goal to have them work on MediaWiki.
Finally, Goal #3, which includes Wikimedia Deutschland’s TCB/Community Tech team’s work to develop and implement features and tools submitted to the community wish list, is less aligned to movement priorities and the least likely of all the goals to lead to impact at scale. Given that this team primarily aims to support the German speaking community, it is unclear the extent to which this work will benefit the movement beyond one language community.

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

Wikimedia Deutschland is proposing to increase funding for software development unit from 1,360,000 Euros in 2015 to 1,590,274 Euros for 2016, an increase of 17% or 230,274 Euros. Administration costs of 448,370 Euros are on top of that.
Wikimedia Deutschland is requesting a restricted grant to cover 1,500,000 Euros of the total 2,038,644 Euros.
Proposed staffing changes include 4.5 FTE new employees: 1 FTE project manager for Wikidata, 2 FTE Engineering Managers (with Annual Plan Grant funds); transfer and expansion of 1.5 FTEs providing fundraising support (non-APG funds).
Some items in the budget seem unlikely to result in impact: Floating Capacity of 119,778 Euros, Partnerships of 74,596 Euros, FOSS Workshops 22,012 Euros and International development events of 55,919 Euros.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

Forthcoming

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

Major strength
Strength
Neither a strength nor a concern
Concern
Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Strength	Wikidata is strategically aligned with both WMF goals and long term strategic focus, although this could be better articulated.
P2. Potential for impact at scale	Strength	Wikidata has high potential as a growing Wikimedia project. WMDE needs to improve its tracking of metrics and strengthen both the quality of the data and input tools for adoption of other data sources in order to understand success. Some of the other areas of work in the Software Development program may not have impact at scale. For instance, the TCB/Community Tech is not replicable to other language communities, and it's not yet clear the extent to which this team serves other languages.
P3. Objectives and evaluation methods	Neither	Together with the WMF, WMDE has defined baseline targets to measure success against five key areas: quality, internal, external, community health and partnership usage. Progress has been made in the development of metrics over the last several months. WMDE needs to further build the metric tracking as it is currently unable to evaluate some important Wikidata metrics. However, many other metrics and indicators are included in this proposal.
P4. Diversity	Neither	There is no evidence of diversity as a priority in the software development program in terms of design and testing. However, the WMDE software development team has some gender diversity.
Organizational effectiveness
O1. Past results	Strength	TCB/Community tech team has results specifically for the German community. Wikidata's general growth has been impressive and continues to increase. Mediawiki core improvements include Catgraph, Deepgraph Gadget, and Watch edits.
O2. Learning	Strength	WMDE's software team is engaging with a wide group of stakeholders. The WMF/WMDE engineering teams have had excellent interactions and communication and have shared learning and experience.
O3. Improving movement practices	Strength	WMDE is documenting and sharing its work and approaches more broadly with the movement. Product Management, for instance, sends out weekly updates on key developments and usage to a public list. The dashboard will be an excellent tool for sharing data publicly.
O4. Community engagement	Strength	The team actively engages on lists and IRC and requests feedback and participation. They host outreach events and actively encourage discussion and project development. They also showcase and highlight content. Additionally the TCB/Community Tech team is surveying and engaging audience in a similar approach to the WMF team.
O5. Capacity	Strength	WMDE has the organizational and technical capacity to deliver on this plan.
Budget
B1. Past budgeting and spending	Concern	WMDE has provided a number of budgets but it has been difficult to understand costs for Wikidata, as distinct from other projects.
B2. Budget is focused on impact	Concern	Some items in budget will be high impact for WMDE and enable it to achieve its goals, particularly in the area of analytical support. However, other items appear to not be high impact, such as "Floating Capacity"(119,778 Euros), Partnerships (74,596 Euros), FOSS Workshops (22,012 Euros) and International development events (55,919) Euros. The largest concern is Floating capacity since in software development the largest costs are labor cost.

This staff proposal assessment is the work of FDC staff and is submitted by: KLove (WMF) (talk) 22:34, 9 November 2015 (UTC)

Staff proposal assessment framework

Major strength. This is something the organization does very well, and this is a strong indicator of future success.
Strength. This is something that the organization does well, and this could indicate future success.
Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
Concern. This is something that the organization does not do well, and this could make funding the application a risk.
Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested.
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O3. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O4. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O5. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O6. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.