Grants talk:APG/Proposals/2015-2016 round1/Wikimedia Deutschland e.V./Staff proposal assessment

Clarifications on a few points from the staff assessment

Thank you KLove (WMF) for taking the time to analyze our proposal, and for the willingness of the FDC to accommodate this unusual proposal for restricted funding. We appreciate the positive recognition of our strategic direction, solidified leadership and organizational capacity, as well as of the importance of Wikidata.

However, there are a few points in the assessment that seem to directly contradict the positive points you make. We feel that these still require some clarification and we hope to allay some concerns raised. Christof Pins (WMDE) (talk) 22:10, 11 November 2015 (UTC)

Thanks for your response and detailed comments, Christof Pins (WMDE), I will respond below to some of these comments. KLove (WMF) (talk) 21:49, 13 November 2015 (UTC)

Thank you for your responses and for updating your assessment, KLove (WMF). We added one further answer to your additional question in the budget section below. Christof Pins (WMDE) (talk) 13:49, 16 November 2015 (UTC)

Metrics

“Together with the WMF, WMDE has defined baseline targets to measure success against five key areas: quality, internal, external, community health and partnership usage. However, WMDE needs to build the metric tracking as it is currently unable to evaluate most of those metrics.”

Unlike FDC staff, we are confident in our capabilities in terms of measurement and evaluation of Wikidata metrics. The metrics applicable in a software development context may be somewhat different from the program metrics FDC staff and committee reviewers are accustomed to in the APG context. The evaluation part of the proposal reflects the complexities of an open source software engineering project. Not everything fits in the SMART Objectives/Targets/Global Metrics mold. However, the multi-level evaluation plan we suggested goes beyond what is typically tracked in the engineering field. We developed it in order to transparently manage the Wikidata project together with the community, measure its impact and, last but not least, to accommodate the perspective of the FDC mechanism and community review. The 2015 WMDE Progress Report likewise demonstrated our comprehensive approach to tracking metrics, and was well received by the reviewers.

Specifically, of the 35 targets/ indicators/ metrics stated in the proposal only 5-6 of these are still subject to metric development efforts in 2016:

• Usage and impact tracking related to software features planned for 2016 (e.g. the article placeholder extension) in most cases will be straightforward to set up, and will occur when the characteristics and roll-out plans for each feature are defined through the agile development process (e.g. “number of articles created/ translated based on the article placeholder”). Other usage metrics such as “total number of Wikidata uses in Wikimedia projects” are in place.

• Our concept for quarterly reviews of the agile development process and publishing the corresponding metrics is indeed new and exciting for us. We are confident to explore an extremely important field of evaluation for the movement by combining agile processes with impact measurement methods. This will enable WMDE to develop its steering capability in the field of software development, and the FDC to provide informed community oversight of the activities.

• Based on learnings from 2015 (Freebase!) there is a clear understanding of qualitative selection criteria for ‘new data partnerships of high quality/ relevance’. These will be refined and articulated as we process, engage in and learn from data partnerships in 2016.

• Finally, some of the future metrics WMoran (WMF) mentioned in his expert input (e.g. for external uses of the Wikidata database by external platforms or certain engagement metrics) will require collaboration between WMDE and WMF that has already started in the last months.

All other metrics to evaluate the components of our software development program are already in place as part of the multi-level evaluation plan described in the proposal.

• All Wikidata KPIs listed in the global metrics section are already being tracked, published and analyzed on a monthly basis.

This is understood. KLove (WMF) (talk) 07:48, 15 November 2015 (UTC)

• Wikidata quality metrics: The majority of these are currently available and measured, (e.g. “% of Wikidata statements with a non-Wikimedia reference” and the “statement per item rate”). We will combine an appropriate mix of relevant metrics on the public dashboard during Q1/Q2 so together they can provide a comprehensive picture for learning and quality improvement.

Looking forward to the development of the dashboard; thanks for this update.

• Other program metrics: We will use several existing Wikipedia statistics to monitor the effect of new software features on Wikipedia projects affected. We have a dedicated metrics plan in place to evaluate on- and off-wiki communications for community centered software development and FOSS outreach.

Thank you for providing this information. We certainly understand that the global metrics model is not relevant to much of the work included in this proposal. The area of concern did not come from the full list of metrics, but focused on the the Wikidata KPIs that were agreed between WMF and WMDE. We understand that progress has been made on these important metrics. We also understand that several are still in development and need to staffing resources and WMF collaboration to be tracked. Based on this, however, we are changing the assessment from "concern" to "neither." Thanks for all the information here. KLove (WMF) (talk) 07:48, 15 November 2015 (UTC)

TCB / Community-oriented software development

“Finally, Goal #3, which includes Wikimedia Deutschland’s TCB/Community Tech team’s work to develop and implement features and tools submitted to the community wish list, is less aligned to movement priorities and the least likely of all the goals to lead to impact at scale. Given that this team primarily aims to support the German speaking community, it is unclear the extent to which this work will benefit the movement beyond one language community.”

We disagree on this point, which was also positively acknowledged by the WMF expert review. The work of the TCB Team is designed to have deliberate, significant and intentional impact beyond German-speaking community at three levels:

• The TCB engineering work may be largely based on requests from the German-speaking community, however the features and improvement that result benefit the software and systems used by all projects in all communities: MediaWiki, Wikibase, Wikidata, Wikimedia Commons. The close collaboration with the WMF Community Tech Team will help to assure this.

• The wishlist component of the community-centered software development process developed by WMDE has been adopted with only a few changes by the new WMF community tech team. The WMF community wishlist survey was published on November 9th. So contrary to this comment, WMDE is at the forefront of innovating and testing processes that are then scaled by the movement.

• Volunteer hackers are supported in co-creating with international movement hacker communities, in order to extend the impact beyond the German communities.

Thank you for providing several examples of the work the TCB team does that benefits communities beyond the German-speaking ones, including the WMF community tech team's adaptation of the TCB team's approach. These are helpful to have identified, and we will make sure the FDC is aware of these. KLove (WMF) (talk) 08:04, 15 November 2015 (UTC)

Budget

“Some items in the budget seem unlikely to result in impact: Floating Capacity of 119,778 Euros, Partnerships of 74,596 Euros, FOSS Workshops 22,012 Euros and International development events of 55,919 Euros.”

In case these items are being listed to pave the way for reducing WMDE’s funding request by creating an artificial line between Wikidata and ‘other projects’: All of these items are integrated with each other, and are crucial to the maintenance and further development of Wikidata.

• Floating Capacity: Please do not punish WMDE for being transparent. This is largely a matter of how costs are presented, in alignment with the activities in the WMDE annual plan. From an accounting perspective, we could have easily allocated the floating engineering capacity cost to each team. Instead we chose to show that agile development requires flexibly deployed human resources, and that this is what enables our teams to react and efficiently work with the ever changing internal and external requirements in the engineering work flows. In addition, this item pays for part of the salary our head of software development, whose leadership you praise in the assessment.

Thanks for this. I am still unclear on how you could allocate the floating capacity cost to each team. Do you mean each activity? I see Floating Capacity in table 6b, where teams are not mentioned. Has WMDE budgeted for floating capacity in previous years? KLove (WMF) (talk) 08:31, 15 November 2015 (UTC)

Hi Katy, we have never allocated software development staff costs to single activities in our budgets before (we projected one overall department budget in the past, and so it did not reveal how much time was spent on what activities). The detailed breakdown was done to give our leadership, as well as the FDC a better understanding of the resources needed for the different activities. However, it is not possible to predict with 100% accuracy how much time will be spent on each activity. By creating a ‘floating capacity’ item for the first time we made the small portion of flexibly allocated resources transparent. Both product teams as well as potentially most of the activity clusters in table 6b will benefit from the ‘floating capacity’ (aka ‘flexibly allocated software development staff’). From an accounting perspective, the line item could have been distributed either across the two teams or across the activities. We chose to show it instead. Cheers, Christof Pins (WMDE) (talk) 13:49, 16 November 2015 (UTC)

• Data Partnerships are an important part of the strategy to grow Wikidata beyond the data in Wikipedia. By obtaining relevant and high quality data, the database will grow, become overall more relevant, widely known and the data that is added will in turn benefit the Wikimedia projects globally. Developing data partnerships requires human resources and a strategically rooted approach. As you write yourself in the Programs section of this assessment, our ‘ FDC request for operational support, product support and analytics support reflect their needs to mature the product at scale’. Cutting here would be strategically unwise and put WMDE into a position of either turning down data partnerships in 2016, or Wikidata quality being compromised by unstructured intake of data.

• FOSS Workshops and International Events are part of the plan to achieve exactly what the assessment seems to be concerned about: impact at a movement level. These activities will make sure that volunteer developers continue to join our movement, stay active, feel appreciated, come to work for us, contribute meaningfully, and work with their peers internationally on projects that benefit the whole movement.

Thank you for providing this context here and above. KLove (WMF) (talk) 08:31, 15 November 2015 (UTC)

Many thanks again for your efforts, KLove (WMF)! Please let us know if you are in need of any further information before the FDC deliberations start. Cheers, Christof Pins (WMDE) (talk) 22:10, 11 November 2015 (UTC)

And thanks for yours, Christof Pins (WMDE). KLove (WMF) (talk) 08:31, 15 November 2015 (UTC)