Grants:IdeaLab/Redesigning Global Metrics & its support

Thank you everyone who provided feedback on updating Global Metrics! Global Metrics has been updated based on all the community feedback received. A summary of the new grant metrics requirement and a summary of all the feedback (and changes made based on the feedback) can be found here.

The new page for grant metrics now up as well.

Idealab

Redesigning Global Metrics & its support

Update Global Metrics, as well as the tools and resources that support the collection of that data.

contact emailshoustonwikimedia.org, abittakerwikimedia.org

idea creator• Shouston (WMF)

project manager• Shouston (WMF)

advisor• JAnstee (WMF)

volunteer• Abittaker (WMF)• రహ్మానుద్దీన్

this project needs...

volunteer

developer

designer

community organizer

researcher

created on21:11, 2 June 2016 (UTC)

Background

The Community Resources team, in conjunction with Learning & Evaluation, conducted a two-month long retrospective on Global Metrics, a requirement of all grantees requesting grants from the Wikimedia Foundation. While the results of this retrospective revealed there were benefits to having a requirement such as Global Metrics, it also revealed many issues associated with the structure of the requirement, the metrics used, and the training, resources and tools that support the collection of this information.

Moreover, the retrospective revealed that while Global Metrics was originally intended to provide a high-level overview of certain grant-related outcomes - in the areas of Participation and Content - the information collected through Global Metrics is being used in many more ways than intended, in ways that it was not designed to support well. Particularly, this retrospective revealed that the primary needs Global Metrics should be designed for are as follows:

1. Help those within the Wikimedia movement (specifically grantees, Committees, WMF Staff) need to understand the outputs & outcomes of funded activities. This was the original problem the first iteration of Global Metrics was intended to solve, and has with mixed success.
2. Help WMF Grant staff and Grant Committees understand historical and future achievements, to make informed decisions about grant funding.
3. Help grantees with adaptive planning, by providing useful data that will help improve the planning and designing of their activities / programs

Project idea

What is the problem you're trying to solve?

Based on the result of the retrospective there were three major categories of issues raised:

1. Global Metrics does not include many outcomes that grantees and grant committees find important and useful. The most commonly reported missing outcome was retention.
2. The definition of the current Global Metrics is unclear, inconsistent across grantees, or too aggregated to be useful.
3. The collection of Global Metrics is difficult and time intensive, primarily due to issues with tools, supporting documentation / support, and difficulty concerning attribution.

For more detail on all the issues raised can be found in the retrospective report.

What is your solution?

The solution we are proposing has three components:

1. Simplify the requirement of Global Metrics, i.e. the structure, the measures used, and the way it is included in grant proposals and reports. The goal of simplifying the information requested by WMF is that it will hopefully provide more space for the other metrics and data that grantees find useful. As Global Metrics will never be comprehensive, it will always need to complement other measures and data.
2. Update, simplify and integrate the extensive existing documentation and resources (e.g. the learning patterns), as well a create a new resource: The Metrics Library. This library will aggregate the many quantitative metrics and qualitative measures for capturing outputs & outcomes, that have already been used by current or past grantees, as well as link to the relevant grant or tool (if it exists). This new resource aims to answer the commonly reported issue of "What outcomes could I be capturing for my project? How could I do it? Who has done it before?", by aggregating the collective knowledge in grants and moving toward a community of practice.
3. Update Wikimetrics to become the primary tool for collecting Global Metrics, e.g. resolve long-standing user-interface issues, update the metrics included and their definition, etc.

While it is these three solutions together that can address the three problems stated, the second and third parts depend on the first, and need a longer timeframe to implement. As such, this idea focuses on first part of the solution: simplifying and updating the requirement of Global Metrics.

We have created two proposals based on the suggestions we heard for changing the requirement. There were two other solutions suggested that were not pursued: removing the entire requirement of Global Metrics, and have requirements that are specific to each grant program. See the appendix for more details on why.

We invite your feedback on all parts of these proposals - what you like, what you would change, and suggestions for completely new solutions!

Timeline

Month	Acitvities
8 June 2016	Consultation and voting starts
30 June 2016	Consultation and voting ends
1 - 15 July 2016	Review feedback and any additional proposals
18 July 2016 29 July 2016	Final selection of replacement for Global Metrics. New requirement will be integrated into Project and Conference/Travel grants. Begin discussion on transition plan for APG and Simple APG programs. Begin updating resources and training materials.
August 2016 onward	Continue updating resources, training materials, and tools.
July 2016 to July 2018 or 2019	This next iteration of Global Metrics will last 2-3 years, contingent on the final transition plan for APG & Simple APG. The next evaluation of this grant requirement will be conducted at the end of this timeframe.

Proposed updates to the requirement

How will we evaluate the strengths & weakness of each proposal?

There were many issues raised by community and staff about Global Metrics, more than can be addressed through just changing the metrics or the requirements. However, it's important to recognize where this replacement for Global Metrics will solve issues, create new issues, or lead to the same issues as today. To this end, we identified a set of criteria, based on the project's design principles, that we have used to assess the strengths and weaknesses of each proposed solution (which can been see under the "Assessment of this option against design criteria" section of each proposal).

Proposal 1: 3 shared measures

Structure and measures ▼ Structure Three measures, shared across all grant programs, that grantees will be required to report only when relevant to their project goals. Measures The first measure focuses on Participation. There are two options under consideration: Number of individuals involved. This includes number of people who participated in your program that you reached out to individually or who contributed individually, whom you could follow up with after the event (even if you did not follow up with them). This list of those who should be included or excluded is being created based on definitions currently used by grantees (e.g. APG organizations). Number of editors retained after 1/3/6 months. This is the number of editors who have made at least one edit on any namespaces in any wiki project, 1/3/6 months after the end of the grant. The second measure focuses on Content: Number of pages added or improved, disaggregated by project. For example if a grant works on Wikipedia and Wikisource, this will be two separate metrics: articles added/improved on Wikipedia and primary source document added/proof-read on Wikisource. The third measure focuses on Community building. This will be a qualitative question, focusing on various aspects of community building (e.g. motivation, community health). We welcome suggestions about how this question should be phrased and how the information can be collected systematically / consistently across grantees. Assessment of this option against design criteria ▼ This proposal addresses some of the issues identified in the Global Metrics retrospective, particularly those around the definition of metrics. On average it scores a "medium" on most of the design criteria, with a high score on relevancy and a low score on flexibility. Area Criteria Score Assessment Strike a balance between flexibility and consistency Flexibility Low Would only be "required" when relevant to grant goals, but still depend on other measures to be useful. Strike a balance between flexibility and consistency Consistency High Three metrics would be reported across all programs Be relevant Expressed interest High Metrics all have expressed interest, with updated definitions based on feedback Support accountability WMF Grants to WMF leadership and donors Medium Total participation and retention metrics are used in WMF to demonstrate outcomes of grants to senior leadership. Info not historically used in reports to donors. Support accountability Grantee to WMF Grants Medium-low Gives a broad idea of the outputs of a program, but still lacks context Ease community burden Existence of tools Medium Tools exist to collect all but the Total individuals metric, though these tools need updating. Ease community burden Effort to collect Medium-low Reduction of number of metrics might ease sum burden, but existing tools and resources still need update Ease community burden Availability of guidance Medium Tools exist to collect these measures, but an initial investment is necessary to learn how to use them, and new guidance would have to be written to accommodate new definitions of measures. Comment on the talk page Comment privately via the survey This survey is hosted by Qualtrics and will be under the following privacy statement.

Proposal 2: 3 shared measures + 2 grantee-selected measures

Structure and measures ▼ Structure Three measures, shared across all grant programs, that grantees will be required to report only when relevant to their project goals. The two additional measures will be grantee-selected (where there is no pre-set list). This would create formal space for these "other" or "local" measures that grantees feel tell their story well. The addition of these grantee-selected measure would not replace or exclude any other metrics/measures use for specific projects, program, or activities. Measures The first measure focuses on Participation. There are two options under consideration: Number of individuals involved. This includes number of people who participated in your program that you reached out to individually or who contributed individually, whom you could follow up with after the event (even if you did not follow up with them). This list of those who should be included or excluded is being created based on definitions currently used by grantees (e.g. APG organizations). Number of editors retained after 1/3/6 months. This is the number of editors who have made at least one edit on any namespaces in any wiki project, 1/3/6 months after the end of the grant. The second measure focuses on Content: Number of pages added or improved, disaggregated by project. For example if a grant works on Wikipedia and Wikisource, this will be two separate metrics: articles added/improved on Wikipedia and primary source document added/proof-read on Wikisource. The third measure focuses on Community building. This will be a qualitative question, focusing on various aspects of community building (e.g. motivation, community health). We welcome suggestions about how this question should be phrased and how the information can be collected systematically / consistently across grantees. 2 additional measures, either qualitative or quantitative, defined by the grantee when the grant proposal is submitted. Assessment of this option against design criteria ▼ This proposal resolves some of the issues raised around the definition of metrics, as well as those concerning creating space for other measures. On average it scores a "medium" on most of the design criteria, with high scores on flexibility and relevancy, and medium-low scores on the existence of tools and guidance. Area Criteria Score Assessment Strike a balance between flexibility and consistency Flexibility High Pnly three standard measures to report when relevant, two additional measures open for grantees. Strike a balance between flexibility and consistency Consistency Medium Three measures shared across all programs, but there is a risk that the extra measures submitted will contribute only to the understanding of the program and not cross-program learning. Be relevant Expressed interest High Definitions updated; includes requested space for grantees to submit their own measures. Support accountability WMF Grants to WMF leadership and donors Medium Total participation and retention metrics are used in WMF to demonstrate outcomes of grants to senior leadership. Info not historically used in reports to donors. Support accountability Grantee to WMF Grants Medium-high Two additional measures allow grantees to showcase more facets of their program's objectives and outcomes. Ease community burden Existence of tools Medium Tools exist to collect most of the shared measures, but risk exists that grantees will want to collect metrics that tools do not exist for. Ease community burden Effort to collect Medium Three shared measures can be collected with existing tools, but grantees may select two additional measures that are more difficult to collect. Ease community burden Availability of guidance Medium-low Tools exist to collect the three shared measures, but for the two additional measures, grantees may want to collect the same measures as other grantees but not have the capacity to do so or the guidance to learn how. Comment on the talk page Comment privately via the survey This survey is hosted by Qualtrics and will be under the following privacy statement.

Propose a new solution Make your suggestion

Appendix

Design principles

While this update to Global Metrics aims to address the needs of three distinct situations (i.e. help with understating outcomes, help committees and staff make grant decisions, and help grantees with adaptive planning), these needs represent (at times) conflicting needs: how can one metric be both nuanced enough to provide grantees with relevant data to improve their plans, as well as consistent enough such that WMF can aggregate the data in a meaningful way? How can one set of metrics or learning questions be relevant across all grant programs, given the diversity of grant-funded activities? The answer we heard from grantees, grant committees, and WMF staff is that one set of metrics or learning questions will not be able to meet all these needs perfectly. While there are benefits to having some kind of standard requirement, Global Metrics (or its reincarnation) will always need to complement the other metrics and qualitative data that grantees collect, report, and find useful. To this end, the goals in re-designing Global Metrics is that the updated requirement should:

1. Strike a balance between offering flexibility and retaining consistency to have meaningful, movement-wide information
2. Be relevant to grantee goals, objectives, or measures of success
3. Support accountability, e.g.
   • the information collected should allow WMF to show donors the collective outcomes of all donor dollars invested in grants
   • the information collected should allow grantees to show to WMF the outcomes of the resources (monetary and nonmonetary) invested in grant
4. Ease community burden of identifying and collecting data and reporting
5. Have sufficient community engagement in the development
6. Training, resources, and support that is integrated into the grant processes & resources

Assessment criteria

The following criteria will be used to assess the strengths and weaknesses of each proposed replacement for Global Metrics:

1. Principle: Strike a balance between offering flexibility and retaining enough consistency to have meaningful, movement-wide information
   • Criteria:
     • Flexibility: Does the structure and metrics provide enough flexibility to meet the various needs of grantees, grants committees, and WMF Grant staff?
     • Consistency: Are the metrics/question definition specific enough such that everyone is collecting the same information?
2. Principle: Be relevant. As much as possible, metrics should be relevant to the grant’s goals, objectives/measures of success
   • Criteria: Have grantees demonstrated interest in understanding the outcomes captured by these metrics / questions?
3. Principle: Support accountability, e.g.
   • the information collected should allow Wikimedia Foundation to show donors the collective outcomes of all donor dollars invested in grants
     • Criteria: Will the WMF Grant teams be using the information collected to communicate to WMF leadership, donors, or broader audiences?
   • the information collected should allow grantees to show to the Wikimedia Foundation the outcomes of the resources (monetary and nonmonetary) invested in grant activities
     • Criteria: Will the information reported by grantees demonstrate the outputs/outcomes of their work? Is there clear association of metrics to project goals?
4. Principle: Ease community burden of identifying and collecting data and reporting.
   • Criteria:
     • Do the chosen metrics have working, available tools for collecting the information?
     • For each grant awarded, do we anticipate the training & collection of information to take more than 20% of the volunteer or grantee staff time?
     • Is there guidance available to collect this information?

The following criteria will be used to assess the overall re-design:

1. Principle: Community engagement in development of new solutions or resources
   • Criteria: Have the same number (or more) of grantees & committee members given feedback on these proposed solutions, as in Phase 1 of the project?
2. Principle: Training, resources, and support for quantitative and qualitative data collection needs to be integrated into the grant processes & resources.
   • Criteria:
     • Mutual Participation: Will multiple people be able to contribute their own ideas / experiences around metrics?
     • Shared Understanding: Do the existing or new resources establish a common definition of each metrics / question)

Other ideas that were not pursued

There were two other ideas proposed that were not pursued:

• Removing the requirement of Global Metrics: Global Metrics serves a primary function of helping the Wikimedia movement understand the collective outcomes of grant funded activities. Acknowledging the issues associated with Global Metrics, and the fact that they do not comprehensively capture all outcomes, having a single set of metrics reported across all grantees shown that compared to fiscal year 2013-14 (where reporting of outcomes was fully optional), the 1 year of Global Metrics has already results in 1.5-3 times more participants, images, or pages create/improved across grants than before. (This is likely to increase as about half final reports for the fiscal year 2014-15 grants in PEG and APG have not been submitted yet).
• Have a requirement that is specific to each grant program: While the size and complexity of grants varies between grant programs, there is a lot of commonality in goals, activities, and outcomes in grants across grant programs (e.g. contests, technical development, research, community engagement and support is seen across both Project and Annual Plan grants). As such, having measures specific to simply the size of the grant doesn't make a meaningful distinction in outcomes.

Participants

• Volunteer Would be happy to help volunteer Rahmanuddin (talk) 06:50, 16 June 2016 (UTC)