Grants talk:IdeaLab/Redesigning Global Metrics & its support

Discussion of the measures proposed

For Participation measures, there are two proposed: Individuals Involved and Editors retained.

• Do you prefer one of these over another?
• Would you change the definition of either metric? If so, how?

For Community Building:

• What does Community Building mean to you?
• How important or useful is it to collect this information?

For Content:

• Would you change the definition of this metric? If so, how?

Discussion of the structures proposed - Proposal 1 vs. Proposal 2

1. If you would like to vote for Proposal 1 or Proposal 2, can you do so using {{support}} or you can comment without voting using {{comment}}
2. Add your signature with 4 tildes ~~~~
3. If you add a comment, sign at the end of your comment.
4. To respond to others, use :

Proposal 1: 3 shared measures

Support

• ...

Comment

• ...

Proposal 2: 3 shared measures + 2 grantee selected measures

Support

•  Support--Gabrielle Marie WMCH (talk) 07:45, 27 June 2016 (UTC)
•  Support--Sandra Rientjes - Wikimedia Nederland (talk) 12:45, 27 June 2016 (UTC)

Comment

• Re-thinking the metrics approach is a great idea so thank you for undertaking this project that will support us all. This suggestion adds flexibility to the current system so I believe it's going in the right direction, I am very curious to see what comes out from the survey. I think an integrated reporting tool on project pages in Wikipedia could be a fantastic alternative but I have no idea from a technical side what it would entail and if it is feasible. As of today, we would still need to have an ongoing quantitative approach, for example for chapters we need to be able to commit to targets for annual funding requests to the WMF so this proposal seems like a realistic alternative that could be implemented in a very reasonable amount of time and that could also include the qualitative approach which is very much needed. I guess those projects & changes happen over time and iteration is the key to success. The idea of applying the new guidelines to a test group is a good idea just to confirm the assumptions. If WMCH can be of any support, please feel free to let us know. --Gabrielle Marie WMCH (talk) 07:59, 27 June 2016 (UTC)
• This option for me provides a balance between uniform data-gathering, and allowing for the fact that there is diversity among grantees. Focussing on APG-grantees: all will have activities aimed at improving content and participation, and all support communities. But some will emphasize lobbying, others awareness raising among the general public or specific target groups. The two 'extra' metrics will allow chapters to monitor, quantify and justify their work in such areas. I am glad to see that the metric for content will be 'restyled' to fit more Wikimedia-projects - not just Wikipedia and Commons. Concerning possible metrics for community health: together with Wikimedia Israel we developed a framework setting out parameters of community health and identifying angles for ways in which a chapter can develop activities to reinforce these parameters. Perhaps this is of interest? Ofcourse there will always be the problem that it can be difficult to 'prove' the link between chapter/grantee activities and specific measurable impact. E.g., if a chapters generates a lot of positive media-coverage for the Wikimedia-projects, can this be directly linked to increased revenue during banner fundraising? Narrative reporting and room for storytelling remains important. Sandra Rientjes - Wikimedia Nederland (talk) 12:45, 27 June 2016 (UTC)

Suggest you own solution

Leave a comment below!

• Adding a suggestion from a 1:1 interview: Create dedicated capacity to support smaller communities or organizations that do not have sufficient (or any) volunteer capacity to collect metrics, thereby allowing these communities or organizations to expand the number and complexity of metrics collected. This dedicated capacity would in the form of a a person who is interested in collecting and analyzing this information, and dedicated to building the skills needed to use the array of tools available to collect metrics, including but not limited to Global Metrics. This person would not necessarily need to be at WMF, but could be elsewhere in the community as well. Having such dedicated capacity would allow this person to collect not just the basic information - such as Global Metrics - but also perform more complex analysis, e.g. comparing the editing activity of an editor during an event compared with their baseline editing activity. The role of the "local team" (i.e. the grantee or associated volunteer organizers) would be to provide a set of usernames, article pages, etc. to enable a person to perform the analysis, and also the context around the activity. -- Shouston (WMF) (talk) 19:11, 7 June 2016 (UTC)

Bluerasberry comments

I generally oppose the two initial changes, "Proposal 1" and "Proposal 2", because they focus on changing the outcomes produced by Global Metrics and do not address the problem of collecting Global Metrics. I think this discussion is premature, and as a counter proposal, I suggest directing discussion to the technical process of collecting Global Metrics then deciding which Metrics the WMF would like to purchase by supplying resources to community stakeholders. Here are some general thoughts:

The two main choices are similar.

Both proposal 1 and 2 assume that the desired metrics are participants involved, 1/3/6 month retention, media items tweaked, then an undefined community building metric.

Number one request - WMF should take a clear stand on how long this process should take

I would like WMF support in defining how long the global metrics reporting process should take and how much community investment should be made in generating these reports. In the past, I think that WMF has casually requested these reports without considering the time and emotional investment which goes into making them. I would like the WMF to say exactly how long it should take to make these reports, and then have a notice that "if it takes longer than this amount of time, then something has gone wrong. Please contact WMF staff for personal assistance." It is entirely possible that reporting global metrics can take longer, be more stressful, and cause more community tension than the program being reported, and I have not seen this common situation recognized by the WMF in this proposal. If we have no where else to start, then we might start by saying on grant pages "Please expect that global metrics reporting will take as much time and work as organizing a typical program, so double time and cost expectations." If there is a better way to communicate this then I would like that, and I am not sure how to generalize this, but it is approximately true for most programs.

The community building metric does not seem like a metric

Here and now is the time to propose something if there is going to be an idea discussed. If there is no measurable proposal, then strike this now and add something in the next iteration, which is support proposal 1 and oppose proposal 2. I do not find it productive to support a metric which is undefined or which would be defined latter. The community building part does not even sound like a metric right now - it sounds like a request for a subjective description. I would prefer to keep the global metrics all objective and all numbers.

Global Metrics never had community support

The Global Metrics process was presented by the WMF with little or no community input. The pages on meta were written by WMF staff with almost no community participation and little community discussion. This is strange because there are so many stakeholders in this. Minor points related to WMF funding are usually heavily discussed, and "global metrics" is a major influence on both WMF and external funding. I think that there should be some planning for getting community attention on the global metrics reports on others. I would like to make the global metrics creation and reporting process more intrinsically attractive to the communities who make them, but we are not in that place now and it will take time to get there. Because the global metrics process is not currently attractive, the value of this discussion is diminished because the relevant stakeholders are not giving the usual amount of attention to this that they would in other, similar high impact but more attractive community issues. There is a high expectation of fluency in Western nonprofit management and grantmaking for entering this discussion, and this is not a skill set in which the Wikimedia community has historically sought to invest. The global metrics concept is highly influential in determining who is fit to get WMF recognition as a good leader in the Wikimedia community, and this process is going to elevate certain skill sets and types of people and discourage other types of people from taking community leadership roles. I know this has to happen but I wish it could be slower and more thoughtful. I would prefer to not require professionalism to be heard in the Wikimedia community because so much community culture has come from non-professional channels in the past.

Community desires simplicity in reporting

At this point the primary community desire in the WMF global metrics process would be simplicity in reporting. Considering comparable professional reporting systems, such as those used by communication professionals which manage similar metrics for Facebook or Twitter community engagement projects. For Facebook and Twitter, community participants click once to join projects and a dashboard generates all sorts of metrics on outcomes, hardly with the communication professional being conscious of how the numbers are generated. The established industry of metrics tracking, which includes advertising and communications, should be a model and precedent for how to effectively get a metrics reporter to desire to report metrics in a digital platform.

Reuse outside of WMF is not considered in current proposal

The Global Metrics are the numbers that the WMF needs to look good as an organization. They are not necessarily the numbers that the Wikimedia community needs to do planning, or to make participants proud, or that community members have to present to other community stakeholders like GLAM institutions or school partners. In Global_metrics/Review/Summarized_feedback#Demographics_of_those_surveyed_and_interviewed missing voices include non-WMF funders of Wikimedia projects and people who have not applied for WMF grants but who comment on grant outcomes or Wikimedia community group budgets. Ideally, the process of collecting global metrics would produce a product that could be shared with other stakeholders, like for example, the routine case of reporting a relevant impact to an organizational partner. If anyone drafted a comparison of "what WMF wants", "what GLAM institutions want", and "what Wikimedia community organizers want", that might be insightful because it would demonstrate where there are overlaps and what is the special interest of a certain stakeholder. Right now, this converstation is still dominated by the WMF, and there is an implicit assumption that what WMF wants is the same as what everyone else wants.

Discourage financial incentives for generating subjective reports and lies

When WMF asks for metrics that creates a financial incentive to give the WMF something that they want. In current reporting processes the metrics reporting system creates a conflict of interest for grant recipients to provide WMF with what that organization wants to hear. In the Western world there is more cultural tolerance and experience for this sort of thing but calling this "global metrics" and asking it of everyone has historically had some harsh cultural impacts in the developing world. I think that the global metrics process has been the origin of some brutal community conflicts and I feel that the WMF is not sensitive to this. A good way to minimize the emotional impact of collecting global metrics is to automate as much of the collection process as possible, to remove human decision as much as possible from the reporting process.

The introduction of WMF money into weaker economies which want American dollar makes this metrics reporting process a career-changing influence on many people's lives. Wikimedia projects are supposed to be community fun and I encourage caution in requesting global metrics because some people are already beginning to dedicate careers to generate these metrics, and it is likely that in the future some people's life work will become WMF metrics accounting if this process is not softened. I fear Western cultural encroachment and financial influence in other economies, and I fear the lack of consciousness and recognition that the WMF is doing this. The brand of the community movement does not warn of this and this is not a desired outcome of the global metrics process.

I fail to recognize the importance of defining most global metrics

Wikimedia platforms generate huge amounts of data. If Wikimedia projects ran like a Facebook group, then participants in an event or program would click once to join a project then automatically be tracked with a timestamp until the project is halted. There could be any number of metrics generated in this process - thousands of them. WMF could take what they want and anyone else could take other ones. "Global metrics" as discussed here hardly have relevance at the local level, but have huge value when various types of programs and behaviors can be collectively discussed in the global Wikimedia community. For that reason, I am not sure that it even should be a goal to get local communities to watch their global metrics, because those numbers often will not make sense outside of processing and comparison with other similar groups and programs. Only the WMF will do collective processing of all global metrics from all groups, and if that is the case, the WMF might want to draw from a pool of thousands of possible metrics that the community members report through an automated process (like the education dashboard tool) rather than trying to get community groups to invest time in appreciating 3-5 so called "global metrics". Global metrics could be re-imagined as a reporting process rather than a few numbers, because a report of a list of usernames and a date range tells most of the story for most projects.

Thanks - would talk more. I care about this. Blue Rasberry (talk) 05:53, 24 June 2016 (UTC)

Hi Blue Rasberry - Thanks so much for providing such detailed and comprehensive feedback. Apologies for how long it took me to respond - things were a bit hectic with Wikimania and the recent holiday. You brought up some interesting points that I’ve been thinking over, and while I don’t have a response for all the points you brought up, I wanted to respond to a few and give some additional context where it might help.

In regards to your proposal to focus on addressing the problems of collecting Global Metrics: I completely agree that changing the metrics included in Global Metrics in no way addresses the problems around collecting Global Metrics. This is one of the reasons we are proposing a more holistic solution to address the three major issues we heard from the retrospective, one of which is that collection of this information is difficult because of the tools (or lack thereof). However, resolving the issues around tools will take more time, as we need to understand the specific issues people currently have - whether those are technical issues (e.g. the tool doesn’t do xyz at all), documentation & training issues (e.g. I don’t even know what tool to use or how to use it), etc. However, given there were more issues with Global Metrics than just the tools-related ones, we didn’t want to wait on addressing the others while we worked on the tool issues.

In regards to some of your general thoughts:

• I really like the suggestion on giving parameters on how much time and effort this process should take, as it was really surprising the level of staff time (of Wikimedia organizations) that was going into GM, which doesn’t even include the unaccounted for volunteer time. I don’t have a good baseline for setting these parameters (e.g. what’s too long for a single program vs. what’s too long for an annual plan?), but the smaller grants might be a good place to experiment with giving this kind of guidance, as they have smaller scope and (theoretically) it should take less time to collect this information. However, I will continue to think about how this could be at least piloted.
• Also agree that Proposal 1 and Proposal 2 are similar, if not mostly the same. As a bit of background on where these proposals came from, there were four proposals for changing Global Metrics that we heard during the retrospective for updating Global Metrics, and these were the two which had the most feasible structure (these were the two proposals that weren't pursued). The metrics that are included in each proposal were based on feedback we heard during our retrospective as well, as the areas of Content & Participation were identified as both important and useful, though there were different comments with which exact metrics were the most important / useful to specific task and specific groups (e.g. grantees, grant committees, WMF staff). However, retention came up consistently when we asked “What’s missing?”, which is why we wanted to see if others agreed that retention should be something reported across grants (when relevant to the grant’s goals).
• In regards to community building: I hear you when you say that you would prefer to keep metrics "all objective and all numbers." We included it because one of the most consistent pieces of feedback WMF grants has gotten is that numbers don’t tell the full picture, and in fact lack much of the context that gives them meaning and value. Moreover, Global Metrics today simply focuses on the outputs of activities, rather than longer term outcomes that are more motivating and may be more central to the goals or achievements of the program. Introducing community building (and to some extent retention as well) was a response to this feedback, as a way to capture these other important outcomes. Like you noted, “community building” is really ambiguous and means a lot of different things to different people. Our intention in including it in this proposal was to see (1) if others agreed it was a useful and important thing to capture and (2) whether it is at a place that it can be captured and reported in a systematic way as part of a grant report (vs. though some other, more appropriate channel). I hope this context helps explain why it's a part of the proposal, and if responses are resoundingly negative then we won't included it in the finalized solution.

Lastly, in regards to your points around re-use beyond the Wikimedia movement and financial incentives: One of the things that came to light in the retrospective is that Global Metrics is being used in many more way that anticipated, and many more way that it was ever designed to support. It was created for a very specific use - to capture certain outcomes at a broad level, mainly for the use within the WMF Grants ecosystem - but the information has been used in many ways, with varying degrees of success. As we said in our design principles, these different needs represent (at times) conflicting needs, and I do not think that Global Metrics will ever be sufficient to satisfy this spectrum of use cases; “other” / “local” metrics have and will always be important to all parties - WMF, grantee, and grant committees included - in telling and understanding the full story well, whether that’s to WMF or to an external partner. So one of the main things we have been exploring in this phase was (1) whether and how we can start to look at Global Metrics and these “other” metrics in a more unified sense, asking questions such as “is it important to have a distinction between Global Metrics and “other” metrics?” and (2) if the distinction between Global Metrics and “other” metrics should remain, how WMF can make clear that these “other” metrics are equally important.

I hope this gives some background on why this proposal overall looked the way it did. I am currently going through all the feedback we received - privately and publicly - so I will look for ways that I can integrate the suggestions you made, particularly the improvements to reporting. Thanks again for your feedback on this topic! -- Shouston (WMF) (talk) 18:03, 11 July 2016 (UTC)

Shouston (WMF) Thanks for the thoughtful feedback. From your responses, it seems to me that we have similar ideas about developing the metrics concept.

I would talk more if it is ever useful to do so. Thanks for any progress you make developing the issue. Blue Rasberry (talk) 18:36, 11 July 2016 (UTC)