Research and Decision Science/Data glossary/Essential metrics

Background

Metrics and measurements for decision making have been in demand in the Wikimedia Foundation and the Movement for many years. Over the past years hundreds of metrics were defined by different Wikimedia Foundation teams alone. While some metrics have benefited from (relatively) significant research, engineering, documentation, and communication resources for their definition, maintenance, and raising awareness about their usage, the majority of the metrics remain for the most part unknown to decision makers, inaccessible, unmaintained/under-resourced, or sometimes contradictory/inconsistent.

Wikimedia Foundation has recognized the above challenges for multiple years and committed some resources to tackle them. More recently and in fiscal year 2022-2023 annual plan WMF recognized the need for creating and/or maintaining a small set of core metrics for decision making. That work resurfaced the need for revisiting the existing metrics, standardizing them when relevant, and developing a criteria for deciding whether to dedicate more resources to their maintenance.

As part of fiscal year 2023-2024, Wikimedia Foundation prioritized work to bring some organization in the way we think, dedicate resources, and use metrics. We recognized that while there will always be a need for a small set of metrics (currently referred to as core) to inform important decision making, the organization and the Movement needs more metrics than just a few for decision making. At the same time, with the limited resources, it was evident that we could not commit to standardize, maintain, document and make accessible hundreds of metrics.

Between July and December 2023, we brought together a representative group of experts and stakeholders in WMF as part of a working group to focus on one task: co-develop and propose the criteria for essential metrics with the understanding that the criteria, once accepted, will be used to determine which of the many metrics WMF has developed over the years are to be considered and resourced as essential.

In the rest of this page, you will learn about more details that came out of discussions and deliberations of of the working group which is now captured as part of the Essential Metric definition.

Definitions

This section is currently a draft. More information will be available by April 30, 2024. In the meantime, if you have questions about the section, please reach out to us in the talk page or contact Leila Zia <leila@wikimedia.org>.

The definition of Essential Metrics includes multiple terms that require further specifications for us to be able to use the definition in practice. In this section we expand these terms, the intention behind introducing them, and how one can interpret and use them in practice.

FAQ

We will expand the FAQ section as we hear questions about the definition and implementation of essential metrics.

Q. Why should an essential metric be in "production" and what does that actually mean in practice?
A. There are many (endogenous and exogenous) factors that can affect the computation of measurements for a metric and the definition of a metric itself. As as a result a metric that is used in frequent decision making must be reviewed and updated as relevant. By in "production" we aim to communicate that for a metric to be considered essential by WMF, the organization must dedicated resources to it. At the minimum, a metric in production must have:

• Defined business owners or data stewards who can make decision and answer questions about the metric.
• Planned engineering resourced dedicated to enable standardized data pipelines, data quality checks, monitoring and alerts, service-level-objectives, automation, comprehensive technical documentation, and data data visualization.

Q. Who can develop metrics that can be considered to become essential metrics?
A. Anyone. We want to make sure to leave the space open for others to develop metrics that we can consider to categorize as essential metrics. This will allow us to benefit from the extensive knowledge of those closest to the places where decisions need to be made as well as those with more resources (for example, those in research or academic institutions) to develop new metrics.

Q. Why is it a consideration for a metric to be used frequently in decision making for it to be considered an essential metric?
A. The primary reason is that our resources are limited and we must choose which metrics we dedicate our resources to. As a result, if a metric is used for only a limited number of times for decision making, even when the decisions are of significant importance, we won't need to productionize such a metric or dedicate long term attention and resources to it. As a result, the metric is not considered "essential" with the specific definition we use here.

Q. Can new metrics be defined and considered as essential or is this criteria going to be applied only to already existing metrics?
A. New metrics can be defined and considered as part of the essential metric criteria.

Q. How do you determine who is going to be affected by one or more decisions based on the metric (referring to the "transparency" requirement for essential metrics)?
A. Our vision is that operationalizing transparency for metrics can be done through the development of "Metrics Cards", inspired by Machine Learning Model Cards. If such an implementation is adopted, the proposer of a metric and/or the committee that reviews the proposal will need to call out the audiences/groups that they know may be affected. At the same time, transparently sharing Metrics Cards allows those who may be affected by the decisions to surface potential challenges with making decisions based on specific metrics.

Q. Is a metric that is developed or needs to be developed for annual planning purposes automatically considered to be essential?
A. No. The frequency in decision making is important to consider when making a decision as to call a metric essential or not, which effectively triggers medium to long term resources to be dedicated to the metric. If we expect a metric to be used only for a few quarters or even just a couple of years by a small group of folks for decision making, the metric will need resources but is not necessarily considered essential.

Q. What do you mean when you say WMF's "strategic direction"?
A. At the time of developing the criteria for essential metrics, WMF started developing an updated strategy for the organization. The decision making related to this multi-year organization strategy, once it becomes stable, will require development or maintenance of metrics that can be considered as essential.