Research:Standard metrics

This page is about a 2013/14 project for metrics standardization. For overall edit statistics across Wikimedia projects, see Statistics.
Metrics standardization, Wikimedia Research & Data Showcase, March 2014

Researchers, analysts, and product managers use a wide variety of metrics (from "monthly active editors" to "user's giving proportion in the dictator game"[1]) track and evaluate phenomena related to the Wikimedia projects. This page collects metrics which are suitable for wide use, which will make it faster to develop new research projects and easier to compare existing ones.

These metrics are mostly quantitive, but qualitive metrics are worth standardizing too. For example, researchers sometimes survey Wikimedia users and contributors about their subjective satisfaction with software. It would be sensible to devise a standard, well-considered way of asking such questions.

A high-level overview of the design of Rolling Monthly Active Editors, June 2014

Background edit

 
Analysis example. An example of sensitivity analysis for the new editor definition: monthly count of newly registered users on the German Wikipedia performing at least one edit in their first day/week in the article namespace or across all namespaces.

Overview edit

One way to group standard metrics is into 5 categories:

New users
these metrics provide indicators on the acquisition, activation and productivity of users joining Wikipedia or other Wikimedia projects for the first time.
Community
these metrics measure the overall composition, growth and volume of activity of existing communities, including both human and automated activity by bots.
Content
this category of metrics measures the growth and dynamics of content creation, including edits, new articles, uploads.
Curation
these metrics measure the quantity and quality of curation and moderation activities, such as reverts, deletions, blocks.
Traffic
these metrics measure traffic and readership of Wikimedia projects.

Evaluation edit

Each metric and user class definition comes with supportive analysis whose goal is to understand how sensitive its definition is to specific parameter choices and whether the metric captures the same phenomenon in different projects. We strive to run sensitivity analysis across projects in different languages and of varying levels of maturity, but we welcome feedback to improve these definitions and to identify edge cases, particularly for smaller projects or projects with uncommon policies, where the proposed definition may not accurately capture the quantity it attempts to represent.

We also expect the use of these metrics in the first iterations of the design of Editor Engagement Vital Signs to reveal anomalies and interesting facts that are hard to anticipate until series for each metric are automatically generated for each Wikimedia project.

New users edit

Newly registered user edit

A   is a previously unregistered user creating a username for the first time on a Wikimedia project.

Depends on
none
Used in
New editor

New editor edit

A   is a newly registered user completing   edits to pages in any namespace of a Wikimedia project within   days since registration ( ).

 
New editor
Standardized definition
  •   = 1 edit
  •   = 1 day
Depends on
Newly registered user
Used in
Productive new editor

Productive new editor edit

A   is a new editor who completes at least   productive edit(s) within   time since registration ( ).

 
Productive new editor
Standardized definition
Depends on
New editor
Used in
none

Surviving new editor edit

A   is a new editor who completes at least   edits within   time since registration ( ) and also completes   edits in the survival period  .

 
Surviving new editor
Standardized definition
  •   = 1 edit
  •   = 1 edit
  •   = 1 day
  •   = 30 days (~ one month)
  •   = 30 days (~ one month)
Depends on
New editor
Used in
none

Community edit

The editor model edit

The editor model is a suite of metrics which include subclasses of and funnel rates for monthly active editors.

Rolling active editor edit

A   is a registered user who completed   edits to pages in any namespace of a Wikimedia project between   and  .

 
Active editor (rolling)
Standardized definition
  •   = 5 edits
  •   = 30 days

Rolling new active editor edit

A   is a newly registered user who both registered and completed   edits to pages in any namespace of a Wikimedia project between   and  .

 
New active editor (rolling)
Standardized definition
  •   = 5 edits
  •   = 30 days
Depends on
Newly registered user
See also
Rolling active editor

Rolling surviving new active editor edit

A   is a newly registered user who both registered and completed   edits between   and   and continued to complete   edits between   and  .

 
Surviving new active editor (rolling)
Standardized definition
  •   = 5 edits
  •   = 30 days
Depends on
Newly registered user
Rolling new active editor
See also
Rolling active editor

Rolling recurring old active editor edit

A   is a user registered before  , completed   edits between   and   and continued to complete   edits between   and  .

 
Recurring old active editor (rolling)
Standardized definition
  •   = 5 edits
  •   = 30 days
See also
Rolling active editor

Rolling re-activated editor edit

A   is a user who completed less than   edits between   and   and completed   edits (but was not a R:newly registered user) between   and  .

 
Reactivated editor (rolling)
Standardized definition
  •   = 5 edits
  •   = 30 days


Other community metrics
edit

The following metrics do not form part of the Editor Model and are computed daily. These metrics will be delivered in stage 3 (2015-Q1)

Daily unique registered editors edit

A   is a user who is not a flagged bot and completed at least   edits on date  .

Standardized definition
  •   = 1 edits

Daily unique anonymous editors edit

A   is an unregistered user who completed at least   edits on date   via the same IP address.

Standardized definition
  •   = 1 edits

Daily unique bot editors edit

A   is a user who is a flagged bot and completed at least   edits on date  .

Standardized definition
  •   = 1 edits

Daily unique page creators edit

A   is a user who completed at least   page creations across all namespaces on date  .

Standardized definition
  •   = 1 page creation

Daily unique media creators edit

A   is a user who completed at least   media creations on date  .

Standardized definition
  •   = 1 media creation

Content edit

these metrics will be delivered in stage 3 (2015-Q1)

Daily edits edit

  is a count of the number of edits saved by any users on date  .

Standardized definition

no parameters

Daily edits by registered users edit

  is a count of the number of edits saved by non-bot-flagged registered users on date  .

Standardized definition

no parameters

Daily edits by anonymous users edit

  is a count of the number of edits saved by anonymous editors on date  .

Standardized definition

no parameters

Daily edits by bot users edit

  is a count of the number of edits by flagged bot users on date  .

Standardized definition

no parameters

Daily pages created edit

  is a count of the number of page creations across all namespaces on date  .

Standardized definition

no parameters

Daily media created edit

  is a count of media creations on date  .

Standardized definition

no parameters

Curation edit

these metrics will be delivered in stage 4 (2015-Q2)

Traffic edit

Page views edit

See Research:Page view.

Unique devices edit

See Research:Unique devices.

Supplementary resources edit

Notes edit