Learning patterns/Effect of page visibility on contribution

A learning pattern forevaluation

Learning Pattern Library

Effect of page visibility on contribution

problemMany wikis use the Flagged Revisions MediaWiki extension, and it is an important question which setting is better: all edits being instantly visible for everybody, or only after the review process (which can sometimes take days or even months).

solutionAn experiment which compares the effect of the different settings on the number of contributors and number of edits, and answers the question of whether the instant ability to edit Wikipedia attracted the vandals.

creator• Samat

created on22:49, 18 January 2021 (UTC)

status:DRAFT

What problem does this solve?

Since its introduction in 2008, many wikis started using the Flagged Revisions MediaWiki extension, which provides a systematic way to patrol the modifications to the articles, and to rate page revisions. Around the same time, the exponential growth of the contributions to the Wikimedia wikis broke, followed by a significant drop in the number of editors and number of edits in the following years. The debate about the reasons started already that time, and some people pointed to Flagged Revisions as one of the factors: when anonymous or newly registered editors modify an article, the modification won't appear in the article until a reviewer (patroller) accepts it. This delay can range from a few minutes to a few years, depending on the topic, the complexity of the modification, the number of active reviewers etc., but typically it is some days or weeks. This time is too long for newcomers: after they check the article several times, they don't understand why their modifications do not appear in the article, or when they do understand the system, they don't understand why their modification weren't accepted (whether they were refused or not). The impression is the same: it doesn't make any sense to contribute, because their modifications don't appear in the articles, they are not able to change articles, Wikipedia is not editable. With this impression, the motivation has been lost, and it is very likely that they won't come back and try again for a long time (if ever).

What is the solution?

The Flagged Revisions extension was enabled on the Hungarian Wikipedia in 2008. The effect of that change had been a perennial discussion topic. After a lot of debates, many theories and hypotheses, the community decided in 2015 to start an experiment where all visitor can see the latest version of the articles. Unfortunately, there was no developer capacity to implement the community decision, therefore the decision was realized only in April 2018. The results of the experiment was analyzed one years later, during summer 2019 in the frame of the Hungarian editor retention program. The evaluation started in June based on the user activity data available on stats.wikimedia.org/v2, but it had to be restarted in July because of a methodological change in the data set. The statistical evaluation is complimented with an analysis of the vandalism ratio based on ORES data (for which the model had to be retrained first, and a dataset created).

The results of the analysis can be read on the Hungarian Wikipedia (in Hungarian), a summary is available in English, and the main conclusions were presented on the Wikimedia CEE Meeting 2019.

The main conclusions (based on the 12 months of the experiment):

1. The number of anonymous edits increased by 30-40%; the number of anonymous editors increased by ~100%.
2. There was a slight increase in the number of edits by logged-in editors; the more active editors we see, the less positive the effect is.
3. There was no practical increase in the registration rates.
4. The ratio of bad faith or damaging edits inside the anonymous edits grew minimally (2-3%), though the absolute number of these edits increased significantly (proportional to the increasing number of anonymous edits).
5. The good faith and useful (productive) anonymous edits increased by ~30%.
6. Retaining the increasing number of anonymous editors and transforming them into active logged-in editors were not effective and very successful.

Finally in November 2020, the community decided to change back to the original configuration setting which hides the new unpatrolled contributions from readers. Based on the decision, the configuration change happened in December 2020, therefore the whole experiment was 32 months (2.67 years) long at the end.

When to use

If you are interested what are the positive and negative consequences of changing the page visibility settings of the Flagged Revisions.

See also

• Flagged Revisions extension on MediaWiki
• English summary of the analysis
• Presentation slides on the Wikimedia CEE Meeting 2019
• Requests for comment/Flagged revisions should display latest versions

Related patterns

• Evaluation of data affected by seasonal or calendar effects