Research:Metrics/revert rate

Revert rate is a simple proxy for the quality of an editor's work. Assuming that low quality edits tend to be reverted, editors with fewer of their revisions reverted should be doing higher quality work.

Construction

The common/naive approach to measuring revert rate is simply divide the number of an editor's revisions that were reverted by the total number of revisions they performed. I.e. if $R$ is the set of revisions an editor performed and $R_{r}$ is the subset of those revisions that have been reverted, then the revert rate is $|R_{r}|/|R|$.

Due to the counter-intuitive nature of having a measure of quality that approaches zero for high quality and one for low quality, revert rate is often re-construed as "success rate" which measures the proportion of edits that were *not* reverted. I.e. $(|R|-|R_{r}|)/|R|$

Detecting reverts

While there are a few definitions of a "revert" and different methods for identifying the reverts in the revision history of pages, the most commonly used method for detecting reverts uses checksums of revision text to look for the presence of identity reverts. Identity reverts occur when a new revision is added to an article that exactly duplicates a previous revision, thereby removing the changes made by any intervening edits.

See Research:Revert detection for more discussion.

Limitations

• Reverts happen for many reasons; some of which have little to do with the quality of the reverted edit. That makes this measure noisy. Increasing the number of observations (revisions) is assumed to deal with this problem.
• Revisions need time to be seen and reverted. Tools like ClueBot and Huggle minimize this issue.

Usage

• Halfaker et al. used the rate of reverts in an editor's recent history as well as tenure, experience and content persistence to predict the quality of work that editors performed to build a model for predicting reverting behavior in the English Wikipedia.
• Halfaker et al. used the rate of reverts for an editor before and after a revert event to look for learning effects
• Halfaker et al. used the rate of reverts for an editor in the days before and after revert events to identify a learning effect, but were unable to differentiate whether the difference in revert rate reflected a change in the quality of work an editor performs or a change in the boldness of an editor's edits. This potential confound was overcome through the use of a content persistence based metric for productivity.
• Halfaker correlated the rate of reverts with the rate of survival for new editors in the English Wikipedia.
• Halfaker et al. built off of the above work to implicate the rate of reverts as a primary cause in decreasing retention in the English Wikipedia using a regression model. They went on to show evidence that algorithmic tools for performing these reverts exacerbated the effects.