Research:Article feedback/Net value

After an extensive round of testing and development, version 5 of the article feedback tool is set to be deployed across all articles in Wikipedia. However, an important question remains: Would this be beneficial? While there have been studies of the volume and quality of feedback, there's only been a partial examination of the associated costs of the newcomers who enter via the call to action. In order better understand the effect that a full deployment of AFTv5 would have on the English Wikipedia, I'd like to measure the net value that the system brought to the community during its limited deployment over the last few months.

How can we measure net value?

The common way to measure net value is to measure it's gross value and subtract any relevant costs. Such a formal equation for AFTv5 would look like

${\mathit {net\_value}}(\mathrm {AFTv5} )={\mathit {value}}(\mathrm {AFTv5} )-{\mathit {cost}}(\mathrm {AFTv5} )$.

There are two places in which the article feedback system is intended to bring value to Wikipedia: (1) by bringing in feedback from readers that is useful to editors and (2) by convincing more readers to make edits. To add to my formal statement,

${\mathit {value}}(\mathrm {AFTv5} )={\mathit {value}}(\mathrm {feeback} )+{\mathit {value}}(\mathrm {newcomers} )$.

As with most open contribution platforms, any value comes at a cost. To compliment to the places in which article feedback brings value, there are two costs that must be factored: (1) the cost of moderating feedback of current Wikipedians and (2) the cost of deleting/reverting/banning those new editors that are undesirable (bad-faith). Completing my formal statement,

${\mathit {cost}}(\mathrm {AFTv5} )={\mathit {cost}}(\mathrm {moderation} )+{\mathit {cost}}(\mathrm {undesirable\ newcomers} )$.

I'd like to be able to concretely measure the value and cost of AFTv5 in such a way that I could simply evaluate the equation defined above to determine whether the net value of AFTv5. A positive value would mean that AFTv5 is providing a net benefit to Wikipedia/Wikipedians and a negative value would mean that AFTv5 is providing a net cost.

Sadly, I do not expect my work to be so simple. While this calculation is commonly trivial with things that can be assigned a common unit of value (e.g. money), this evaluation becomes non-trivial when measuring more nebulous things like the utility of a user interface component to a complex, community constructed information resource like Wikipedia. For example, it is difficult to compare ${\mathit {value}}(\mathrm {newcomers} )$ with ${\mathit {cost}}(\mathrm {undesirable\ newcomers} )$ since desirable new editors bring much more to Wikipedia than what is easily measurable (edits to articles). For example, some of them may take up the call to fight vandalism themselves -- potentially negating the cost of some undesirables. Similarly, comparing ${\mathit {value}}(\mathrm {feedback} )$ with ${\mathit {cost}}(\mathrm {moderation} )$ is difficult since a particularly useful feedback submission may result in the detection of an important issue that would have gone unfixed.

Despite these unclear comparisons, the more data we fill into this equation, the more object and complete our understanding of AFTv5 will be. I hope that, by measuring these aspects of AFTv5's value and cost, I'll provide a better understanding of what the presence of AFTv5 should mean to the editing community and frame the discussion of its full deployment around the hard facts of data.

Far better an approximate answer to the right question, which is often vague, than the exact answer to the wrong question, which can always be made more precise. -John Turkey, Annals of Mathematical Statistics