Research talk:Post-edit feedback

Delivering post-editing feedback

Possible tools for delivering post-editing feedback include:

• Heavily modified CentralNotice banners, using an API call to determine editor attributes and delivery to every editor who meets the qualification.
• Talk pages, especially via bot
• Email
• Another method for delivering arbitrary JavaScript to editors (prospective)

I am moving these ideas to the talk page as the scope of this experiment is focused on instantaneous feedback displayed upon successful completion of an edit while all these methods are about delivering asynchronous feedback --DarTar (talk) 23:21, 8 May 2012 (UTC)

Idea

Noting this so we don't forget.... both the confirmation and gratitude messages could be used as hooks for MoodBar, where we ask people to rate their experience editing afterwards. Steven Walling (WMF) • talk 01:08, 8 June 2012 (UTC)

WikiProject invitation

Mockup of post-edit WikiProject invitation

Post-edit messages could be used for inviting users to join WikiProjects, ideally in combination with a simplified sign-up process (which shouldn't be too hard to do in a hackish manner if the WikiProject page follows a predictable format). This would likely not have huge impact without overall improvement to how WikiProjects work, but it would be very interesting to see what kinds of sign-up rates we can get.--Eloquence (talk) 01:55, 13 June 2012 (UTC)

Yes, task recommendations or invitations of this kind is definitely a followup item. The challenge in this case is discovery of what WikiProject an article is tagged with in sufficient time to deliver a notification post-edit. That information is in a fairly predictable format, but since it is encoded in wikitext via categories on the talk page, it may require the kind of processing that isn't ideal for such a real time notification, so another method of delivery may be more appropriate to test. Steven Walling (WMF) • talk 17:52, 13 June 2012 (UTC)

It should be a fairly quick API call, but it might be useful to pre-load it while the user is still editing the page.--Eloquence (talk) 20:49, 13 June 2012 (UTC)

Best idea, yet!!

Hi :-) I think that giving post-edit feedback such as described in the research documentation is the best idea that I've heard so far. I'm really interested in hearing the results. FloNight (talk) 22:52, 15 June 2012 (UTC)

Glad it's something that sounds appealing! The feedback is much appreciated. :) We are shooting to start the first phase of this test (the confirmation message) next Thursday, if there are no delays. Steven Walling (WMF) • talk 00:37, 16 June 2012 (UTC)

I started using a w:en:Fitbit last week. (The company is located in SF). One of the aspects of it that works for me is the feedback it gives. Be interesting to see if us being more interactive will reach out to some people and draw them in more. I'll be eagerly following the outcome of the phases of experimentation. FloNight (talk) 18:19, 16 June 2012 (UTC)

I saw one of these pop up today - "your edit has been saved" - and thought it might have been you guys... Nice one. Wittylama (talk) 02:45, 22 October 2012 (UTC)

User quality heuristics

The following are criteria for post-hoc filtering to ensure that our analysis takes into account only new good-faith editors:

• no blocks
• rejection-to-successful-edit ratio of no less than .80, where rejection is defined as either reverted or revision deleted
• no test accounts of team members (WMF accounts)
• no legitimate alternate accounts (will be in Category:Alternative Wikipedia accounts)

Additional factors to consider/filter for:

• users with no mainspace edits (e.g., all userpage, sandbox, etc.). This is not in itself a measure of low quality, but is not a sign of a user progressing into valuable contributions to the encyclopedia.
• no suspected sockpuppet tags on userpage. If a user has this without actually being blocked, we should assume good faith, but it's not promising
• the presence of warnings, especially level two and three standard warnings, on user talk page may be appropriate as a quality metric related to the rate at which someone is rejected. Theoretically, a user might have 9/10 of their edits stick, but do something in the minority of poor edits which is egregiously bad. However, the likelihood that editors would receive no corrections early on is pretty slim, and particularly bad users are blocked anyway.

Could you let us disable it?

I've no problem with the general idea, but perhaps a checkbox could be put into the user preferences to let people disable it if they don't like such messages? I find it a little annoying and unnecessary, personally, but as an experienced and heavy editor I'm clearly not the target audience. I've managed to make them go away by editing my user CSS file, but this feels like kind of a hack. Thanks. --69.244.72.161 14:53, 22 October 2012 (UTC)

Hi anon. We have chosen not to make this a preference so far, for the following reasons:

• No other site which includes these kinds of messages allows for a preference to hide it (Gmail and other Google applications, Twitter, Dropbox, etc.). This is because consistency is a key part of a confirmation message like this.
• Like these other sites, this confirmation is designed specifically to reduce annoyance and, after you've seen it for a while, to become something that your brain ignores as part of the normal functioning of the site. For now, I realize it probably seems very new if you're at all used to editing. But it only appears for two seconds, and I bet if you give it time, you won't see it after a while.
• Adding a preference would add to the bloat of checkboxes for individual features. The community and the WMF are actively trying to remove preferences that aren't used by many people, and we don't want to add a preference by default for every new feature.
• So far I have only seen a relatively small group of people either asking to remove it or actively hiding it. For instance, only 29 editors in English and 16 in German have chosen to add the CSS snippet. For power users like yourself, personal CSS is specifically designed to enable these kind of tweaks. When considering other options, I don't think it's fair to add yet another preference that would be visible to all the tens of thousands of other editors, but only used by a handful.

We said effectively the same stuff on the English and German Village Pumps, so apologies if that was repetitious in any way. Steven Walling (WMF) • talk 19:27, 23 October 2012 (UTC)

Edit metadata

Of possible interest to you: Grants:IdeaLab/Edit metadata. Thanks!
--Gryllida 10:47, 9 December 2013 (UTC)

PEF-2 Results

Why haven't the results of PEF-2 been published here? The experiment is long time ago. -- MichaelSchoenitzer (talk) 10:34, 19 October 2014 (UTC)

Hi MichaelSchoenitzer, you find the PEF-2 results here and here,

• "The results from the test above showed no statistically significant difference in the productivity or quality of editors who received the feedback message about editing milestones, compared to a control group. This was somewhat surprising given that experiments with wiki contributors outside Wikipedia concluded that historical feedback was one of the strongest motivators. Considering those prior results, the failure of this experiment may have been due to a variety of reasons, including that unlike the previous confirmation, the message appeared only intermittently and was easier to miss entirely. In any case, we are not currently recommending that a feature to support this implementation of the feedback message be built."

Really interesting too is Research:Editor milestones, Conclusions:

• "Being awarded a barnstar for making 1,000 edit to the article namespace does not appear to have a significant effect on a English Wikipedians' short-term or long-term editing activity. More work may be necessary to determine whether a different barnstar metric (e.g., more or less than 1,000 edits, a certain number of edits in a successive editing session, etc.) or a different awarding treatment (e.g., community-awarded barnstars, rather than staff-awarded) produces an effect."

And, Research:Newsletter/2012/April#Recognition may sustain user participation:

• "Professor Arnout van de Rijt and graduate student Michael Restivo wanted to test the hypothesis according to which receiving recognition for one's work in an informal peer-based environment such as Wikipedia has a positive effect on productivity. To test their hypothesis, they determined the top 1% most productive English Wikipedia users among the currently active editors who had yet to receive their first barnstar. From that group they took a random sample of 200 users. Then they randomly split the sample into an experimental group and a control group, each consisting of 100 users. They awarded a barnstar to each user in the experimental group; the users in the control group were not given a barnstar. The researchers found their hypothesis confirmed: the productivity of the users in the experimental group was significantly higher than that of the control group. What really took the researchers by surprise was how long-lasting the effect was. They followed the two groups for 90 days, observing that the increase in contribution level for the group of barnstar recipients persisted, almost unabated, for the full observation period."

And, Editor Survey 2011/Wikipedia Community#4.3 Positive reinforcement works:

• "A majority of editors pointed out that positive interactions – like receiving compliments and Barnstars from fellow editors – and positive experiences – like an article making it to the front page – made them more likely to edit Wikipedia. 78% of those editors who received compliments from other Wikipedians said that such interaction increased their likelihood to edit. Similarly, other positive interactions that increased the likelihood of editing include: having an article selected as a feature article (76%); having an article make the front page (72%); having a self-provided picture used in an article (71%); receiving a Barnstar.(70%)."

So, what to make of all this? Automatic feedback about milestones may not word. Personal awards by other editors may work. Awards to editors with 1000+ edits may have almost no impact on their editing volume. There may be a lot more studies and research than i recall, User:Halfak (WMF), any hints to what i forgot? ;-)) --Atlasowa (talk) 16:45, 19 October 2014 (UTC)

It looks like this study was mostly done by Renklauf, so I'm not intimately familiar with the details, the quote you pulled seems like an apt. summary. As far as the effects of feedback on edit activity in general -- there a ton of studies. Here are a couple more that came to mind right away:

• [1] -- Newcomers and experienced editors respond to feedback differently. Specifically, negative textual feedback seems to increase newcomers focus on the task whereas positive feedback appears to increase newcomers overall engagement.
• [2] -- Feedback in the form of reverts reduce engagement, but lead to improvements in quality that are large to make up for lost productivity. Newcomers are especially demotivated by reverts.

There's a lot of stuff here and it would take me a little while to build up a proper lit. review on the subject. Right now, I better get back to today's work. --Halfak (WMF) (talk) 19:56, 20 October 2014 (UTC)

Great stuff. I knew your paper [3] and just read the other one. Fascinating, thanks! An overview would be great, of course. I'm worried that the results of all studies, surveys etc. do not really impact how WMF builds stuff, i.e. automatically sending revert notifications without the edit comment that explains why. Or, not caring to build automatic, targeted, positive feedback notifications [4]. --Atlasowa (talk) 13:49, 21 October 2014 (UTC)