Research:New editor support strategies/Wikimania session notes

Copied from: https://etherpad.wikimedia.org/p/Wikimania2016-discussion6d

Topic

Edit review & curation tools: Creating a supportive process for new users

https://wikimania2016.wikimedia.org/wiki/Discussions/Quality_tools

https://www.mediawiki.org/wiki/Edit_Review_Improvements

Introduction

The Discussion Room is a space for open and facilitated discussions at Wikimania. Participation of the audience in the session is critical, because there are no speakers, and there is no expert panel!

3 rules are the basis: Focus on YOU. We are interested in discussing and triggering individual action, things people can personally do and change to improve our Wikimedia projects and movement. We trust the discussion can be much more interesting if we do not focus on what others should do ("the others", Wikimedia chapters and Wikimedia Foundation). Be constructive and polite. Disagreements animate discussions and they can allow us to unfold all issues related to a topic. Let's avoid personal attacks, let's consider that we have different backgrounds and let's aim at making everybody comfortable in sharing their legitimate point of view. Be short and on topic. Let's create space for everyone to express his/her opinion.

Starting questions

Who uses edit review & curation tools
These tools are essential to detecting vandalism and bad quality edits (agree/disagree)
The use of edit review and curation tools are discouraging for new users (agree/disagree)

Introduction

5 minutes introduction of that new concept

Objectives

Identify the most important bottle necks in edit review & curation tools, and the approach to address them.
What is your reaction to the project?
What do you think we should do to make it successful.
How can we bring reviewers to this new activity?
What would make reviewers most effective in the job of supporting newcomers during edit review?
How can we make the process rewarding for reviewers, so that they stay involved?
Should we add a feature when user is ready to save?
What are the particular needs, and most promising features, across our communities?
What about mobile?

Attendees - wikis, and usual tools used

David Richfield (facilitator) - New page curation. Popups Twinkle enwiki
Galio (facilitator) eswiki manual / oldschool / watchlist.
Tar Lócesilion - plwiki - special pages e.g recent changes / pending changes (flag review). Watchlist. (favourite wiki to receive notifs - plwiki)
user:Trizek (WMF) - Collaboration team - frwiki
User:Quiddity (WMF) - Collaboration team - enwiki / meta / outreach / mediawiki. Watchlist, some Twinkle.
User:JMatazzoni (WMF) - Collaboration team - Prod mgr of collab team. Notification systems (e.g. cross-wiki notification)
User:Mattflaschen-WMF - Collaboration team - Software eng. in collab team. Also enwiki / commons / wikidata / tech . Twinkle mostly. Watchlist.
Gabriel Wicke - interested in newbie experience. People feel rejected when reverted, don't always realize why they were reverted. Wants to improve the funnel.
User:Geraki - elwiki . Looking for tools.
User:Legoktm - enwiki etc. Twinkle and some custom script
User:Isarra - Twinkle / huggle previously, now not really active. Does work on skins.
user:Lantus (WMCH)
User:NKohli (WMF)
James F. — User:Jdforrester (WMF)
your name here. Log in to etherpad!!!!!!!!!!!!

Notes

- PLEASE FEEL FREE COMPLETING THE NOTES AND ADDING YOUR NAMES IF I MISS THEM **

Benoit - Introduces new Edit Review Improvements (https://www.mediawiki.org/wiki/Edit_Review_Improvements (ERI), aimed at identifying users who make good faith edits

Introduction of everyone

Joe's introduction
- The project is in a very early phase.
- The "Edit Review Improvements" name is catchy but not its final product name.
- The tools are important but can drive away new users.
- This is extremely prominent when semi-automated and high-speed tools like Huggle are used. Important to keep new users.
- Don't want to rewrite the edit review tools: too many; good community support; generalisation is hard (different languages and processes, for example).

move upstream to where the editors find their work.

- Want to separate struggling new good-faith editors from problem editors, and support them.
- First goal is to find automated ways to distinguish these users. Rely on ORES (https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service): tool that can analyse edits and make predictions about whether an edit will be reverted, will be damaging, and whether it's in good faith. Will present the data to users: we're not sure where this should be presented - filter on RC? Page triage? Machine-readable additions to the feeds used by tools like Huggle? Maybe intervene while editor is still working (before save, for example).
- Example: I'm a vandalism patroller using Huggle - I want to know most damaging edits had been removed pretty sure we can do better job prioritizing damaging. So, make their work more efficient in addition to getting good faith new users out of the way.
- Example: I'm interested in welcoming/supporting new users who might be making good-faith edits which are liable to be reverted.
- A stream of teachable moments.
- The program enables WMF to work at the level it feels more effective, developing a platform. The platform has the potential to establish Edit Review as a new "venue" to help new users, but the platform alone won't ensure that such activity takes place.

Benoit
- Asking for reactions to the project and best ways to focus it

Reactions:
- David - As someone who spends a lot of time in curation (PageCuration, Watchlist and AFD) on enwiki, he would welcome any tool that reduces noise in "vandal fighting" and allows a more "human" dealing with new editors. By noise he refers to having to review edits that don't involve the kind of edits he would like to review –edits that can be reviewed automatically or that are not problematic.
- Gabriel - Like to have a platform interface, a good API. Likes the idea to have a feed. Maybe a gadget that makes it possible to integrate it with other tools.
- Matt - Important to have a good way to identify damaging edits form good-faith editors. Don't want to just click the red button. If we don't have a better way of handling bad edits from good editors, the current users will stick with their old ways.
- Benoit - Ask for any ideas about the best way to integrate this tool into the existing interface. How to make new users "get it"
- Isarra - Twinkle is used when you're already on a page, but this might get you to that page.
- Legoktm - Need to be able to track users and their problematic edits. Perhaps they originally wrote an article that was deleted, and then continue on writing good content, it could help us understand their positive progression

Track improvement in user's skills. Maybe creating a score for a user?

- David - Use navpopups and other tools to do a rough mental score of a user's history of editing.
- Snuggle has a good feature to summarise user history.
- Benoit: Who supports the feature to get a user's "score" / "trend"
- Legoktm - Not necessarily a score, but tool(s) to see a general trend and patterns.
- JForrester - Amins might not like it if their score goes down as they shout at people ;-]
- Benoit - How can we make supporting and reviewing new users important? How can we make that a cool thing to do?
- Isarra - Do it like the Teahouse? build a subcommunity
- David - good point to mention teahouse, would like to know more about what research and insights they've reached, and what they found easy and challenging.
- Matt - Teahouse actually had impact on retention. It's an example of how technological improvements and better social interactions can together have an impact, whichi is a lesson we can take here. Can have good machine learning, but good social cues Instead of leaving Template:Uw-test3, have a personal interaction.
- Benoit - Agrees with Matt in that interactions are essential for editor retention. More messages that redirect to that kind of place (friendly places)
- Isarra - most people want to interact and help new users, but can't find the right ones to reach out to: just leave a message and hope.
- Teahouse is by invite. Teahouse hosts say it would be great because it would make it easier for people who want to help.
- Joe - we've talked to some teahouse hosts, and they think this would be good lighter step than committing to being a "teahouse host".
- Non-English: do you have a teahouse on your wiki? (Different from a general Helpdesk.) Galio - Not really. In eswiki we had an active mentoring program. Problem is not the lack of helpers, but patrolling tools often provide counterproductive effects. Maybe need to make it clear that automated messages are automated, not personal.
- Gabriel - About 45% of new articles are deleted. This is bad for new users, especially when people put effort into them.
- Matt - Editors often click save between edits, so you need to predict when they'll be done. Check whether it's so bad that it has to go immediately, or whether it's OK to leave it for a bit. Data says that about 7 minutes is the timeframe of an edit series, while patrollers tagging an edit takes about 2 minutes. Frustrating if you're busy with that. When you're "vandal fighting" you're fighting against Google™. There is a difference between patrolling edits and new articles. For a new article you only have three choices. You can revert, do a follow up edit, leave in place (then hopefully communicate). May be worth handling new articles differently, since there you can't revert (just delete or leave with optional tag).
- Benoit - Would it be useful to have a button to review edits from a new user?
- Joe - We need to understand that the IA won't necessarily give a clear explanation for exactly Why it thinks an edit should be reverted. We might be able to provide editors with a box to check for "I'd like some help"?
- Gabriel - Visual editor save dialog seems like an obvious place for this. If it's made clear that it's a "best guess by a poor machine", they will be less sensitive.

Matt: To be useful, it is important to let the user know why (otherwise, it may be even more frustrating), but ORES can't say that. Maybe "very few refs" "no paragraphs" "possible copyright violation" etc. but that will have to be a different, manually-created system.

- Benoit - Might be different from wiki to wiki. Some wikis might have a stricter requirement for citations. Will have to deal with that complexity.
- David - Would the tool be tailored for each wiki? What works at one language doesn't necessarily work at another language edition of Wikipedia.

How to bring reviewers to this?

Will this help new users and experienced users?
- Isarra: we need to try!
What should we avoid?
- Work with the community, so that they don't do stupid things.
- One community misused AbuseFilter to prevent IPs from editing. This might be a burden on the stewards etc.
- People may use the tool in a way that is not the one it was conceived for (i.e. using the tool in a harmful way that affects new editors)
- David - Recalls backlash with the VE in enwiki. Two reasons: misunderstanding of/in the way it was implemented, and misunderstanding of the experience of new users. Experienced Wikipedians didn't realize what the tool meant for new users.
Making it clear that this information can free up time and energy for human interaction, not replacing editors, will allow you to have fun. If you like reverting vandals, this will also help :-]