Grants talk:IEG/Revision scoring as a service/Renewal

Initial feedback

Hi とある白い猫 and He7d3r,

Thank you for submitting this renewal request. I'm posting some initial feedback below, some of which has already been discussed in a recent meeting with とある白い猫:

• In light of your significant achievements so far in building an AI infrastructure for Wikipedia, it makes great sense that you want to to extend your service into new territories--and support developers in doing the same.
• In your budget, I'm not clear about the total amount you are requesting for your renewal. Is the total request for $30,000? Or are you asking for $2,400 for hardware plus $30,000 for 2 grantee developer stipends, for a total of $32,400?
• I would like you to add a section to your renewal that lays out SMART goals and measures for each of the bulleted areas you'll be focusing on in your Scope. Setting these targets now will help you think about how to plan, track and report your successes at the end of your grant period.
• As noted in the feedback on your final report, I'm glad you are thinking about how to track adoption by tool developers. You've indicated that adoption rate will be one of your measures of success. In line with the previous bullet, please specify how much adoption you are aiming for.
• I encourage you to add a goal to build a sustainability plan, taking a long-term view beyond the six-month duration of this renewal. At the end of this renewal, you should know what's next for this project, and we're happy to partner with you to help figure that out.

Starting now, I'm giving the IEG committee and the broader community 10 days to share their thoughts on this request before we move forward with a final decision on whether to renew this IEG. I'll be notifying the IEG committee and I expect you'll notify relevant areas of the community as well. Thanks again for all your efforts so far!

Cheers, --Marti (WMF) (talk) 23:55, 2 July 2015 (UTC)

Hello Marti,

• We hope to expand and cover more backlogs in the course of the next six months with a focus on improving existing tasks as well as venturing into areas such as Wikipedia 1.0 assessment, Edit type classification. On top of that we are excited about the third party developers of our system such as WMDE, NASA, Carnegie Mellon University, Wikiproject X in no particular order. In order to better serve these third party developers we will also make significant improvements in UI so that creating campaigns can be conducted by anyone.
• We are only requesting $30,000, the $2,400 mentioned would be the hardware cost had we not used wikimedia labs. We are able to cut back on cost by using Wikimedia Labs. It is meant to be an indication of the scale of our project right now.
• We will try to add it this weekend.
• Measuring this may be difficult. For example we will not have direct access to NASA use since it will be on a private wiki. They may be uncomfortable providing statistics.
• We did consider some longer term goals, we will try to substantiate them.

-- とある白い猫 ^chi? 10:03, 11 July 2015 (UTC)

"big ab appropriately fit model"

What does this mean?--Anders Feder (talk) 06:24, 5 July 2015 (UTC)

Hi Anders Feder. That was a typo. Thanks for catching it. I changed it to "build an appropriately fit model". --EpochFail (talk) 15:38, 7 July 2015 (UTC)

feedback and comments from Thepwnco

@とある白い猫 and He7d3r: Hello and thanks for the renewal request, as well as your final report - I just finished reading it and wanted to congratulate you on the success of this project to date! I will also echo Marti (WMF) and say thank you for writing your final report in a way that makes a complicated technical project easy to understand and appreciate.

I think it is a great idea to renew this IEG and agree with many of the endorsements you've received so far. I have a few comments/questions though:

• My understanding is that the expansion of your project (to model edit type classification, etc) will also depend on handcoding through wiki-labels. Given the challenge of attracting volunteers to this type of work, have you considered different/new strategies for outreach and communicating the value of labelling? I assume the funds set aside for community organizing is part of this..?
• In your final report you write that the project will likely never be done (!) - That being said, I'd like to see more details on sustaining the project once the service is extended in the 3 ways outlined in this request and after 6 months of additional funding.

thanks. -Thepwnco (talk) 00:25, 13 July 2015 (UTC)

One thing I'm doing, hoping to attract more people to help in the labeling process, is to propose patches for existing anti-vandalism tools (see e.g. this list: phab:T90034#1305842), or to create new ones (e.g. ScoredRevisions), to incorporate the usage of scores produced by our project in their daily actions. Considering that right now we only have a model for "reverted" edits, which is only an imprecise approximation to the model for "vandalism" we really want, when the users find revisions whose scores do not match what they would expect, we can point then to the way to improve the predictions, which is the manual labeling of revisions specifically for "vandalism". Helder 21:48, 13 July 2015 (UTC)

Hello Thepwnco, when we say the project will never be done, we mean that we have a longer plan in mind. We want this system to exist as a service forever and we hope to add new capabilities to it based on community demand. -- とある白い猫 ^chi? 06:46, 24 July 2015 (UTC)

Feedback and comments from Steven Zhang

Apologies for the last minute post - a few questions and comments I have.

The aims of this renewal appear to be reasonable however I would like the goals (documented in the scope) to be a little bit better defined, and the measures of success to more comprehensive. I also wonder how sustainable this project is, so am wondering if this has been thought of and a plan can be put together.

The budget seems high to me but understandable given the cost of people with these sorts of skills. Steven Zhang (talk) 00:58, 13 July 2015 (UTC)

Hello Steven Zhang, I would differ you to Research talk:Revision scoring as a service where you can see the measures we have taken to make the project sustainable and horizontally scaleable. We are creating a system that can handle tens of bots demanding scores of the recent changes feed at once. -- とある白い猫 ^chi? 04:35, 24 July 2015 (UTC)

Approved for another 6 months

Hi とある白い猫 and He7d3r,

Congratulations again on your successes so far! Based on what you've created to-date, I'm approving this 6-month renewal to provide continued support to expand your work. This approval is dependent on the grantee updating the project goals to address a plan for sustainability, and tightening up your targets. We will work with you on this update, and we'll continue our offline conversations about how to support long-term needs of this project. We look forward to partnering with you! Our grants administrator, Jtud (WMF) will be in touch soon to setup your renewed agreement for $30,000 so you can continue this IEG for another 6 months. Best wishes, --Marti (WMF) (talk) 20:27, 13 July 2015 (UTC)

Hi とある白い猫 and aetilley, thanks for adding a sustainability plan to your project scope and for tightening up your targets, as requested above. One last request: your measures of success should echo your scope of work. If you could make sure that the sustainability plan is reflected not only in your scope, but also in your measures, then the requested changes will be complete (the existence of a sustainability plan will be your measure of success). Let me know if you have any questions. Warmly, --Marti (WMF) (talk) 23:05, 27 July 2015 (UTC)