Talk:Community Tech/Grant metrics tool

Questions

Questions to anyone who reviews the page.

General questions

• Do you understand how to use the tool? Would it be easy for you to use (without instructions, tutorials, etc.)?
• Would you use this tool for your programs or events?
  • If so, what makes this tool useful to you? Can you give us an example (or a few examples!) of a program or event for which you would use this tool?
  • If not, what would make the tool more useful to you?

Small questions that don't have an answer yet

Please help us answer these questions!

• When an event/project is over (and no new data is anticipated), do we need an option of "closing" the project so the data is un-editable / un-changeable?
• Should participants be able to sign up directly to the program or event (like in the Dashboard)? How can that be done while respecting the request for privacy?
• Support CSV uploads of usernames (like in Wikimetrics)?
• Include the ability to duplicate an event, so that you don't have to re-enter usernames, etc.?
• When entering a Wikimedia project, should there be a dropdown window that lists all language projects?
• This tool will have the ability to export a CSV of all edits that are counted in the metrics. Would this be useful to you? It will look like the Recent Changes page on wiki.
• The interface will be translated through translate wiki. Who could we contact to help translate into your language?
• Add an "event type" into the program/event description? (e.g. so that all edit-a-thons, writing contents, photo contests, can all be categorized)?

Questions/suggestion from Ido

First of all, cool idea and a much needed tool! (Also, "Computers are better than people at counting things, and they enjoy it more" is a catchy motto!).

• Is there a way to have participants input their usernames/intended content (e.g. by inserting a template to their new/existing user talk pages?) - that way the burden of actually collecting is taken away from event organizers? I think this is a crucial point to at least consider, knowing full well that this might be impossible.

• (On a similar note, what does "(This could be generated automatically from a hashtag)" mean? it's remarked about username lists)

• Data - I think an event type field (a drop-down with options such as editing meetup, hackatahon, etc.) can be optional, but may serve as a strong data-point so that in the future grantees (and maybe WMF employees? depending on privacy stuff) can compare similar events between different grantees or even grantees could measure improvement in running events and/or retaining editors.

Alleycat80 (talk) 15:19, 1 April 2017 (UTC)

@Alleycat80: Hi Ido - I know we talked about some of these in person, but just following up to say that we've clarified stuff and added your questions about participant sign-up and event type into the open questions for this next round of feedback that we're about to begin. -- Shouston (WMF) (talk) 19:07, 13 July 2017 (UTC)

How much did this cost?

I'd like to know how much money the Wikimedia Foundation spent so far on developing this piece of software. Thanks, --Frank Schulenburg (talk) 18:41, 21 June 2017 (UTC)

@Frank Schulenburg: Hi Frank - There has been no development on this piece of software so far. This is still just an idea, with some concrete wireframes (created and iteratively improved by the Comm Tech team) so others can understand how it might work. The intention is to get more community input before any development begins. -- Shouston (WMF) (talk) 21:13, 22 June 2017 (UTC)

Hey Sati, thanks for the quick reply. I really appreciate it! Best, --Frank Schulenburg (talk) 22:49, 22 June 2017 (UTC) 

I was wondering this same thing. This looks so much like the dashboard. The description of the dashboard is not accurate on this page. The data which the dashboard collects could be further processed to generate everything this tool does without further user involvement. Blue Rasberry (talk) 10:44, 25 July 2017 (UTC)

Hey Sati, as there's still some activity here (which seems to indicate that WMF is determined to move on with this project), let me renew my question: how much money did WMF spend so far on this (i.e. your time, everybody else's time). I'm asking in my role as a volunteer Wikipedia editor who's really concerned about spending money on something that looks like re-inventing the wheel. Best, --Frank Schulenburg (talk) 00:17, 23 August 2017 (UTC)

Hi Frank - I don’t know the the total number of hours that’s been spent on this project, as I don’t manage anyone who has worked on it. I can say that I’ve spent time talking with at least 20 grantees, to understand the problems they encounter when collecting metrics and using tools to do so; as a part of this, we discussed this proposed tool, what they thought about it in general, and how they thought it could be improved. I believe taking the time to understand the vast array of problems related to collecting metrics is worthwhile, especially as each grantee has context-specific issues that enable or prevent them from using certain tools.

I also don’t believe that this tool is reinventing the wheel. As I said to Camelia below, this tool is basically an update to Wikimetrics, one of the primary tools that grantees use to collect metrics today, and a tool that has a lot of known problems. What we are doing is throwing out the bad parts of Wikimetrics, keeping the good parts, and giving it a new interface that utilizes some of the UI from the Dashboard since it’s proven easy to use. So we’re not exactly creating something new; we’re building on what already exists and improving it with the new/better information we have from grantees about their context and problems. -- Shouston (WMF) (talk) 22:58, 28 August 2017 (UTC)

There seem to be multiple great misunderstandings here. I hope that WMF staff do not make investment plans in this without an attempt to come to mutual understanding with users of the dashboard. I feel like I am coming to this with some background on wiki community organization and available metrics and I find much of this page and this conversation to be baffling. Thanks. Blue Rasberry (talk) 15:33, 30 August 2017 (UTC)

@Bluerasberry: I'm happy to talk and discuss any misunderstandings around this tool, so let's find a time when we can do that. I'll email you off-wiki to figure this out logistically. I will say though that some of the feedback I've gotten has been from people who are or were users of the dashboard. Obviously, I don't believe that is comprehensive feedback, or represents all dashboard users; I only mentioned that in case you thought I was purposely excluding them. When we talk, I'm also happy to share more specifically about the issues they mentioned encountering when collecting metrics, and some of their feedback (all of which I'll keep anonymous, but of which I can share the main points). -- Shouston (WMF) (talk) 21:34, 30 August 2017 (UTC)

Incorrect statement about P&E Dashboard

Just a heads up: "For example, in Wikimetrics and the Dashboard, all the edits a person makes within the time period is counted toward the program." is actually false; there is a way to do this in the P&E Dashboard. It's not well documented, so I understand the confusion! But if the facilitator changes the page type to "Visiting Scholarship", the Dashboard then *only* collects data on articles specifically added through the "Add articles" functionality, not edits by those users to other articles. Wiki Ed originally built this page type for our Visiting Scholars program, which is why it has a confusing name; in our program, existing Wikipedia editors get access to university resources, so to measure the impact of the program, we wanted to only track the edits they made using those resources, rather than all edits. For each program page, you can choose what page type it is, so if you (for example) have an edit-a-thon with a lot of experienced Wikipedians, you might want to choose that page type; the facilitator then just needs to make sure that any articles edited by anyone as part of the edit-a-thon gets added to the P&E Dashboard. --LiAnna (Wiki Ed) (talk) 19:11, 21 June 2017 (UTC)

@LiAnna (Wiki Ed): Hi LiAnna - Thanks for the new information! I'll update the page to reflect this distinction. -- Shouston (WMF) (talk) 21:13, 22 June 2017 (UTC)

more about dashboard

I want to respond to the section on the dashboard -

This tool will differ from the dashboard in a few ways:

1. This tool can only be used after the program is over, and will only focus on reporting the outputs of the program. In contrast, the Dashboard has the ability to track what people are doing during the event, and can also be used for program management.

I fail to understand this. How is it different whether a reviewer looks at edits during a program or a week later? The data is the same, right? Blue Rasberry (talk) 10:55, 25 July 2017 (UTC)

Yes, the data is the same. The difference we were highlight was about how people use the two tools. Some organizers who use the dashboard want to look at what their participants are doing while the program is still running. This is especially true for those who are running very long programs (e.g. education programs, where the teacher would want to look at what their students are doing during the semester). This functionality of “looking at what people are doing while the program is still running” is included in the dashboard but not in this tool. It would have been clearer to say “the Dashboard has the additional ability to track…”; I will change this to make it clearer. — Shouston (WMF) (talk) 21:07, 26 July 2017 (UTC)

• This tool will measure different metrics. Unlike the Dashboard, this tool will report on new editor retention. Additionally, this tool will report "pages created or improved" by Wikimedia project, instead of reporting it as a single number.

The Dashboard gets the list of participants. Cross-checking for retention is a matter of arithmetic. Is there any reason why that could not be a minor modification to the dashboard? The dashboard reports "new articles" and "articles edited". How are those metrics different from "pages created or improved"?

Agreed, this is a minor point and probably more confusing then helpful. The main difference is that retention is calculated automatically, and that pages created or improved is reported by a Wikimedia project rather than as one lump sum. — Shouston (WMF) (talk) 21:07, 26 July 2017 (UTC)

• It will be easier to correctly attribute a person's edits to a specific program. For example, some organizers use a talk page template to indicate which articles were edited as a part of a program. But then they have to manually enter each article into the Dashboard, to correctly calculate program-only edits. With this tool, only the template would need to be entered (along with usernames, if you have them). This will work for any template or category.

I fail to see how this statement describes anything about the dashboard. The dashboard operates with one click - "user join program". After that it tracks what they do. It is not routine for any user or program coordinator to enter anything manually into the dashboard. After setup, the program coordinator does not return to the dashboard except to collect metrics for reporting. After signup, users never return to the dashboard. In typical programs over a date range, there is no differentiation between "the users' activity" and "the activity which is part of the program".

There are many cases where “the user’s activity” and “the activity which is part of the program” are not exactly the same. For instance, let’s say that someone signs up on the dashboard for a long-term writing contest. Unless that program was setup as a “visiting scholars” program (see Lianna’s comment above your comments), the dashboard would count all contributions that person made within the time period of the contest. But participants usually do others stuff on Wikipedia (or another wiki project) that are not connected to the contest at all. But those “other” contributions are still counted toward the program, and counted in the metrics of the program. I’ve heard from many grantees that this issue is one of the main reasons they wind up doing manually labor to change/fine tune the results from the tools. — Shouston (WMF) (talk) 21:07, 26 July 2017 (UTC)

• It will be a simpler tool with less functionality than the Dashboard that focuses exclusively on tracking the end metric, which might be faster for those running smaller events/programs.

In what way is it simpler? The dashboard requires wiki account creation then clicking a "join button" program by each participant, along with a one-time 2 minute event setup by the coordinator.

I completely agree that its pretty easy to setup and join programs on the dashboard. However, there is more functionality to the dashboard that just what you included; for instance, there are tutorials that are hosted within the dashboard. We were trying to point out that this tool would not include the extra “bells and whistles” that are in the dashboard today. So it was a statement that the tool was simpler because it simply does less things. However, I see your point that the way we said it was misleading, and I’ll update it. — Shouston (WMF) (talk) 21:07, 26 July 2017 (UTC)

• Programs will be private, and accessible only to program organizers rather than the general public.

Could you not add a privacy option for dashboard courses? Also, creating privacy around public information has lots of problems which are not discussed here. Most wiki programs need to be accessible to multiple groups unless there is a safety issue. Wiki is designed for collaboration and prohibiting access to public information can create a lot of problems.

We originally discussed making this tool public, but our legal department recommended that we make it private for legal considerations. However, making it private does not mean that only one person will have access; we have scoped out the ability for organizers to share access to those who need them. We haven’t figured out all the details (e.g. should there be view vs. edit access?), but we all agree that multiple people need access to the information. Since I’m not the best person to talk about the legal stuff, if you want more information I’m happy to loop in someone from Legal, as they are the best people to comment on this.

This tool looks like it was designed by people who were unaware of the dashboard and have not used it. Blue Rasberry (talk) 10:55, 25 July 2017 (UTC)

@Bluerasberry: Hi - I have responded to your comments in line; I hope they clarify some of the confusion. — Shouston (WMF) (talk) 21:07, 26 July 2017 (UTC)

@Shouston (WMF): I think I understand your position a little better. If I can further clarify the Wikimedia community position, then ping me. Thanks for replying. Blue Rasberry (talk) 14:38, 2 August 2017 (UTC)

@Shouston (WMF): I replied to your email. Let's try to talk next week. Thanks. Blue Rasberry (talk) 14:24, 1 September 2017 (UTC)

General feedback about metrics tool

As someone who agrees that measuring shared metrics is currently quite hard, I think that a new tool can be quite a time-saver. In terms of interface, there would need to be a number of different options to be useful:

1. Enter time-range of event and editor names and, based on those two things alone, calculate (1) new editors, (2) items improved/added (so not edits in total!), listed per project and language version and in total (in some events, Commons, WD and WP are all edited together, and sometimes WP gets edited in multiple languages during an event, so don't limit results to just one "home project"), and (3) (new) editors retention.
2. Option to cross-ref that with edits in a certain category, or a hastag summary, for example when doing an online writing contest.
3. If possible, separate files added and files edited in Commons. Files added are sometimes more easily counted with other tools, so it can be valuable to separate those from other edits.

That's it for now. Will add more if anything else comes to mind. --AWossink (talk) 08:44, 27 July 2017 (UTC)

@AWossink: Thank you for your feedback! We have already built in the functionality to do #1 and #2 into the current specifications of this tool. "Content pages" will be either in either the Main or File namespaces, so items on Wikidata would definitely be counted, not edits. \ The only thing in #2 we haven't done yet is support hashtags, as the use of that is really nascent right now; we'll keep it on the list of things for future consideration, especially as adoption of hashtags grows. In terms of #3 I believe that will happen already, as we have a "pages created" and "pages edited" metric; since most grantee work on Commons is within the File namespace, I believe we'll only be counting edits there (instead of the main namespace). So pages created in the file namespace = files added; pages edited in the file namespace = files edited. However, I'll make sure to specify that. Thank you again! -- Shouston (WMF) (talk) 17:22, 4 August 2017 (UTC)

Feedback from Ivana (WMRS)

Hi everyone, here's my feedback. I hope it's not too late for it. First of all, I'll answer to the general and small questions and then give examples of some of the problems we bumped into.

General questions

• Do you understand how to use the tool? Would it be easy for you to use (without instructions, tutorials, etc.)?

After reading the page, it seems that this tool would be quite useful and a time-saver for organizers and project leaders. It looks like it would be easy to use, but I think there should be some kind of tutorial.

@IvanaMadzarevic: Good idea about the tutorial; I've added that to the list of needs. In the meantime, is there a way we could make the tool more intuitive, so the people might not need the tutorial?

• Would you use this tool for your programs or events?

Yes, definitely. We've already used Wikimetrics in the past, so if this tool might be better from Wikimetrics, it would mean less time for gathering metrics for us and more time for program activities.

• If so, what makes this tool useful to you? Can you give us an example (or a few examples!) of a program or event for which you would use this tool?

We would gather metrics easier and more correctly. For example, one of the projects within the Education program is Wikipedia in schools. If I understand correctly, in this tool we would define Wikipedia in schools as a program and specify several different events which represent specific projects in educational institutions (e.g. Wiki student at Faculty of Mathematics in Belgrade or Wiki Gymnasium at Ninth Gymnasium in Belgrade). This is not related just with the Education program. It can be used in GLAM. Program would be Edit-a-thons and every edit-a-thon would be a separated event.

Small questions

• When an event/project is over (and no new data is anticipated), do we need an option of "closing" the project so the data is un-editable / un-changeable?

I think there's no need for closing the projects. There can be events (e.g. edit-a-thons) where some editors participate online and organizers are not familiar with their participation. Later when we discover it, we can change that in this tool and update metrics. This can easily be the case with one of the education program (e.g. when students forget to sign in).

@IvanaMadzarevic: Thank you for answering this question - I haven't gotten much feedback here, but the small consensus is that there is no need to close programs.

• Should participants be able to sign up directly to the program or event (like in the Dashboard)? How can that be done while respecting the request for privacy?

For the small projects this is not necessary, but when we have 200-300 students, it would easier for the organizers. Don't have the answer for the second question.

This is consistent with what I've heard from others, but we all don't have a good answer for the surrounding issues, e.g. privacy, or new editors getting confused when asked to sign up (which I've heard happens from other organizers who considered using the dashboard).

• Support CSV uploads of usernames (like in Wikimetrics)?

So far I've always uploaded usernames by coping, but I guess some use tables.

• Include the ability to duplicate an event, so that you don't have to re-enter usernames, etc.?

Yes.

• When entering a Wikimedia project, should there be a dropdown window that lists all language projects?

This would be probably a good thing to include in a tool.

• This tool will have the ability to export a CSV of all edits that are counted in the metrics. Would this be useful to you? It will look like the Recent Changes page on wiki.

At this moment, I don't see how this would be useful for the evaluations and reports, but maybe others think differently.

• The interface will be translated through translate wiki. Who could we contact to help translate into your language?

Maybe @Mickey Mystique and Miljan Simonović: are willing to help. :)

Thank you for the recommendations :)

• Add an "event type" into the program/event description? (e.g. so that all edit-a-thons, writing contents, photo contests, can all be categorized)?

Definitely. Some events are not a one-time-thing (e.g. Wikipedian in residence is an "event" that lasts a month or two) and it would be better to categorize them differently. If this would be a case, would it be possible to include some filters so that, for example, we can get metrics just for the Wikipedian in residence (if we have two or more WIR) or just for edit-a-thons?

From what I've heard from other organizers so far, this is a "nice to have" feature, meaning it's not something people need immediately but it would be nice to have in the future. Would you agree with that, or is the ability to add "event types" a critical functionality that you need for your programs?

Other questions/suggestions

• Is there a possibility to divide projects? For example, if we have one participant who is a part of the Wikipedia competition and Education project at the same time, how can we divide metrics for these two projects if they're in the same time period?

I hadn't thought of this problem, and I agree it's a tricky one to solve. Right now, there wouldn't be a way to divide a users contributions by program (especially for simultaneous programs), unless their contributions are specifically tied to a program on wiki via something like a category, template or hashtag. For instance, if it's a writing competition, and all of their submitted articles have a competition-specific talk-page template/category added, then you could use this tool to count just their competition edits by entering their username and the template/category. This would count just their competition edits, instead of counting all of their edits in general. However, outside of something like a template, category or hashtag, there isn't is a good fix to this problem that I know of.

• When we use Dashboard, do we get articles in all name spaces or those include only articles in main name space? If it includes articles in all name spaces, is there a possibility to get only those in main name space? Same question for the new tool.

To my knowledge, in the Dashboard, the article metric counts only contributions to the main namespace. For this tool, it would only count things in the main namespace, except for Commons, where it would only count things in the File namespace.

• I agree with AWossink regarding the first statement.

--IvanaMadzarevic (talk) 12:09, 10 August 2017 (UTC)

@IvanaMadzarevic: Apologies for how long it took me to respond, but thank you so much for your feedback Ivana! I have incorporated it into that larger set of feedback that I've gotten. I've also responded in-line to some of your suggestions or questions. Feel free to ping me if you have any additional questions or concerns. -- Shouston (WMF) (talk) 18:37, 22 August 2017 (UTC)

Feedback from Camelia (WikiDonne)

I hope to be in time for my feedback. I don't have clear what is about Categories section and Templates section (a list of the categories and templates used in the articles?).

I use WikiEducation Dashboard for WikiDonne's event (courses, contests, editathons) and I find that Grant metrics tool (GMT) is very similar to what Dashboard just do now.

Similitudes:

• Possibility to insert the project language
• Possibility to insert a whole list of users, and not one by one
• Start and end date
• New articles
• Improved articles

Differences:

• GMT doesn't allow to assign articles to users
• As it is any connection between the user and the assigned article, is not clear if GMT can track the revisions filtered by both user and article (with the improvment we did at Wikimania in Montreal, now Dasboard can do this)
• Dashboard doesn't (yet) misure the retention. The WikiEdu team is working on this issue.
• Dashboard doesn't track the number of new editors
• Dashboard doesn't export totals
• Dashboard doesn't track the Categories and Templates

In conclusion, as having many tools can be confusing, instead of create a new tool, I suggest to improve the Dashboard existing now with useful functions such as:

• Possibility to change the name of the event (the event can have an ID as unique identifier)
• Possibility to choose the type of the event from a combobox (a course for example need more time than a editathon or contest)
• Possibility to close finally the event without being possible to change it again
• To avoid errors in typing, a combobox for languages is better than a textbox
• Misure the retention and interest (numbers of revisions on the articles page/talkpage after the event is finished)
• Possibility to set the privacy of the event on private/public
• Possibility to insert in every article the categories and templates used. This way it can be check the completeness of an article
• Possibility to track the quality of the article by checking in every article the presence of warnings ({{stub}}, {{Citation needed}}, {{Notability}} etc).
• Possibility to export totals in csv, sheets etc
• Possibility to print the pages

--Camelia (talk) 09:41, 18 August 2017 (UTC)

@Camelia.boban: Hi Camelia - Apologies for how long it took me to respond. Thank you for taking the time to review and give your feedback on this proposal! You were definitely not too late :) You’re completely correct that for those who use the Dashboard today, what is proposed is similar to the Dashboard (i.e. it’s a tool that collect usernames and tracks metrics). And for those who want to use the dashboard, they should; I believe grantees should use the tool that’s right for their program.

But the Dashboard isn’t the right tool for all programs, and I’ve heard from a number of grantees that they either don’t use the dashboard or choose not to use the dashboard, for a variety of reasons. For those people, having another tool that works, is easy to use, and collects all the metrics WMF asks for is really important. Unfortunately there is no tool like that today - Wikimetrics has a lot of known problems, and other tools aren’t maintained. So this proposal is really about updating Wikimetrics - keeping the good parts, replacing the bad/broken parts. It’s just that this overhaul it so big, it’s essentially going to feel like a new tool. Reading the proposal again, I can see how that’s not readily apparent, as the explanation is buried in the FAQ. That’s my fault for not making this clearer.

But I also hear for you and others that updating the Dashboard is important, for those who do use it for their grants. I’ve heard a lot of good feedback about the Dashboard (publicly and privately) and I’m going to aggregate and anonymize that feedback, and share it with the Dashboard team. Obviously, it’s up to that team to prioritize what to do with that feedback, but at least it will be shared.

I hope this response addresses your concerns about creating another tool. But if not, I’m happy to talk on-wiki or off-wiki some more! — Shouston (WMF) (talk) 21:23, 28 August 2017 (UTC)

Hi Sati, thank you. I'm allways glad to see tools that make our work easier, so I'm offering myself to test the new tool when will be ready. Thank you, ;-). --Camelia (talk) 22:34, 2 September 2017 (UTC)

Hi Camelia! Hope you've been well :) Would you still like to be a tester for the new tool? I sent you an email, but just in case: There would be nothing to do now; only in late January once the prototype is ready. I would probably take no more than 1 hour. -- Shouston (WMF) (talk) 23:28, 11 December 2017 (UTC)

Absolutely yes, I'm ready to test in the late january. In the mean time, have a Happy Christmas . --Camelia (talk) 10:09, 13 December 2017 (UTC)

@Camelia.boban: Wonderful! Thank you so much :) I'll reach out in late January/early February. I hope you have wonderful Christmas and New Years too! -- Shouston (WMF) (talk) 18:28, 15 December 2017 (UTC)

@Camelia.boban: Version 1 of the grant metrics tool is finally available for testing! It took a bit longer than anticipated, but I hope you are still interested in testing the tool and sharing your feedback :) Here's the link: https://tools.wmflabs.org/grantmetrics/. If you are still interested, would you have 30 minutes to talk in the next few weeks about this? I can set up a google hangout, Skype call, or phone call - whatever works for you. Thank you again! -- Shouston (WMF) (talk) 22:58, 8 March 2018 (UTC)

Yep, any day after 8PM CET (no weekend). Thank you. --Camelia (talk) 23:05, 8 March 2018 (UTC)

Feedback on P&E Dashboard from user research

Hi Shouston (WMF)!

I’ve heard a lot of good feedback about the Dashboard (publicly and privately) and I’m going to aggregate and anonymize that feedback, and share it with the Dashboard team.

I would love to get that feedback. Most of what was on the roadmap so far for grants metric tool was also on the 'nice to have' list for the Dashboard, and getting confirmation that some of those things were high priority for grant reporting gave me the little nudge I needed to get them done when I had some spare time in the last week. (Namely, tracking by category or template, per-wiki metrics and new editor retention counts, and a private program option... all live on P&E Dashboard as of yesterday.) But I don't get to spend nearly as much time talking with P&E users as I do with users of Wiki Ed Dashboard, so getting that info is super great for finding easy wins like that. It also helps hugely with keeping up a good supply of beginner-friendly coding tasks available. Thanks much! --Sage (Wiki Ed) (talk) 19:54, 8 December 2017 (UTC)

Hi Sage (Wiki Ed)! Apologies for not reaching out sooner! Other work has been a hectic, so I'm a bit behind on sending this over to you. How would you like me to send the info over? I'm also happy to chat with you directly if you think that would be easier / faster :) -- Shouston (WMF) (talk) 21:06, 8 December 2017 (UTC)

Shouston (WMF): if you have interview notes or some-such, I'm happy to get them in whatever form you've got them in: sagewikiedu.org. Or happy to schedule a chat — maybe sometime next week? — if you prefer.--Sage (Wiki Ed) (talk) 00:00, 9 December 2017 (UTC)

Sage (Wiki Ed): Let's do it over chat - it will be easier for me to provide context of the feedback, as well as answer questions that might come up. I'll reach out to you now via email to schedule a time. Talk to you soon! -- Shouston (WMF) (talk) 18:53, 11 December 2017 (UTC)