Wikimedia Meta-Wiki

Community Tech/Event Metrics

< Community Tech


The third most popular wish of the 2017 Community Wishlist Survey asked for a “next generation tool” to quickly and easily provide the metrics that event organizers need to help measure their projects and demonstrate the impact of their work. This page tracked the development of that tool, which we named Event Metrics.

This page documents a project the Wikimedia Foundation's Community Tech team has worked on or declined in the past. Technical work on this project is complete.

We invite you to join the discussion on the talk page.

The Event Metrics tool is live. Please give it a try; all you need is a wiki login. The Event Metrics help page provides a walkthrough of the tool and definitions of all the metrics.

Main improvements

edit

Event Metrics is based on—and will replace—the existing Grant Metrics tool. Below are a list of the main new features we plan to add (unlisted are the many interface and back-end upgrades that will make the system more friendly and capable generally). These improvements will give users many new types of data, in formats they didn't have before, and enable whole new types of users to define and track events. (As always, these plans are subject to change in the event we encounter unforeseen difficulties—last updated Dec. 21, 2018; check Updates for developments).

New downloadable reports (CSV and Wikitext)
  • Event Summary provides totals for pages created and improved, files uploaded, Wikidata items created and improved  and other contributions as well as impact metrics like pageview for these. More info.
  • Pages Created lists all new articles along with their creators, accumulated pageviews, etc. More info
  • Pages Improved lists all pages that were edited along with their editors, avg. pageviews per day, etc. More info.
Filtering improvements
  • Category filter upgrades: The Category filter lets organizers restrict metrics to articles in particular categories. We're working on improvements that will make Category filtering more flexible and useful:
    • Category filtering of uploaded files: To make the filter useful for tracking images, videos and other types of uploaded files, we’re also adding the ability—on Commons only—to  filter files by categories. Read more.
  • Independent Category or Participant filtering: Currently, all events must include a Participants list. This can be a problem for people who organize events that don’t include formal sign-up. So we’re eliminating that requirement and allowing you the flexibility of filtering by categories, participants or a combination of both (you are required to pick one of these options). Read more.


Follow this project—and tell us what you think

edit

Project updates

edit

The new Event Metrics is LIVE (in beta)—please try it (March. 31, 2019)

edit

The Community Tech team has launched the big upgrade to Event Metrics I’ve been talking about on these pages. With this release, event organizers get many new metrics and powerful new features for measuring event participation, contributions and impact (see above for a list).

Please give it a try; all you need is a wiki login. If you have events already in the program, even from when it was called Grant Metrics, they'll still be there. You need only click “Update data” to get new metrics and reports for these existing events.

The sooner people give Event Metrics a try the better. Right now, CommTech is still on active assignment to address Event Metrics problems. We’re listening and ready. It’s too late to add new features (though we want to hear your ideas), but we can make adjustments to things that are confusing and can fix things that aren’t working. (We’ll always fix real “bugs” to the program, of course, but right now the definition of what counts as as a problem is broader than it will be once we’ve moved on in a few weeks and shifted our focus to the next group of users’ Wishlist wish.)

Consider the program in beta right now; we tested it but are sure to have missed things. So please have a look and tell us what's broken, what's confusing and what you love! Here’s the talk page where people will be reporting issues from now on.

Pinging you one last time: the new Event Metrics is live

edit

I’ve pinged you because you voted for the Wishlist wish that inspired this project and we thought you’d want to know that it is live. Please give it a try. As it explains in the post above, the sooner the better. Thanks again for your advice and support.

@Pbsouthwood:, @Bspf:,  @Donald Trung:, @Wikinade:, @Nick Moyes:, @Zhangj1079:, @OrsolyaVirág:,  @B20180:, @Exilexi:, @Rachel Helps (BYU):, @Mickey83:, @Becksguy:, @VIGNERON:, @Ozzie10aaaa:, @Superchilum:, @Romaine:, @Theklan:, @JenOttawa:,  @Ɱ:, @Ckoerner:, @Arian:, @Kerry Raymond:, @Doc James:, @Muhraz:, @Hexatekin:, @Pamputt:, @Satdeep Gill:, @NickK:, @Townie:, @Martin Urbanec:, @Vachovec1:, @Battleofalma:, @YjM:, @Theredproject:,  @Mckensiemack:, @Hasive:, @Nattes à chat:, @Winged Blades of Godric:, @Meredithdrum:, @Artchivist1: JMatazzoni (WMF) (talk) 04:58, 1 April 2019 (UTC) [reply]

@Joalpe:, @Ijon:, @Krishna Chaitanya Velaga:, @Discott:, @David1010:,  @Liuxinyu970226:, @Jc86035:, @Megs:, @Thomas Obermair 4:, @Shizhao:, @Paucabot:, @FULBERT:,  @Rhododendrites:, @JaxieDad:, @PointsofNoReturn:, @Annamariandrea:, @Halibutt:, @Masssly:, @Voltaireloving:, @Sarahobender:, @Davidpar:, @HugoHelp:, @Vallue:, @Medol:, @Tiputini:,  @Monikasj:, @Stinglehammer:, @Caorongjin:, @John Cummings:, @Fixer88:, @Spiritia:, @Epìdosis:, @LornaMCampbell:, @Mtmlan84:, @Ouvrard:, @Mauricio V. Genta:, @Anthere:, @Kvardek du:, @Jorid Martinsen (WMNO):, @Uncommon fritillary: JMatazzoni (WMF) (talk) 05:01, 1 April 2019 (UTC) [reply]

@African Hope:, @Astrid Carlsen (WMNO):, @Richard Nevell (WMUK):, @Alangi Derick:, @Talueses:, @Wittylama:, @VMasrour (WMF):, @TrMendenhall:, @Jmatazzoni:, @Ecritures:, @MichaelMaggs:,  @Pharos:, @Heathart:, @John Andersson (WMSE):, @Haxpett:, @Psychoslave:, @Anne-LaureM:, @Sylvain WMFr:, @Yohannvt:, @Lirazelf:, @ArtEmJJ:, @Jack who built the house:, @Chicovenancio:, @جواد:, @Xavier Dengra:, @לסטר:, @Esh77:, @Bijay chaurasia:, @Shangkuanlc: —JMatazzoni (WMF) (talk) 05:04, 1 April 2019 (UTC) [reply]

Thank you @JMatazzoni (WMF):, I will look at it as soon as I can. Cheers, Psychoslave (talk) 12:29, 1 April 2019 (UTC)[reply]
Thanks! ccing our main organizer, @Raggachampiongirl:Heathart (talk) 03:11, 8 April 2019 (UTC)[reply]
Thank you for flagging me @JMatazzoni (WMF):. I will try it out! JenOttawa (talk) 12:14, 12 April 2019 (UTC)[reply]

New designs for data display and filtering interface (March. 26, 2019)

edit
 
Summary data: the Event summary page presents its many new metrics in an easy-to-read format.

I’m writing to share designs for the new interface the team is working on for one of the main Event Metrics pages—and in particular for the new filtering interface that will be part of the page.

New  ‘Summary' data display

edit

All of the mockups here show the redesigned Event Summary page. This is arguably the main page of the tool, where you can see top-line metrics about your event, download reports and add filtering. The image at left shows how the page presents all the new metrics the system produces. For easier comprehension, the numbers are organized into sections: “Contributions,” “Impact” and “Participation.” (Don't worry if you don’t understand all the metrics here; in practice, the little gray “info” icons next to each figure will provide definitions.)

 
'Filters required': a red error box explains what the user must do to produce metrics.

New filtering interface helps you get set up

edit

One of the main differences between Event Metrics and the former Grant Metrics tool is that Event Metrics doesn’t require users to  filter contributions by a list of participants' usernames ("Participant" filtering). Instead, organizers can chose between filtering by Participants or by the new Categories filter—or they can combine the two to focus their event even more narrowly. Filtering is required, partly for performance reasons—we don't have the capacity to report on everything that happened on the wikis during your event. And the different filtering types work differently in some ways (e.g., Participant filtering applies to all wikis, while Categories are specific to individual wikis).

This two facts points to a challenge we had in designing the system: with the new filtering capabilities comes a certain amount of complexity. To use the system effectively, users will need to understand a bit about how it works. To help you with that, the new filtering interface provides context-sensitive help. Here are a couple of examples of how it works.

 
'Event partially configured': the user has added filtering for some but not all wikis.

“Filters required”

The mockup at right shows the Event Summary page as it will appear just after a user has created an event but before applying any filters. The red box in the image announces that the user needs to add filtering and explains the requirements. This information is reinforced by messages and red icons on the filtering panel headers.

“Event partially configured”

In the mockup at left, the user has partially configured the event. The system shows metrics for some wikis but not all (I’ve cut off the page for space reasons). The yellow box tells the user which wikis lack filtering and explains what the user needs to do to fully configure the event and get metrics for all wikis.

What do you think?

edit

If you want to read more on the new interface, this ticket goes into detail. To learn more about how filtering works, see the Event Metrics help pages. Our hope is that the interface you’ve just previewed will help users learn what they need to know. Please have a look at the mockups. Do you like the new data display? Does the context-sensitive help seem effective and clear? Do you have questions? Let us know what you think; we're listening.

Revised project plan and priorities (Feb. 20, 2019)

edit

As the team has worked on and learned more about Event Metrics, we’ve had to revise the schedule and feature list somewhat to cut scope for what has turned out to be a challenging project. The list below shows the revised list of main deliverables we're working toward and the rough order of feature releases, which you can also find in the Status and schedule section of this page.

New downloadable reports

Event filtering improvements

As you can see, the first push is to deliver new reports and lots of new data—particularly in the area of “impact” metrics, which measure the size of the audience event contributions garner.

Next, we’ll make the Category filter significantly more flexible and useful. In particular, we’ll make it independent of Participant filtering, which is currently required for all events. That will make Event Metrics useful for events that don’t involve a formal sign-up process. We’ll also add the ability to use Category filtering for files on Commons—in order to better serve image drives and similar events (Category filtering currently works only for pages). Finally, the ability to filter by talk-page categories will help WikiProjects and others who classify pages and files by putting categories on talk pages. (The system will behave as though the talk-page categories were applied to the associated main pages—counting only the main-page changes.)

Some of you will notice that a feature we'd been hoping to build, Worklist filter, is no longer on the to-do list. The reason for this, as I alluded to above, is that Community Tech has a limited amount of time we can spend on each project, and integrating Worklist with the existing Event Metrics reports and filters would be a complex job. It would also require creation of a completely new user interface for event setup, so as to clarify a series of confusing interactions among filters, wikis and contribution types. Category filter, meanwhile, exists already, and with the improvements listed above we hope it will work for many events to achieve similar goals.

Please let us know what you think about that, and give any other questions or comments about this plan. What could we do to make it better? Which features are you most excited about? Are there any you would skip? We’re listening. —JMatazzoni (WMF) (talk) 21:22, 20 February 2019 (UTC)[reply]

Details on metrics planned for two new reports (Feb. 12, 2019)

edit

The team is making good progress toward creating the two new Event Metrics reports slated for our first release. Our plans are firm enough now that I’ve published a Help page detailing all of the metrics we expect these reports to include: Event Metrics/Definition of metrics.  

As this page makes clear, the new reports provide many numbers that weren’t available in the Event Metrics predecessor, Grant Metrics. In particular, so-called “impact" metrics—like “Views to pages created” or “Avg. daily views to files uploaded”—will now let event organizers and their partners gauge the size of the audiences their contributions are garnering.

The Definition of metrics page spells out how each metric is calculated and exactly what it does and doesn’t include. I know organizers will be interested in these specifics, so I hope you’ll take a few minutes to look these definitions over and offer your thoughts. What are you excited to see? What is unclear or doesn’t seem right? I’ve set up a section on the project talk page for your comments. We’re listening. —JMatazzoni (WMF) (talk) 23:48, 12 February 2019 (UTC)[reply]

Oct. 26: Metrics changed, delayed or dropped

edit

Last month I proposed a list of specific figures we were planning to include in a series of new Event Metrics reports. As we’ve started working on the first two of those reports—Event Summary and Pages Created—we’ve learned more about what we can and can’t get from our databases at scale. In response, our plans have evolved and adapted. In this post, I lay out the changes we think make sense, so that you can please respond and offer your advice. (To get the specifics and find out what is in those two reports under construction, check Event Summary and Pages Created tickets.)

Metrics changed or limited

edit

The following are all metrics we plan to include, but each has changed in some way compared to what I’d originally imagined.

  • ‘Pageviews to files uploaded’ This is likely going to change to become “Average. daily views to files uploaded”—which is to say, a per-day figure instead of a cumulative one. The reason: while we can easily figure out what pages a given file has been added to, there is no easy way to know the date when the image was added to a page. Without that date, it’s impossible to accurately figure out the total pageviews accrued. So here’s the workaround we’re proposing: 1) determine the pages the file is on at present, 2) look at the past 30 days of traffic to those pages (in order to smooth out normal daily fluctuations), 3) divide by 30 to express that total as a daily average (learn more about this here). Does that sound like a useful approach? How important for your work is this metric?
  • ‘Namespace’ Grant Metrics (the program we’re transforming into Event Metrics) currently tracks only Main namespace for things like pages created, edits made, etc. (It looks at other namespaces when necessary for specific metrics, like “Files uploaded.”) We’d thought about giving organizers filters to let them selectively track a few other namespaces. But that flexibility adds too much complexity to the project (for a first version, anyway). Also, people should know that each namespace we search essentially doubles the system load and wait times. And we're already pretty concerned about both of those. On the other hand, the Draft space is pretty heavily used by new users—particularly on English Wikipedia, where newbies aren't allowed to create articles in Main. So, as an experiment, we're planning to include Main + Draft namespaces (for the wikis that have Draft). If wait times soar, we may have to back that out again. On the other hand, if the experiment goes well, we can consider adding another space. (A side note: for searches that use the Category filter—coming soon!—we’ll apply Talk page categories to Main namespace pages. This will make the Category filter work for Wikiproject members and others who, by convention, apply their special-purpose categories to Talk instead of Main pages. Read more on this.)  Does this experiment sound valuable? Are there other namespaces you would want to include? If you think so, please tell us why that namespace is very important for your work.
  • ‘Still exists?’ & ‘New page survival rate’ On the new “Pages created” report, the “Still exists?” metric tells you whether a given article has been deleted. On the “Event summary” report, the “New page survival rate” is an aggregate figure that tells what percentage of created pages remain. We can get this information from the Archive log—but there’s a catch. The log doesn’t record categories. What that means is that  if you use the soon-to-be-released Category filter to define your event, you won’t get these survival metrics. Again, a better but more complex way to get this information exists. It requires us to store a record of all pages created during the event, which would provide a basis for comparison going forward. That’s a big change to how our system works, and it won’t come soon.  So in the meanwhile, on the theory that data about deleted pages is important for organizers, we’re planning to offer these metrics in the partial form just described. You can read more about this in the ticket.  Does that interim solution sound reasonable? How important to you is it to know which pages were deleted?
  • ‘% women (estimated)’ Organizers have asked for a way to track participation by women, who are underrepresented in wiki work. But there’s no reliable way to derive users’ gender from their wiki contributions. So this metric—planned for the Event Summary report—is unique in being something that organizers will have to enter themselves, based on their own headcount or survey. I asked organizers about this last week during a presentation at WikiConference North America, and their response was that yes, we should include the metric. But they said it shouldn’t ask just about women vs. men. So the new plan is to give organizers the option to record “# of Women”, “# of Men” and “# of Other” for any event.  If organizers fill in those fields, they’ll see the numbers reflected back as percentages in both event and program reports. Here’s the ticket where this is described. Your thoughts?

Metrics dropped or delayed

edit

The following metrics were proposed but have proved more challenging than expected. At present, we have no plans to develop any of them in the near term. Your input could change those plans. But know that resources for this project are limited: if we take on any of these complex metrics, other features will have to drop out.  

  • ‘Words added’ This metric would provide a very handy measurement for the amount of work done during an event. Organizers I spoke with at WikiConference North America said this is something they would like to have, even if the figure were inaccurate (as it probably would be). But calculating words added at scale across multiple scripts and hundreds of languages  is a very big challenge. Recognizing word limits is actually a non-trivial engineering problem. It might be somewhat obvious in English, but is definitely not in other languages where “space” (and dot, and dash, etc) are not the real indicators of word separation. On top of that, we’d need, for every given page, to look at the content and analyze (“parse”) it, which would add a potentially significant load to the rest of our already heavy querying methods. We’ve created a ticket to investigate the issue; please add your suggestions if you have ideas.
  • 'Description' This was meant to be a short description of a file or article—taken from the first sentence or from wikidata, etc. The idea was that it would make reports like “Images uploaded” or “Pages created” more understandable to bosses, partners and others who might read these lists of sometimes obscure file or article names. It could be done, but would require us to parse the wikitext of pages, which is hard and resource intensive. Let us know if this seems like an important omission.
  • 'Wikidata claims added' A “claim” is a statement + an identifier—basically a discrete fact in Wikidata. We will report on Wikidata items created—entirely new entities. But, once again, this data is not something we can simply look up. Tracking the claims within items would require the system, again, to parse the wikitext itself, which is resource intensive at scale. Reactions?
  • Bytes, edits, words ‘changed subsequently’ These three figures were meant to track the work that happens to an article “subsequently” to the conclusion of the event. They were proposed as a way to index community interest in a given article, as measured by people continuing to work on it. But these figures are subject to the same limitation described above under “‘Still exists?’ & ‘New page survival rate’”: to do them properly, we’d need to make a record of all pages created during the event, to use as a basis for comparison going forward. We want to do that, and these figures may become possible in the future.  If these figures would be important to your work, please explain why.

Sept. 29, 2018: Which new features should we release first?

edit

We’ve started writing up tasks for the program of improvements to Event Tool I proposed last week. This leads to an important question: in what order should we build and release these new features? I’m writing today to lay out my thinking on that—and to see if you agree: share comments here.

Broadly speaking, the project involves three types of features:

  1. Six new downloadable reports
  2. A series of new filters that let you customize those reports
  3. Expanded and redesigned onscreen data displays —plus some other features if we can get to them

Three stages

edit
 
This wireframe mockup for the 'Event Summary' data screen shows a greatly expanded set of event metrics, along with a new layout that organizes data into sections for easier comprehension. A breakout table at bottom gives per-wiki results. At upper-right, the user downloads one of six new metrics reports. (Comment on this wireframe.)

Releasing these features in the general order listed above seems like the quickest way to bring the most value to users. This follows from the observation that the two main use cases for event metrics seem to be 1) to post them on wiki, where the participants and others can see them, and 2) to report outcomes to bosses, partners, grantors, etc., possibly in depth.

  • Stage 1, new downloadable reports: Building the downloadable reports first will help organizers with both of the use cases mentioned above. Users will get a lot of data not previously available from Grant Metrics in a choice of two formats: wikitext or spreadsheet. We can release the reports one by one as they’re ready.
  • Stage 2, filtering tools: In stage 2, we'll add the new filters and filter controls. The ability to limit metrics to a “Worklist” of articles (to improve or create) was highly requested, so it seems right to build that filter first. Then we can wade into the tricky business of providing controls that will let users turn different filters on or off, to tailor reports for different purposes or audiences. (A filter that enables organizers to limit results by categories was part of the original Grant Metrics project and is already in the works.)
  • Stage 3, onscreen data displays: In stage 3 we'll roll out onscreen displays with lots more data than currently and a new design. These metrics screens provide organizers with a read on event outcomes that is relatively detailed yet quicker to access than downloadable reports. (The wireframe mockup at right shows all the new data we'll be adding and a new design—what do you think?)  If we still have time by this stage in the process, we'll try to get to some other features that have been popular with organizers.  

Follow this project—and tell us what you think

edit

Let us know if the approach described above makes sense to you, or what you’d change.

  • To follow this project, go to the page for the Event Tools tag in our task-management tool, Phabrictor, and become a “Member”.
  • To see the the tasks for this project that we’re about to start working on, follow this link (which filters the Community Tech backlog board by the “Event Tools” tag). The tasks slated to move forward are listed in the column “To be estimated.”

Sept. 19, 2018: Proposing a product plan for ‘Event Tool’

edit

A month ago I posted on the talk page to ask what type of metrics and metrics features event organizers want. After analyzing the lengthy discussion that ensued, I proposed a set of metrics features and new metrics reports for what I’ve been calling Event Tool. Organizers responded again with comments and requests, and I  added some additional features (e.g., the ability to track audio and video plays and to get a gender breakdown of events/programs).

Today, I’m recommending  that we move forward with the program of work described in the two metrics posts mentioned above, making these posts (here and here) essentially the blueprint for this round of Event Tool development. The goal will be to turn Grant Metrics into a simple but capable tool that will meet the metics needs of most event organizers—a “one-stop shop,” as one organizer called it, for getting the data that many organizers now laboriously assemble from various sources.  

We need your feedback

I’ve called this metrics proposal “ambitious but achievable,” and I think that’s a good description. We can’t promise to deliver every single feature described, but the posts referenced above outline the product direction in detail. Now it’s your turn again: does this look about right? Is there anything we can add that would make it work better for you? And—since it’s possible we won’t get to everything—are there any features here you think we could drop or put on our low-priority list? Please leave your comments in the talk page section provided.

Why this direction?

edit

The Wishlist wish that kicked this project off, and which 111 people voted for, is completely focused on metrics, so new data and reporting features were a given from the start. Plus my research with organizers made it clear that metrics is one of the areas where they need significant help. Many other possibilities were considered, however.  

As part of the product-design process, I began by examining all stages of wiki event workflow, looking for pain points and framing possible solutions. Organizers expressed support for many of those ideas—particularly automated wiki page creation, event signup, and assistance with wiki account creation. But as the team investigated these event-management ideas, a few things became clear: accommodating the wide variety of organizers’ needs would make these tools more complex than they at first appeared. And many of these features don’t produce user value until the whole system is built and can work together. In some cases, the desired features are otherwise dependent on some other project. As just one example, some of the wiki account-creation improvements discussed require first that we build an event signup system. Event signup, in turn,  requires us to collect email, so for security reasons it can’t be built unless we move Grant Metrics from its current location, on the Tool Forge servers, onto our production servers.

Such complexities made event management features a big job. Meanwhile, in terms of project size, the proposed metrics features are already at the very high end of what’s allowed for a Wishlist undertaking.

Which is not to say that the event-management tools explored will never be built. The WMF’s annual plan this year recognizes organizers as “fundamental implementers” of the free-knowledge movement and  a "core asset.” The plan authorizes a Movement Organizer Study, scheduled for this winter, to better understand organizers’ needs and create a framework for “making strategic investments” to support them. Those future investments may well take up some of the Event Tool work discussed in these pages. (Organizers are also welcome to make a second Wishlist proposal.)

Next Steps

edit

First, we want to hear from you. Do you agree with the proposed direction? What is missing? What is unnecessary?

If it looks like organizers agree with the plan, then I’ll begin prioritizing and specifying the particulars for these new functions. We’ll also begin work on designs, and I’ll start showing those for comment as soon as I can. Since we’ll be expanding the  mission of Grant Metrics, it will probably be renamed: “Event Metrics” is the favorite at the moment.

One of the many advantages of the fact that Grant Metrics is an existing tool is that we'll be able to add features to it one by one, instead of having to wait to get a whole system up and running. So if organizers like the general plan, the team should be able to start rolling out new features in the next few months.

Reaching out again to people who voted for this Wishlist item:

edit

Because you voted for the Wishlist wish that inspired this project, we thought you’d want to know that it’s about to move into the development phase. The post above proposes the outlines for a metrics tool aimed at event organizers.  If that sounds like you, we’re eager for your input. Please read the posts and watch this page if you have a continuing interest.

@Paucabot:, @FULBERT:,  @Rhododendrites:, @JaxieDad:, @PointsofNoReturn:, @Annamariandrea:, @Halibutt:, @Masssly:, @Voltaireloving:, @Sarahobender:, @Davidpar:, @HugoHelp:, @Vallue:, @Medol:, @Tiputini:, @Monikasj:, @Stinglehammer:, @Caorongjin:, @John Cummings:, @Fixer88:, @Spiritia:, @Epìdosis:, @LornaMCampbell:, @Mtmlan84:, @Ouvrard:, @Mauricio V. Genta:, @Anthere:, @Kvardek du:, @Jorid Martinsen (WMNO):

@African Hope:, @Astrid Carlsen (WMNO):, @Richard Nevell (WMUK):, @Alangi Derick:, @Talueses:, @Wittylama:, @VMasrour (WMF):, @TrMendenhall:, @Jmatazzoni:, @Ecritures:, @MichaelMaggs:,  @Pharos:, @Heathart:, @John Andersson (WMSE):, @Haxpett:, @Psychoslave:, @Anne-LaureM:, @Sylvain WMFr:, @Yohannvt:, @Lirazelf:, @ArtEmJJ:, @Jack who built the house:, @Chicovenancio:, @جواد:, @Xavier Dengra:, @לסטר:, @Esh77:, @Bijay chaurasia:, @Shangkuanlc:

@Joalpe:, @Ijon:, @Krishna Chaitanya Velaga:, @Discott:, @David1010:, @Liuxinyu970226:, @Jc86035:, @Megs:, @Thomas Obermair 4:, @Shizhao:, @Pbsouthwood:, @Bspf:, @Donald Trung:, @Wikinade:, @Nick Moyes:, @Zhangj1079:, @OrsolyaVirág:, @B20180:, @Exilexi:, @Rachel Helps (BYU):, @Mickey83:, @Becksguy:, @VIGNERON:, @Ozzie10aaaa:, @Superchilum: —JMatazzoni (WMF) (talk) 22:16, 3 October 2018 (UTC)[reply]

@Romaine:, @Theklan:, @JenOttawa:,  @Ɱ:, @Ckoerner:, @Arian:, @Kerry Raymond:, @Doc James:, @Muhraz:, @Hexatekin:, @Pamputt:, @Satdeep Gill:,  @NickK:, @Townie:, @Martin Urbanec:, @Vachovec1:, @Battleofalma:, @YjM:, @Theredproject:, @Mckensiemack:, @Hasive:, @Nattes à chat:,  @Winged Blades of Godric:, @Meredithdrum:, @Artchivist1:, @Quiddity: —JMatazzoni (WMF) (talk) 22:21, 3 October 2018 (UTC)[reply]

Sept. 6, 2018: Proposed metrics features—what do you think?

edit

A “next-generation toolset” for producing more and better event metrics is the main requirement of the Wishlist proposal that launched this effort. Toward that goal, we asked what type of metrics and metrics features event organizers want, and received a lot of good ideas. I’ve studied that input carefully and worked through the various suggestions with our engineers to judge the level of effort involved in each. We can’t do everything; some of the features proposed would push this project beyond the scope of what a Wishlist wish can take on. But based on your suggestions, this post lays out a vision for an ambitious yet achievable reporting tool based on Grant Metrics (which we may need to rename to reflect its new, broader mission).

Below you’ll find 1) descriptions of the new features we are proposing, followed by  2) a diagram comparing existing vs. proposed Grant Metrics reports and 3) a detailed breakdown of all the new data we’re adding (look for the label “NEW”).

We need your feedback

Event organizers are a diverse group with varied needs. The first release of the tool described below is designed to provide the metrics that typical event organizers need while remaining relatively simple and easy to use. It probably won’t work for everyone—will it work for you? What could we do to make it work? Please let us know: I’ve posted a series of questions on the talk page to help structure the discussion there.

Lots of new data and new ways to get it

edit
  • Impact metrics: People asked for figures to help demonstrate the impact of their efforts,  so we’ve added new reports detailing Pages Created, Pages Improved, Commons Uploads and Wikidata Contributions, with new figures for # of pageviews, # of articles in which images were placed and much more.
  • Six new downloadable reports (in two formats): We’re proposing to add six new reports that you can download either as a spreadsheet file (.csv) or wikitext (see the diagram and descriptions below). The wikitext versions provide a quick solution for posting figures to wikis. The thinking is that they should offer less data than the spreadsheet files, on the theory that no one wants to post or edit a giant wiki table (does that sound right?).
  • More informative on-screen metrics: In addition to the downloadable reports, we’re adding around a dozen new metrics to the program-level and event-level screen displays.
  • Ongoing, cumulative metrics: By their nature, impact metrics (e.g., pages views to articles created) continue to build over time. The system will enable you to track such figures indefinitely, though you'll need to trigger updates manually, as described below. (Statistics relating directly to the event—e.g., # of participants or articles created—will not change once the event period you've defined ends.)
  • New descriptive information: To make reports more understandable for managers, partners and other non-specialists, they’ll include more explanatory information, such article or image descriptions, information about the type of event and the names of event partners.

Give feedback here about the ideas in this section

New tools for customizing reports

edit

A new set of filters will enable you to output different metrics for different purposes or audiences. You’ll be able to turn each of the following options on or off.

  • Limit metrics to “Worklist” articles only:  Restrict reports to a “Worklist” of articles to be improved or created that you enter by copying and pasting (as a batch, not one by one).
  • Limit to specific categories:  As an alternative or addition to the above, you'll be able to restrict metrics to articles in a particular category or set of categories—including categories you define specifically for this purpose (task already in progress).
  • Limit to participants list: Limit metrics to contributions from participants that you enter by copying and pasting a list of usernames. Grant Metrics does this by default now; we'll make it optional.
  • Limit by wiki: For events that encompass multiple wikis, this option will enable organizers to focus reports on just one.
  • Specify desired namespaces: Grant Metrics currently counts contributions to Main namespace only. Main will remain the default, but we’ll let you pick your preference.

Give feedback here about the ideas in this section.

Assumptions, limitations, ideas for future releases

edit
  • Real-time (though not automatic) metrics: Grant Metrics provides the ability to get metrics on demand—during an event, for example. The organizer must trigger the update by pressing an Update button (i.e., updating does not run on a schedule).  
  • Auto-updating of wiki tables: As described above, the system will output reports as wikitext tables for easy posting. We could add the ability for on-wiki tables to update automatically when you update your figures in the tool (see the preceding item). If this is something people want, it would not be overly difficult to build.
  • Focused on metrics we can uniquely provide: Some users suggested tracking metrics from Facebook and other social media channels used for event promotion.  While this sounds useful, the ever-changing variety of popular services, our general lack of experience interfacing with them, and the complex privacy issues involved puts this out of scope for now.
  • Track citations added: This was a popular request and for good reason, since the presence of citations are a good indicator of how substantial and durable contributions are liable to be. But unlike the other metrics listed below, which can be gathered by examining metadata, identifying citations would require the system to parse the actual wikitext or HTML content of pages, which would be difficult and resource intensive.
  • Login required:  Only people with your login will be able to see your metrics. One organizer expressed a desire to be able to send links to pages in the system as a way of distributing reports. It makes sense, but the permissions-management issues involved put this idea out of reach for now. (It will be easy, however, to post the metrics on wiki, as described above.)
  • Useful on mobile: Grant Metrics works on mobile, but needs improvements to work well. (If mobile event metrics are important for you, please speak up.)

Give feedback here about the ideas in this section .

Sept. 6, 2018 (continued): New data and reports in detail

edit
 
A diagram showing the hierarchical structure of Grant Metrics reporting as it is now and how the new downloadable reports will fit in. New reports are shown in blue.

The post above describes the new reporting features proposed for Grant Metrics at a high level. This post outlines in detail all the new data types and reports we're proposing. Look below for the label “NEW.”

Multiple Programs-level reports

edit

Metrics for all an organizer’s “programs” (each program contains multiple events).  

All My Programs — on-screen report

edit

Existing metrics per program

  • List of all my programs, with links to each
  • # of events in each program
  • # of participants in each program.
  • # of new editors
  • # of pages created
  • # of pages Improved
  • # new editors retained after 7-days

Proposed metrics per program  (NEW)

  • # of edits made
  • # of bytes added to pages  
  • # of bytes removed from pages  
  • # of words added
  • # of Commons uploads  
  • # of Wikidata items created
  • # of Wikidata claims created (claim=statement + identifier)
  • # of views to all pages created (cumulative, since the event)
  • # of views to all pages edited (cumulative, since the event)
  • # of views to all images uploaded (cumulative, since upload)
  • # of plays to uploaded video and audio files (cumulative, since the event)
  • # of pages in which uploaded images are placed
  • # of pages created that were subsequently deleted
  • Gender breakdown of all programs [must be manually entered per event by organizer]

Give feedback here about these reports and the data in them.

Single Program-level reports

edit

Metrics for each event in a single program

Single Program — on-screen report

edit

Existing metrics per event

  • List of all events in program, with link to each
  • # of participants
  • # of  new editors
  • # editors retained after 7-days
  • # of pages created
  • #  of pages Improved

Proposed metrics per event  (NEW)

  • Length of the event (days / hours)
  • Event type (editathon, content drive, training session, etc.)
  • # of edits made
  • # of bytes added to pages  
  • # of bytes removed from pages  
  • # of words added
  • # of Commons uploads  
  • # of Wikidata items created
  • # of Wikidata claims created (claim=statement + identifier)
  • # of views to all pages created (cumulative, since the event)
  • # of views to all pages edited (cumulative, since the event)
  • # of views to all images uploaded (cumulative, since upload)
  • # of pages in which uploaded images are placed (cumulative, since upload)
  • # of plays to uploaded video and audio files (cumulative, since the event)
  • # of mainspace pages created that were deleted (after the event)
  • Gender breakdown of program [must be manually entered by organizer per event]

Single Program — CSV & Wikitext downloadable reports (NEW)

edit
  • Spreadsheet (.csv) and wikitext files of everything above plus
  • Extended retention figures (e.g., 1 mo, 2 mo, 6 mos, 1 yr)

Give feedback here about these reports and the data in them.

Single Event-level reports

edit

Metrics for activities of a single event

Single Event summary — on-screen report

edit

Existing information

  • Event name
  • Wikis involved
  • Start time/date (with timezone)
  • End time/date
  • Date/time of last data update
  • # of participants
  • # of  new editors
  • # editors retained after 7-days
  • # of pages created
  • #  of pages Improved
  • List of all participant usernames

Proposed information (NEW)

  • Event type (editathon, content drive, training session, etc.)
  • Event partners (GLAMs etc.)
  • Event venue
  • Short description
  • Length of the event (in days / hours)
  • # of edits made
  • # of bytes added to pages
  • # of bytes removed from pages  
  • # of words added
  • # of Commons uploads  
  • # of Wikidata items created
  • # of Wikidata claims created (claim=statement + identifier)
  • # of views to all pages created (cumulative, since the event)
  • # of views to all pages edited (cumulative, since the event)
  • # of views to all images uploaded (cumulative, since upload)
  • # of pages in which uploaded images are placed
  • # of plays to uploaded video and audio files (cumulative, since the event)
  • # of mainspace pages created that were deleted (after the event)
  • Gender breakdown of event [must be manually entered by organizer]

Single Event summary — CSV & Wikitext downloadable reports (NEW)

edit
  • Spreadsheet (.csv) and wikitext files of everything above plus
  • Extended retention figures (e.g., 1 mo, 2 mo, 6 mos, 1 yr)

Give feedback here about these reports and the data in them.

Pages Created  — downloadable reports  (NEW)

edit

CSV

  • Page name
  • Page URL
  • Short description/first sentence
  • Username of creator
  • Wiki  
  • Namespace
  • # of edits to the page during event
  • # of edits to the page subsequently
  • # of bytes added to page during event
  • # of bytes added to page subsequently
  • # of words added to page during event
  • # of words added subsequently
  • Current article class (where available)
  • # of pageviews to page (cumulative, from creation to now)
  • Avg. pageview to page per day
  • Does page still exist?

Wikitext

  • Page name / link
  • Short description/first sentence
  • Username of creator
  • Wiki
  • # of words added to page during event
  • # of pageviews to page (cumulative, from creation)
  • Avg. pageviews to page per day

Pages Improved  — downloadable reports  (NEW)

edit

CSV

  • Page name
  • Page URL
  • Username(s) of editor(s)
  • Wiki
  • # of edits to the page during event
  • # of bytes added to page during event
  • # of bytes removed from page during event
  • Net % change in page bytes during event
  • Bytes added to page subsequently
  • # of words added to page during event
  • # of words removed from page during event
  • Net % change in words during event
  • # of words added subsequently
  • Current article class (where available)
  • Avg. pageview to page per day
  • Does page still exist?

Wikitext

  • Page name / link
  • Short description/first sentence
  • Username of creator
  • Wiki
  • # of bytes added to page during event
  • # of words added to page during event
  • Avg. pageview to page per day

Give feedback here about these reports and the data in them.

Files uploaded — downloadable reports (NEW)

edit

CSV

  • File name
  • File URL
  • Description
  • Media type (image, video, audio, text, 3D)
  • Source
  • Username of uploader
  • # of pages in which file is placed
  • # of pageviews to file (from upload to now)
  • # of plays to uploaded video and audio files (from upload to now)
  • Avg. pageviews to file/day
  • Does file still exist?

Wikitext

  • File name/link
  • Description
  • Media type (image, video, audio, text, 3D)
  • Source
  • Username of uploader
  • # of pages in which file is placed
  • # of pageviews to file (from upload to now)
  • # of plays to uploaded video and audio files (from upload to now)

Give feedback here about these reports and the data in them.

Wikidata Contributions —  CSV and Wikitext downloadable reports (NEW)

edit
  • Item label and/or Q number
  • URL
  • Description
  • Username of editor(s)
  • Was item created? (y/n)
  • # of claims added to item
  • Does item still exist?

Give feedback here about these reports and the data in them.

August 17: Event Tool metrics—tell us what you want

edit

As I noted in my last post, better and more metrics will be a focus for the Event Tool project. I’m writing now to ask event organizers for your ideas about metrics and reporting: What data do you need to show? What are you looking to accomplish? What filtering and other features are most important?


To provide some prompts for your thinking, I’ve posted a breakdown of the reports available now from the existing tools Grant Metrics and the Program and Events Dashboard, which many event organizers have used. Our operating assumption is that the Event Tool will use Grant Metrics for reporting (in fact, it may ultimately be just a series of enhancements to Grant Metrics).  So as you look at the lists linked to above, pay particular attention to Grant Metrics, especially to any key features you think it lacks.

To help organize the discussion, I’ve listed some specific question areas on the talk page; please provide answers there. The categories are:

Important links

edit
Retrieved from "https://meta.wikimedia.org/w/index.php?title=Community_Tech/Event_Metrics&oldid=24959343"