Wikimedia Meta-Wiki

Research:Notifications

This page documents a completed research project.


Key Personnel

edit
  • Aaron Halfaker
  • Dario Taraborelli
  • Fabrice Florin
  • Oliver Keyes

Project Summary

edit

The aim of this project is to evaluate the usage and impact of the new notification infrastructure (also known as Echo) designed by the Wikimedia Foundation and first deployed on the English Wikipedia on May 1, 2013.

Research Questions

edit

Our primary focus is on two sets of questions:

Usage
  • Who is sending and receiving notifications?
  • What is the volume of notifications delivered by type?
    • By comparison, the total number of mostly-MediaWiki email notifications sent is about 3.3 billions per year.[1]
  • How many notifications are delivered on-site vs other delivery methods?
  • What is the relative volume of notifications triggered by bots?
  • What is the conversion rate for different types of notification?
  • How does the use of private notifications compare to public ones (e.g. talk pages)?
Impact
  • How do notifications change user behavior?
    • Do notifications drive new editor retention and productivity?
    • Are there notifications that increase the impact of negative warnings?
  • At what rate of emailing do user's react (turning notifications to web only or opting out entirely)?

Data

edit

Data for research on Echo is mostly obtained by instrumenting MediaWiki via EventLogging. We are using three sources of data, described by the following models:

Schema:Echo
logs notifications generated by the system and their corresponding delivery method. Notifications logged with the 'email' delivery method indicate that they are staged for delivery to users with validated email addresses and not opted out, not that they have effectively been delivered.
Schema:EchoMail
logs how many email notifications are effectively being sent by MediaWiki, categorized by type (individual, bundled, digest)
Schema:EchoInteraction
logs user interactions with individual notifications.

We will use this data to measure conversions and other funnel metrics. We will also use cohort analysis (via the UserMetrics API) to run controlled tests and compare how the productivity and retention of different groups of users is affected by notifications.

On top of EventLogging data, we will monitor changes in preferences (user_properties) to determine the timing and volume of opt-outs or changes in default delivery methods.

We also rely on usertesting sessions and short surveys to study usabilty and UX design of this feature.

Controlled tests

edit
  • test 1 – overall effect of Echo notifications
  • test 2 – effect of revert notifications

Privacy and licensing terms

edit

Unless otherwise specified, all of the above data is private and subject to the Wikimedia Privacy Policy. Daily aggregates are released in the public domain and can be obtained via the Editor Engagement dashboards.

edit
Retrieved from "https://meta.wikimedia.org/w/index.php?title=Research:Notifications&oldid=17742109"