Research:Growth and diversity of Technology team audiences/Report

Identified audiences

Many of these audiences are shared across teams. Some of the audiences could be listed in multiple categories based on the team that's engaging with them—for example, many people who consume data from Wikimedia APIs do so for research purposes, but to the Services and Analytics team they are more likely to be considered "platform consumers".

Other audiences could arguably fit into multiple categories even within the same team, depending on the way the team is currently engaging with them. The Semantic MediaWiki extension has grown into a full-fledged independent open source project with a thriving developer community of its own—Semantic Mediawiki developers could be listed under "platform developers" or "platform partners" depending on whether the MediaWiki Platform team is engaging with individual contributors or the project as a whole.

Researchers

People and organizations who use Wikimedia Technology platforms (including research output and data resources) to perform research.

Platform contributors

People who make technical and non-technical contributions to our platforms.

Platform consumers

People and organizations who adopt our platforms or consume their resources.

Platform partners

Groups and organizations who work with Wikimedia Technology teams to improve our platforms or integrate them with their own technological platforms.

Stakeholder matrix

			Audience types
Team	Platforms discussed	Communication channels	Researchers	Platform developers	Platform consumers	Platform partners
Research	Research papers, curated datasets, code libraries, WM data dumps, APIs, Analytics dashboards	public mailing lists, Research newsletter, research.wikimedia.org, personal correspondence, metawiki, @Wikiresearch, blog posts, conferences and lectures.	academic researchers, data storytellers, NGOs	Code library contributors	Media entities, research educators	External collaborators, Grantmaking organizations
Scoring Platform	ORES, JADE, WikiLabels	Wiki-ai mailing list, mediawiki, village pumps, GitHub issues, IRC, Phabricator, conferences, lectures, and hackathons.	academic research analysts, model builders	direct contributors to ORES platform, developers of client applications ('local technologists'), WikiLabels campaign organizers	Orgs like Google and Kiwix who want to know which versions of an article to index	,
Cloud Services	ToolForge, Toolsadmin, CloudVPS	public mailing lists, IRC, wikitech, Phabricator.	academic researchers, Wikimedian researchers, the 'open knowledge-curious'	Bot and tool maintainers, documenters on wikitech
MediaWiki Platform	MediaWiki	MW Stakeholders Group meetings and mailing list, mediawiki, public MW mailing lists, conferences.	Researchers, research educators, data storytellers, NGOs	Volunteer extension developers, Mediawiki consultants		Semantic MediaWiki project, Mediawiki Stakeholders Group, WikiApiary project
Analytics	WikiStats2, RESTbase data API endpoints, datasets, EventStream, Browser Stats dashboard, Reportcard dashboard, WikiMetrics	Phabricator, wikitech, IRC, public mailing lists, personal correspondence.	academic researchers		Wikimedia movement volunteers, Wikimedia chapters & user groups, Orgs that use Wikimedia data (Google, Facebook, Microsoft
Security	Security reviews, vulnerability reports	Phabricator, IRC, conferences, hackathons, personal correspondence.			Developers in the community, independent developers, GSOC participants, MW consultants.	WMDE
Services	RESTbase, REST API	mailing lists, Phabricator, IRC, mediawiki.	Researchers who use the data API endpoints	RESTbase service developers	External MediaWiki users who want "the full Wikipedia Experience", Orgs like Google that use the REST API to crawl Wp
Search	MW Search API, WikiData Query Service, ElasticSearch	mailing lists, mediawiki, Phabricator, IRC, conferences	Wikimedian researchers ("editors who are trying to pull more information together about certain domains")	Open Source Connections (contracted to improve search models)	External MediaWiki users, organizational Search API users (e.g. Google), Orgs with SPARQL-compatible databases (e.g. museums)	WMDE, BlazeGraph

Common challenges

Measuring usage of our platforms

We have infrastructure in place to measure the raw usage of most of our technology platforms. In fact, for any given platform there are often multiple usage metrics available for a given platform, and usage is often (but not always) automatically logged. However, we lack a set of canonical, mutually-agreed-upon metrics within individual platforms and especially across platforms.

Measuring overall usage

Measuring the rate at which our platforms are used by external audiences can be challenging because what constitutes a meaningful measure of use varies by team and by technology, and because of limitations in the ability of the underlying infrastructure to measure usage automatically.

For example, the MediaWiki platform team has several ways of measuring usage of the MediaWiki software outside of Wikimedia projects: through direct downloads, WikiApiary, Git checkouts, and the pingback feature. Each of these measures has limitations: download rate does not indicate whether the downloaded software package was subsequently installed on a server; WikiApiary only tracks publicly hosted instances of MediaWiki; the pingback feature is opt-in. It is not clear which of these measures is most reflective of true adoption.

One place that we look periodically for information is WikiApiary. It has a list of all registered wikis. If someone has chosen to report there. It can also accept info from Wikis that are behind a firewall. The ones that are on the public web, there’s a bot that queries them. So you can go to a page and see how many people are using extension X, and what version of that extension. The other thing: we added a pingback feature. When the update script is run, if the pingback flag is set to true, it reports “home” 6-8 pieces of information that is anonymized, but unique. With downloads, you don’t know if they deployed it, and are running updates. As far as tracking downloads, we check things out of Git, and to the best of my knowledge there’s no one who’s capturing a Git clone. And that still doesn’t mean they’re installing the software. Going forward, having someone maintain WikiApiary would be great. It let’s you see extension status, which pingback does not, and it tells you “who”.

— Cindy Cicalese, MediaWiki Platform

Tracking usage over time

Some Technology teams are capable of tracking platform usage on an ongoing basic through automated processes, according to performance metrics that they have identified as meaningful and useful. For example, the Analytics team uses Piwiki dashboards to track comparative usage (e.g. request rate) of many of their web services over time. For this team, request rate is a useful metric for identifying possible incidents and for prioritizing support.

However, other teams lack the ability to accurately measure usage over time due to infrastructural constraints and/or explicit design decisions. The Services team, which maintains RESTbase, logs the overall request rate over time. However, because of the way RESTbase data caching is set up, the team only has access to the request endpoint (which API is being called) for approximately 10% of request traffic at any given time via RESTbase. To get the endpoint of 90% of their requests, they need to perform ad-hoc Hive queries. This makes it difficult to accurately compare the usage of different RESTbase APIs over time.

The Security team also has unmet needs regarding tracking trends in their security review process. In the past, the team has attempted to define metrics related to service quality and to manually gather monthly snapshots based on Phabricator tasks and tags as a way of making a case for more resources, but has since abandoned the effort.

We don’t have every type of vulnerability tagged [in Phab]. Not all types fit cleanly. I do have some analytics code that tracks some metrics, haven’t had time to implement it. So there’s no way for me to know what all the tasks look like at a point in time. So I wrote a script that takes a snapshot of all of the security tickets, what tags they have, what their status is. Let’s us chop up the tickets and see what teams have the most tickets.

— Darian Patrick, Security

Reliance on indirect usage measures

Depending on the platform, usage logging may not be automated, and is often incomplete or inconsistent. Teams may then rely on indirect measures of usage to supplement or stand-in for direct measures. Several teams reported that they measure usage of their platforms on an ad-hoc basis through the volume of electronic communications from users--public mailing list activity, bug reports, individual correspondence, surveys.

We know people use our published manuscripts based on citation... For code it’s slightly harder; we’re not requesting a canonical citation to see if our code is being used. There are some tools that can do this, but we haven’t been using them. There are also entire areas of data where we believe it is influential, like WikiStats, but we only have a lower bound of what’s being used. We have had outreach events where we ask people what their needs are; we receive requests on mailing lists; but ultimately we don’t do this in a systematic way. We’ve had some consultations back in 2012: who are we serving, what are their needs, are the platforms meeting those needs? It produced some interesting data, but it wasn’t a comprehensive/representative sample. It was people who we were already working with... We discover via the press about people doing amazing work that we were completely unaware of.

— Dario Taraborelli, Research

Measuring existing audiences

Measuring audiences (individual people or organizations that use our platforms) is not the same as measuring raw usage, although many of the same infrastructural challenges that prevent Technology teams from tracking the most useful measures of usage also make it hard to measure audiences. However there are also additional challenges to measuring the usage of our platforms by particular audiences—for example, many available usage metrics do not provide sufficient metadata to connect usage rates with persistent identities.

Capturing unique users

In many cases, even when a team has good direct measures of overall usage for a platform, we are not able to determine the amount of usage per user. For example, the Research team uses Figshare to host curated research datasets (example). Although Figshare tracks downloads and views for each dataset, it is not possible to determine unique viewers or downloaders.

Discriminating between internal and external users

Teams also have difficulties tracking what proportion of usage comes from internal vs. external audience members. For example, the Security maintains a security review process via Phabricator that can be used for WMF-developed software as well as externally developed MediaWiki Extensions. Users of the security review process are identified by Phabricator username, which may or may not be linked to a Wikimedia project account username or other persistent identity marker. This makes it difficult to determine the degree to which external audiences (example: volunteer Extension developers) are utilizing this service.

Characterizing external audiences

Technology teams often have incomplete knowledge about who their external audiences are. The Scoring Platform team probably has a more complete picture of their external audiences (and even the sub-segments within those audience) than most, because the team relies on personal communication to coordinate with contributors and consumers of the ORES platform. The knowledge they gain from communicating with their audiences directly helps them interpret various usage metrics and determine what kind of user is currently consuming platform resources. For example, the team can interpret cues from their Grafana dashboards related to API request size and cache hit rate to determine whether a user is more likely to be performing an offline analysis of their own model, performing historical analysis of Wikimedia content, or performing real-time monitoring.

The MediaWiki Platform team has a sense of the geographical diversity of their enterprise MediaWiki user audience through download metrics, surveys, and conference participation, but the characteristics and use cases of these audiences are often unknown.

There is a MW stakeholders group that meets once per month, that I participate in. They did a survey a couple years ago, and they looked at downloads. The highest download rate of anywhere was China. Interesting statistic. We have no visibility what they’re doing with it! They have no presence within our conferences or communities. Someone from Korea just joined the monthly meeting, but not clear what his use case is. Some Australians too, but they’re not as active in our monthly meeting.

— Cindy Cicalese, MediaWiki Platform

Increasing adoption among existing audiences

Understanding how well we are serving the needs of our existing audiences is important for prioritizing support levels for our current platforms, as well as guiding us where to focus our future development efforts. Currently, our ability to address the needs of existing audiences is hampered by a lack of good information about who these audiences are, and how they currently use our platforms.

Understanding audience needs

Without more data on who the external audiences for a platform are, it can be difficult to understand what kind of support members of those audiences need in order to engage with the platform more fully, more often, or in a more satisfying and/or productive way.

Understanding audience needs is especially important for audiences whose work is integral to the overall success of the platform. For example, training new ORES models, adapting existing models to new Wikimedia projects (e.g. WikiData, Portugese Wikipedia) and improving the accuracy of existing models requires frequent and extensive deployment of WikiLabels campaigns with experienced members of the target wiki community. However, the personnel and language fluency constraints of the Scoring Platform team results in few opportunities for direct, two-way communication with the Wikimedians who organize and participate in these labeling campaigns. As a result, the team has very little idea of how well the WikiLabels platform is serving the needs of campaign organizers and participants.

For the past few years, the Cloud Services team has used an annual survey to learn the needs of the people who use the Toolforge platform.

Most of the things people talk to us about are problems, but the survey tells us that more people are happy than unhappy, and there is some change over time (increase in happiness). There are also three free-form answer boxes where we ask people around their workflows and pain points. The things that people write into those boxes are really valuable feedback, let you argue for more resources to work on our documentation. The survey taught us that our documentation was substandard, but in a very specific way: The hardest thing was that their first week using the product and understanding what was there and how to use it--they wanted tutorials, not Q&As. [Asking for] new contributor onboarding, e.g. “Here are the steps to build an X”. This kind of feedback helps us narrow the roadmap.

— Bryan Davis

Supporting community health

Representatives from several teams voiced concerns that the communities of users that work with or on their platforms are not self-sustaining, or are at risk of decline. Fostering self-sustaining communities that can provide peer support, coordinate activities independently of WMF, and even engage in outreach is important because WMF will always have personnel constraints. Technology platforms, like all other Wikimedia movement projects, work best when activities such as documentation, Q&A, and evangelism are shared among both volunteers and staff.

The Services team cites lack of capacity as the primary reason that they do not perform more community-facing work, which in turn limits the ability of external users to take full advantage of the platforms they maintain.

Very few people install our services to have the so-called “full Wikipedia experience”. They have a lot of problems maintaining and configuring it, because it’s not straightforward to install/maintain these services at the same time, a lot of people don’t because their either aren’t skilled enough, or it’s too much effort. There are two main reasons: the lack of documentation. Because we work in a lot of areas in the platform we constantly have a lot of things to do, and when your to-do list has more than 5 items, documentation takes a backseat. The second part is because we are a very small team, so we don’t have the capacity to do any kind of community-facing work, like work with the developer community to engage them or start to produce tutorials or have seminars.

— Marko Obrovac

Cloud Services notes that participation in their mailing list, which was once an active site for peer support, has declined in recent years—although the reason for the decline is unclear.

The email list has kind of been in decline for the last couple years. We don’t know why. If you go back in the archives around the time that Toolserver/toolabs migration was happening. A few messages a day asking for help or offering tips. Now that doesn’t seem to happen very often, we just put out broadcasts and maybe some amount of discussion in response to that. It was a community during the migration, but it was more about “I”m stuck bringing this software over, someone help me”, or “here’s how to migrate, I just learned how”.

— Bryan Davis

Increasing adoption among potential audiences

In order to grow the audience for our technology platforms, we need to identify barriers that impede the adoption of our technologies by new audiences. Our ability to identify and take advantage of audience growth opportunities is hampered by lack of capacity for performing outreach activities to reach new audiences, and a lack of clearly articulated growth goals.

Growing audiences within the Wikimedia Movement

Increasing adoption of Wikimedia Technology platforms within the Movement is seen as a growth opportunity primarily because Movement entities (whether individuals, wiki communities or affiliated organizations) are already invested in our general goals and are likely to be familiar with our work (at least in a general sense) and reachable through our primary communication channels—mailing lists, talk pages, Phabricator, IRC, and Movement conferences.

However, there are both technical and social barriers to increasing engagement among these audiences. For the Scoring Platform team, one barrier is a lack of contacts within smaller wiki communities. For this team, one solution would be to increase the ability of regional communities (e.g. Chapters) to take on more of the outreach and support work necessary to get ORES-based functionality implemented in smaller wikis (esp. wikis with few or no English-speaking local technologists).

I don’t have steady communication right now. 1 per wiki for 30 wikis, sometimes more, call it 50. Slowly growing. I’d be surprised if we hit 100. There are only so many wikis that are big enough to have a local technologist. One of the things I’m working on right now with the Latin American Wikipedians is to get them to do a hackathon. That’s a way to grow these tech communities. That would push the ceiling a bit. And if we had an active African tech community. Trying to federate.

— Aaron Halfaker, Scoring Platform

Growing audiences outside the Wikimedia Movement

Many teams maintain platforms that are potentially relevant and useful for audiences outside of the Wikimedia Movement. Perhaps the most obvious case of this is MediaWiki Platform team and the MediaWiki core software and extensions. In this case, one of the challenges has been the historically tense relationship between the WMF and members of the enterprise MediaWiki community, who have at points expressed frustration that their needs were not given more priority. Repairing this relationship can increase usage of MediaWiki outside the Movement. The team has taken steps towards that goal, including hiring personnel from the Enterprise development community, including external audience needs within the annual plan, and increasing staff presence at Enterprise MediaWiki conferences.

For the Research team, encouraging use of their curated datasets (as well as data resources maintained by other teams, such as RESTbase APIs and dumps) among professionals in the field of data storytelling (data journalism, data visualization) can allow them to reach new audiences who may be interested in using our resources in their own research, or even funding our internal activities. Outreach to data science educators—who routinely use public datasets for teaching, contests, and hackathons—can also increase awareness of the extent, quality, and scientific relevance of WMFs public data resources.

There’s a huge potential in data storytelling to bring attention to our priorities. It allows us to reach broader audiences, not just future volunteers and researchers, but donors and others who don’t currently know how Wikimedia projects function... When you talk to people who run programs in social computing, HCI, etc, they will tell you that the most popular data sources that they use are from sites like Kaggle, not from Wikimedia... If we think of our role in Tech as implementing the strategic direction of knowledge as a service and infrastructure for open, getting to the point where all the school are using our services sounds like a reasonable goal.

— Dario Taraborelli, Research

Towards these goals, the Research team has developed a free-standing microsite (research.wikimedia.org) designed to increase awareness of WMF research activities and the research resources that we produce among journalists, donors, and potential research collaborators.