Wikimedia Foundation/Communications/Audience research/Research and notes
| This page from the Wikimedia Foundation is kept for historical interest. Any plans, policies, projects, or strategies mentioned may be obsolete. |  |
Audience research: Research and notes
On this page, you’ll find updates from the Audience Research team about their ongoing work and findings.
Research updates
Wikimedia Audience Research Project: Internal Workshops
In October 2016, the Wikimedia Foundation Audience Development team (including Melody Kramer, Juliet Barbara, and Abbey Ripstra) completed five internal Audience Definition Workshops (two cross-functional, and one each with the Reading, Discovery, and Editing teams). The goal of these workshops was to understand: 1) Wikimedia Foundation teams’ current perception of their audiences; 2) Wikimedia Foundation’s current approach to audience development; and 3) gaps in Wikimedia Foundation’s understanding or servicing of audiences that prevent the Foundation from achieving its mission.
Cross-functional participants attended from the follow teams: Office of the Executive Director, Product, Team Practices, Research, Languages, Community Engagement, Learning & Evaluation, Programs, Support and Safety, Community Liaisons, Advancement, Legal and Communications.
Participants were led through the following activities
Who are your current audiences?
Introduction: Why we’re here today and the goals for the workshop. This will be used to develop a framework for the foundation to develop a common understanding of audiences, determine how to segment / research those audiences, and what audiences would be the most impactful to target with research / new products / tools.
Purpose: Get people comfortable and thinking about audiences.
Activity: We ask everyone to say their name, team, and the one thing they’d most like to learn about their current audience(s), and what they could do if they knew that information.
Workflow mapping
We map the ways we currently apply audience understanding internally
Purpose: To develop understanding of the way groups or teams at Wikimedia Foundation work by visualizing how work gets introduced and processed; raise awareness of what’s missing in the workflow or pain points in the workflow; helps different subgroups within a group understand more about the work of others.
1) We ask: Recall a time when you had to design something (a strategy, a product, etc) and you had no idea where to start. What information were you missing? Was there something you needed to know about your user that you didn't have access to?
We go around the room and listen to others. Possible additional questions:
- Start at the beginning. How did you start this process?
- Who was involved?
- How did you guys decide on [this program / feature / product fix]?
- Then what? How did you go about it from there?
- How did you decide what to do?
- Were there any changes in direction / pivots? Why?
Defining Audiences: Who are the current audiences we serve?
Purpose: To develop understanding of current audiences that Wikimedia Foundation teams serve, and to determine what we know about those audiences.
Activity: A brainstorming exercise. Everyone is asked “Who do we serve?” And for 5 minutes, we write down as many audiences we can think of individually, and then share as a group.
After we list as many audiences we can think of, we start with some follow up questions. For 5 minutes, we dive deeper:
- Where do they live?
- How old are they?
- What do they care about?
- Are they end-users?
- If not, are they intermediaries?
- What's their relationship to Wikipedia / Wikimedia Foundation?
- What do they use Wikipedia for (or engage with Wikimedia Foundation for)?
- How has their relationship with Wikipedia / Wikimedia Foundation changed over time?
- Last question: How do we know these things about our current audiences?
This will
- Separate what is hunch/assumption-based, and what is evidence-based (and help people see that)
- Surface current sources of information about audiences (and what we will never know, based on our privacy policy)
Who are your potential audiences?
Who could we serve: Who is not in Wikipedia’s audience right now?
Purpose: To get people thinking about potential audiences and learn what they’d like to know about them. This will help us formulate recommendations.
Activity: We lead a group discussion looking at the questions:
- Who is not in our audience right now?
- What are your biggest questions about who we could serve?
- If you learned the answer to those questions, what would you do with that information?
Who could we serve?
Who could we serve: What are the barriers that exist that prevent people from accessing our projects? What are the biggest opportunities? What are the biggest threats?
Purpose: To get people thinking about potential audiences and ways we may not be reaching them. Learning what we know and don’t know about our potential audiences. What we might do if we had that information.
Activity: For 5-10 minutes, we write down all of the barriers that might prevent people from accessing our stuff. (Structural, technological, cultural, design, etc.). We then use this to talk about the biggest opportunities.
Wikimedia Audience Research Project: Internal Synthesis Workshop Summary
The Wikimedia Foundation Audience Development team and Reboot met in San Francisco from November 7th-8th, for this Synthesis Workshop to translate the findings from these workshops into a framework to guide audience research and engagement throughout Wikimedia Foundation.
On Day 1, we started by reviewing our goals for the session. We then discussed how a framework and resulting tools will be used, which resulted in our alignment on project goals and identification of key stakeholders, use cases, and near term opportunities for the framework. After summarizing our design direction and parameters, we then looked at the multiple ways Wikimedia Foundation staff understand and/or describe their audiences based on what we heard in the Audience Definition Workshops. We then reviewed frameworks from other organizations to look at what makes them successful. Using the different organizing principles for understanding audiences and the success criteria of sample frameworks, we ended the day by talking about what we can map and which frameworks would be most valuable for the goals of this project.
On the second day, we began prototyping different frameworks (mainly the Map of Audience Understanding and the 2x2 Framework of Audiences & Stakeholders). During these sessions we also noted more open questions, action items, observations, and potential recommendations for an audience research strategy. At the end of prototyping, we began to brainstorm criteria for how we will prioritize different streams of audience research for the first round of generative research. We ended the day by outlining a forward workplan for this project, as well as the roles and responsibilities of the team.
Summary of Discussions
We started the two-day workshop by aligning on project goals (using our previously established Top Priorities), potential use cases for this work, timeline, and available resources.
What we're doing and why
This project seeks to:
- Develop an integrated, holistic portrait of our ecosystem of audiences and stakeholders (user, intermediary, and internal audiences) and how each audience currently informs or impacts our work in order to better serve our users.
- Organize understanding of our audiences based on their needs, incentives, capacities, and constraints that influence their engagement with Wikipedia, rather than based on our organizational structure.
- Inform prioritization of generative research to ground our audience portrait and understanding in empirical data. This will begin with a set of prioritized audiences, and we will support other teams at Wikimedia Foundation to conduct their own audience research.
This exercise will help Wikimedia Foundation set product strategy based on a robust and holistic understanding of our diverse audiences and stakeholders. Specifically, it will also help:
- Wikimedia Foundation align programmatic and product strategy, by helping teams see how their activities can be complementary and/or mutually reinforcing across the user experience
- Wikimedia Foundation prioritize our activities and allocate limited resources, based on a strong understand of opportunities and challenges to different audience’s engagement with Wikipedia
- Develop a user-oriented, evidence-driven culture across the organization, by helping organize past research, setting direction on forward research, and helping different teams see where and how they can contribute and/or benefit
- The movement understand what the Foundation is doing and why, and align Wikimedia Foundation and community around a common set of opportunities and challenges to impact
How we describe our audiences
Each member of the group represented one Wikimedia Foundation group in the workshops in sharing back what their group said in the workshops, by preparing post-it notes with their findings ahead of the workshop. We then, as a group, categorized them according to different organizing principles.
We thought about ways we could organize our descriptions and understanding of audiences under the following categories:
- Demographics, including: geography; level of education; occupation; language; level of English; and whether or not they are part of the Wikimedia Foundation Community.
- Affiliation (individual to organization), including: journalists; donors; future employees; Wikimedia Foundation Staff; Wikimedia Foundation Executive Team; Wikimedia Foundation Board; government bureaucrats; and anyone who searches on Google.
- Expertise, including their level of technical expertise and comfort with reading.
- Platform (offline to online), specifically mobile.
- Motivations, including: brand affinity and interest in product; motivations of different types of users (readers, editors); motivation to come to the site and get “addicted”; topics of interest; those who might mentor others to use Wikipedia; and how they feel valuable in contributing.
- How we serve them, including: how well we’re serving them; their information needs; and their level of trust of Wikipedia
- Depth of engagement ( their relationship with WP, from passive to active), including: whether or not using Wikipedia; how they come into contact with the product; how they contribute to Wikipedia (donate, edit, read, share, code, monitor, etc.); how long they have been contributing to Wikipedia in their role; those who consume the product of a tool; and those who produce a tool.
- How vocal they are, including: “the less squeaky wheels”; part of the community that gets heard the most; handful of power users; and the millions who use products but don’t tell us much (silent majority).
Open questions that came up for the team
- How do we show how people move along a curve of engagement or relationship with Wikipedia, especially with no necessary entrypoint or endpoint for users?
- Is this the right language we should be using to categorize? When using Wikimedia Foundation organizational language (e.g., read, edit) it limits how we think about our product
- How do we distinguish contributors v. consumers?
- How do we measure our mission accomplished?
Framework success criteria
We used examples of other audience frameworks to define what would work best for Wikimedia Foundation.
| Example Framework | What we liked |
|---|---|
| BBC Global Audience Segmentation | Layered profiles, high-level view that can be divided into specific groups |
| Mozilla Web Literacy | Interactive, With many different groups there are actions & ways to interact, Could be a familiar framework in terms of actions that may be useful in our context, Can look at how we organize our skills, and where we need to focus |
| 2x2 | Can create personas for stakeholders along spectrum (e.g., institutional, passive), Create benchmarks for people across team / movement |
| User Personas | Have set of sliders for different personas to organize criteria, Include set of tools for how to understand these personas, Can create benchmarks to improve on / measure against |
| Entities, relationships, attributes, & flows | Zoom out to see whole picture, zoom in to see attributes of different entities; On web - info box that gives you key information about different entities (attributes) |
| Semantic Pathway | Can show pathway for users and where different audiences fall along that line, See what you know about a certain user group / group of people on a scale, then step back and compare them; Allows you to look at a lot of different factors; Size could be an attribute |
Framework planning
Phase 1: Pre Generative Research Frameworks
Using the above examples and our organizing principles, we outlined how different types of frameworks could be used to meet the goals of this project in pre and post generative research:
[0] Universe of All Wikimedia Projects
What: A map of all Wikimedia projects and how they relate to one another, including caveats
Why: To explain initial scope and focus on encyclopedia (Wikipedia)
[1] Map of Audience Understanding Today
What: A visualization of all Wikipedia audiences and stakeholders—both those surfaced through team workshops, and additional ones that the team has identified—based on each group’s:
- approximate size
- existing channels of knowledge about each audience
- relationship to other audiences.
Why
This will serve as a stakeholder engagement tool, by:
- reflecting back what we have learned through the workshops to date
- enabling conversations around the project and why it is needed, in part by visualizing the disconnect between size of different audiences and volume of knowledge received about each
- clearly identifying our knowledge gaps
[2] Framework of Audiences & Stakeholders, and supporting tools
What: 2x2 matrix showing the range of audiences that we do and could serve, building on the groups identified in Stage 1.
Audiences will be mapped against:
- characteristic that is intrinsic to the group/profile that Wikimedia Foundation cannot control, but which may really impact how the movement engages with them (e.g. institutional vs individual)
- characteristic that denotes a relationship with Wikipedia that impacts how they currently engage with the movement, and that the movement can impact (e.g. depth of relationship with the movement)
Through visual design, we will also be able to show:
- The size of each audience
- The category of each audience,e g. end-user, intermediary, and/or internal stakeholder
- Other characteristics deemed important
Why these tools are important
These set of complementary tools will enable us to have conversations about Wikimedia Foundation’s existing knowledge about audiences and its gaps, and to engage teams across the organization in ongoing dialogue and work about why addressing gaps are important—and how to do so.
The framework will help provide common language and organizing principles to understand each of Wikimedia Foundation’s audiences and stakeholders, how they relate to each other, and therefore how we need to work together
Possible post-generative research frameworks
[3] User Personas
What: Representation of the goals and behavior of an audience group, synthesized from data collected through the generative research
- Includes more details on audience attributes
- Can be connected back to the profiles in the 2x2 frameworks
Why: Make tangible audience needs and attributes for strategy, product, and program development; Create benchmarks to motivate teams to improve on; Inform and connect back to the 2x2 Framework of Audiences & Stakeholders
Framework prototyping
Map of Audience Understanding Today
We began to develop the Map of Audience Understanding Today by listing out all of the audiences mentioned in the workshops and then categorizing them into audience groups.
Through this exercise, we established the following groupings of audiences:
- Future Employees
- Movement Stakeholders
- Partner Organizations (Government, Publishing, Research, Technology, and Education)
- Consumers (Technology and Content)
- Influencers
- Contributors (Governance, Technology, Content, and Donors)
We had difficulty categorizing audience groups that can be cross-cutting (such as students, educators, researchers, historians, etc.) and those audiences that we are not reaching (potential users and people not yet online).
Framework of Audiences & Stakeholders (2x2)
We experimented with several ways we could map different audience groups on two axes. With the Y-axis being Individual to Institutional, we identified potential X-axis scales as:
- Relationship to Wikimedia
- Depth of engagement
- Consume to contribute
- Level of awareness
As we explored these different 2x2 matrices, we began to develop broader audience groupings that matched their different profiles in the matrix:
Knowledge Stewards (Active volunteers):
- Text editors
- program volunteers
- Bureaucrats
- Tech volunteers
- Photo contributors
- Translators
Wikimedia Newcomers:
- Brand new editors
- Attending first Wikimedia event
- New donors
Institutional Knowledge Stewards
- GLAM
- Educational organizations
- Partner research institutions
- Research funders (National Institutes of Health)
Distributors
- Wikipedia Zero partners (telecoms)
- Technology partners
- Wikipedia Library partners
Infrastructure Stewards (Plumbers)
- Wikimedia Foundation staff
- Wikimedia movement affiliates
- Wikimedia Foundation Board
- Committees (Funds Dissemination Committee, Affiliations Committee)
Connected Global Influencers
- Journalists
- Policymakers
- Bloggers
Rule Makers (Influencers)
- Internet regulators
- Governments
Curious Information Seekers (Proactive Learners)
- Rabbit hollers
- Deep learners
Casual Learners
- Fact finders
- Context finders
Unknowing Learners
- Syndication readers (knowledge graph, Siri)
- Googlers
Passive Distributors (Content Distributors)
- Publishers
- Content syndicators
Money Contributors
- Individual donors
- Major donors
- Endowment contributors
People We’re Not Reaching
- People not yet online (~4 Billion)
- Potential users (readers / editors)
- Readers in censored environment
- People with disabilities
- Readers with special needs
Outliers:
- Vandals
- Trolls
- API consumers
- Grantmakers - Wikimedia
- Organizations using MediaWiki
- Researchers of Wikipedia
- Kiwix readers (offline)
With these broader audience categories, we saw potential in how we can support audience groups that are closer to consume to move them towards contribute by using generative research to answer questions such as: “What are the motivations or characteristics of these Knowledge Stewards that we can instill in Curious Learners to empower them to contribute more?”
Other attributes that could be mapped on a 2x2
- Level of knowledge about group
- Degree of engagement with Wikimedia Foundation
- Estimated Size
- May be difficult because not apples to apples
- But would be useful for Individual range, especially for product teams
- Could also type it in and not scale
- Geography
Criteria for Research Prioritization
How do we prioritize audiences for further research? Some ideas:
- Timeline of strategy decision-making
- Availability of relevant research
- Resources for research (budget; if ongoing research project that might fit immediate needs)
- Staff bandwidth to partner and act on research
- Team enthusiasm
- Team willingness and room to put outcomes in roadmap
Open questions
Needs to be answered immediately:
How do we receive input? What kinds and how often?
For consideration throughout project:
- Who have we lost & why? How do we visualize or map them?
- How do we map / incorporate audiences were not serving and those not revealed in the research thus far?
- How do we frame the acquisition pathway when no necessary endpoint for users? (mission accomplished v. what we want them to do)
- How much does this serve growing audiences v. serving them better?
- Where do bots fit in?
- How could this affect how we do analytics?
Findings
Findings coming soon.