Wikimedia Foundation Annual Plan/Quarterly check-ins/Audiences2 Notes May 2018

Present (in the office): Katherine, Abbey, Nirzar, Marshall, Megan N, Joady, Josh W, Lani, Jaime, Heather, Victoria, James F, Tilman, Toby, Jon K , Danny H, Grace, Josh M

Presentation slides

participating remotely: Michael H. (scribe), Adam B, Alex, Carolyn, Charlotte, Cooltey Feng, Corey, Danny Horn, deb, Dmitry, Eileen Hershenov, Jazmin, Joady, Lisa, Lydia, Maggie Dennis, Margeigh Novotny, MarkTraceur, Michael, Olga, Quim, Ramsey, rho, Stephen N, Toby, Volker E.

Slide 2

• JK: Note also there's a readers metrics meeting on 30 May, this is just highlights of the interventions we've taken.

Slide 3

• JK: red line on the left represents where pageviews had been Jan-march

• Overall, metrics (pageviews) are looking a little better than last year

~8b in March, which is around the higher end of last year

• These represent only pageviews; we'll work on other page interactions in the future, probably June.

Slide 4

• JK: Data is still an issue for us (TN: all of audiences, not just reading), as it has been for many quarters; so it'll be one of two big goals for next FY. Over the next few months we'll figure out what this means specifically. Traditionally, each team had its own approach; we have hired Data Analysts to help us standardize.

• We have traditionally focused on Readers, but are finding that mobile editing is an impediment. We will increasingly focus on mobile editing.

• TN: This fits into our platform view of the world; building platforms to create great experiences for readers as well as mobile editors.

Slide 5

• SEO

• Most of our traffic comes from Google Search; things they do (or don't) impact us greatly. We are working with an SEO consultancy to get a handle on this.

Slide 6

• Marshall: Big goal for coming year. Figuring out how to get a head-start on that work.

• I am talking to as many people/perspectives as I can to get a handle on:
  • What should our framework be for thinking about measurement?
  • What are our needs for data collection?
  • Needs for deliverable creation?

Slide 7

• MM: Two big questions:
• Where do we measure?
• How do we measure?

Slide 8

• MM: We want to have a "garden" of data -- be able to find carrots when we need carrots, etc.

• We have a jungle -- can feel disorganized. (N.B.: Some orgs have a desert—no data!)

• As this chart shows, we currently have the same metrics in multiple sources -- hard to know where to look for what

Slide 9

• Victoria: Shared with Nuria?

• MM: Yes.

• Victoria: Would like to go through it with you?
• Heather: Also Comms.
• Maggie: And CE!

• Josh W: Cross-departmental.

• ACTION: Marshall to present to Victoria, Heather, and Maggie

• TN: That would be an awesome discussion; we have ideas about distribution channels

• KM: Should Maggie be involved

• Maggie: I would like to

• Victoria: My sense was that on the product side, we didn't necessarily have a clear idea of what it was we wanted to measure

• Jon K.: There's areas for improvment throughout

• Victoria: Also complicated is that the customers for this data are not only WMF PMs but the community at large; we have multiple stake holders

• MM: I agree. We have made a lot of good tech decisions, need to contiue to improve

• TN: We need to have a scoping discussion; need to be careful about setting concise goals and figuring out how to get there.

Slide 10

• Olga:

• Page previews:

*Last Q we worked on deploying page previews to english, german Wikipedias.  Goal is to reduce the cost of exploration.  People can interact w/ content from multiple pages w/o opening them all. 
       

• This is probably the biggest change to the desktop experience in a decade, so we wanted to be very careful.

• Analyzed data and published a/b tests; worked with community, comms.

Slide 11

• (page previews examples)

Slide 12

• Now that we're live we can confirm some of our a/b tests results. page previews are used ~50m times/day. If we consider pageviews and page previews together, we can see that people are interacting with about 25% more pages. But note that this includes duplicates; it's counted twice if the user views a preview then opens a page. But we removed dupes in our a/b tests.

Slide 13

• Feedback on the internet was very positive.

Slide 14

• a/b test results:

• - 21% more page interactions despite 3-5% fewer pageviews
• - v. low disable rates (0.01% on enwiki, dewiki)

Slide 15

• We were worried that lower pageviews would negatively affect donation rates, but that didn't happen; in fact page previews may have had a positive impact.

Slide 16

• Worked with comms on well-received blog posts
• the feature got a lot of press

Slide 17

• (press coverage examples)

Slide 18

• Work with communities:
• enwiki community decided not to deploy this 2 years ago. It's good to see that we were able to address their concerns.

• JK: Along with ACTRIAL this is one of the few times we've done an a/b test with the community and saw a big impact
• Olga; We were able to answer community questions immediately because of having data on hand

• Maggie: From a user perspective, this is one of the "best things since sliced bread" features, thank you!

• KM: Yeah, this is awesome.

• TN: It's cool particularly with areas like In the News -- helps with recall

• Between this and compact language links we shipped 2 new things

• KM : People are liking a change on Wikipedia ;)

• Heather: introducing a new design

Slide 19

• PDF rendering: We had to wait until this Q to deploy it, but hoping to hand off to RI this quarter

• Currently the PDF Download button's only on Chrome on Android, and is used ~100-200k times a day; we hope to add to more browsers later

Slide 20

• Olga: Next big project is page issues for the mobile website; these are currently hidden on mobile, which makes it more difficult for readers to gauge quality/reliability compared to desktop. We hope that changing the way we display this will improve transparency and trust.

• Development is planned for this quarter; planning and investigation happened last quarter.

• TN: Community members were not happy that page issues were not shown on mobile, to the point of creating templates that did not result in a good reading experience.

Slide 21

• Alex H: (slide depicts page issue on desktop web)

Slide 22

• (slide depicts page issue link on mobile web)

• Clicking on the grey text (which is a link) displays a modal dialog; but 0 participants in the user testing group noticed this even when prompted to evaluate the reliability of the page.

Slide 23

• We experimented with a few different designs, trying to raise awareness without being too distracting. The slide depicts one of two prototypes we tested.

Slide 24

• (examples of issues)

• They range from issues mainly of interest to editors, to things like neutrality or lack of sources that are of interest to the general reader. One of the broader implications we were thinkign about was the "fake news" question.

Slide 25

• Our goals:

   1. Make the readers more aware of page issues
   2. Evaluate how awareness affects readers' judgments about page quality/reliability

• Results: We saw higher awareness of page issues, and readers appreciated them enthusiastically. (Note that the sample size is small so this wasn't statistically significant; but it's much better than zero awareness.) We found that they increased *trust; they may encourage editing but this needs more investigation.

• (Tilman: )

• By exposing these, we are increasing awareness that Wikipedia is created, monitored, etc. by a community.

• See link on this slide to read the full results.

Slide 26

• NP: We were also able to improve our settings greatly last Q.

Slide 27

• Main goals were around transparency of benefits around opting into beta, and allowing changing text size. Our goal is to increase WP Beta participants.

• JK: Huge issue: beta opt-in users not reflective of the general population. Things that look great in beta don't always turn out that way.

Slide 28

• CG: Last Q, we meant to pilot an awareness campaign in Nigeria; this was delayed due to an issue with our partner.

• We are experimenting now with localized store descriptions and app store assets. We don't have statistically significant results yet but hope to share at next QCI. We want to figure out what features are most important to users in various markets and be able to tailor app store descriptions to them.

Slide 29

• We are delighted to announce that synced reading lists are released! (i.e.: adding to a list on one device will make the change available on all other devices). This was a long-standing request and users are quite happy about it.

• Next quarter we want to improve the experirence for multilingual users: explore feed, search

• We'll be doing user research on this at the hackathon and will have results next quarter.

Slide 30

• (depictions of reading list syncing)

Slide 31

• (representative user feedback examples)

• KM: What % of users are using this feature?
• CG: see below

Slide 32

• CG: The feature has only been out 2 weeks, so too early to see if it drives repeat usage; but slightly over half of people who creating reading lists enabled syncing. 108k users have created lists. This number seems smallish so we're going to run an *awareness campaign this quarter. We've also seen a spike in account creation, now tapering off (currently ~1k new accounts/day after a peak of 1.4k/day).

• (Tilman: just a note that A/B testing such design changes is still a bit of a technical challenge in our technical context (considering our privacy constraints and current lack of a general framework for variant testing); the web team is putting some work into this instrumentation currently. )

Slide 33

• JM:

• (pics of reading lists on iOS)

• iOS and Android released synced reading lists simultaneously.

Slide 34

• This feature is part of our 2-yr arc of the 'better encyclopedia experience' work. One thing we're interested to see is how this affects 'rabbit-holing' across sessions. We want to know how to support this in a mobile-context, where you don't necessarily have the option of opening a bunch of tabs.

Slide 35

• Next Q we'll work on some of the customizability that Android did last Q. This is in keeping with our 'portfolio iteration' strategy wherein the apps alternate developing features so that we can learn from each other.

Slide 36

• Android did a gradual release and so did iOS. Graph shows usage: blue lines are additions to lists and yellow is checks for updates.

• (usage stats since release)

Slide 37

• JM: RI is focused on consolidating code that has traditionally been distributed among multiple sources despite identical functions, duplicating maintenane burden.

• Examples:
  • page library
  • page content service
  • reading lists service

Slide 38

• A benefit of this approach is that it allows us to adopt new tech across multiple platforms quickly. For instance, we were able to use the existing API used in the apps to quickly create a browser extension.

• TN: The upload to commons app developed by the community will begin using our APIs as well.

Slide 40

• JM: Similariy, we were able to use a shared API backend for page previews on the web and link previews in the apps.

Slide 41

• Benefits.

Slide 42

• Ramsey: 3D is here! Launched end of Feb. We have a few hundred files uploaded so far; we're working with a project called Scan the World who are uploading more content. Curators and editors are excited, and are using the files on-wiki.

Slide 43

• Last Q our goal was to support 3D which we did; this Q we'll be working to fix bugs and ?? STL format.

Slide 44

• (examples of on-wiki file usage)

• Note: one example is a wikidata page; what's going on there?

Slide 45

• A: '3D model' is now a Wikidata property

Slide 46

• Partnership opportunities: Smithsonian reached out, and wants to tie our APIs together; talking with Oculus (Facebook) and Magic Leap about supporting new formats and educations usage; and Shapeways wants to donate a portion of the price of 3D files to Wikipedia.

Slide 48

• Structured Data on Commons: We're taking a "crawl, walk, run" approach. Last Q prototyped the most basic possible feature; all of our goals were reached. We prototyped changes to uploadwizard, file page, and ??

Slide 49

• Screenshot of the new commons file page combining wikidata with wikitext; this was one of the trickier prototypes. Thanks to Mark for working this out from a starting point of little Wikidata knowledge.

Slide 50

• Prototyped changes to UploadWizard which are still a work in progress. Working with the community. This is a way easier change than the File page.

Slide 51

• JK: Lessons learned from this period—main themes:
  • coordination is something we're getting better at but still need to work on

Slide 52

• Grace: Added "technical" to distinguish from other work.

Slide 53

• Grace: We focused on delivering customer value; the product manager defines customer value and program manager makes it happen

• We work behind the scenes to ensure progress

Slide 54

• I hired Lani on 1 March and that's going great.

• Hired Jazmin on 7 May.

• Max works on Multimedia team

• Max and Jazmin are going to Barcelona for offsites/Hackathon

Slide 55

• I am partnering with Dan Garry on TemplateStyles; Dan decided not to target ruwiki after getting pushback, we deployed to Wikivoyage, then dewiki, ruwiki, and others wanted it

• Getting some traction on Scrum of Scrums

• Finance asked me to develop an approach for financial decisionmaking during C-Team meetings

• Bringing SoS accountability model to Audiences

• Improving transparency (e.g., decisions about conference attendance)

Slide 56

• Margeigh: Hi from New York.

Slide 57

• I'll start by talking about my first Q and some of my goals around transparent team practices - how research and strategy fits within audiences.

• How do we gather and capture the questions that are coming up on different platforms as they come up?

• How do we collaborate with other teams?

• How can we standardize how research and design work together?

Slide 58

• (depicts the "design research pipeline")

• We're coming together as a team to decide which questions are highest-impact and most urgent to answer. Figuring out how to cluster Qs and who gets resources.

Slide 59

• Another thing I've been tackling is: what is this function and its purview? What kind of activities and outputs need to get generated to inform product strategy? Can encompass anything from explaining on the one hand to running experiments on the other. Looks like a lot of stuff but we have to be partnering with design, analytics, research, comms.

• VC: I"m sure CE is on this list as well.
• ACTION: MN to add CE to upper left quadrant

• MN: Yes, see upper left quadrant.

Slide 60

• MN: Nirzar and I have been trying to standardize titles and expectations/accountability for each title. It's a work in progress but has been shared with T&C.

Slide 61

• Another thing I'm working on is ensuring that Audiences aligns on a 3-5 year plan. Having a series of work sessions to connect movement strategy and the more tactical 3-5 year strategy. I think we'll be involving other teams but that will be scheduled in coming weeks.

• After having participated in the movement strategy track in WMCON, it's important that product people participate in the working groups.

• TN: I asked MN about how product could interact with the movement strategy and this plan is an outcome of that.

Slide 62

• Bigger picture for our function is to provide better data to support product decisions. I'm in NYC today to work on mobile user personas. The product teams want these in order to decide what audience segments to focus on.

• I'm about to start a project with Neil trying to provide a more nuanced understanding of how the different Wikimedia projects differ from one another. What are their distinctive characteristics and what to they have in common? How old are they? How many users do they have? How many pages, pageviews? How many admins?

• We have ~30 different such parameters. The point is to be able to cluster different projects so that we can start to track more nuanced metrics. Rather than trying to move the needle on all at once, we want to enable a finer-grained focus.

Slide 63

• Abbey: Framework of human-centred design we use to pick out what we're working towards.

Slide 64

• Here's where some of our projects fall on this framework.

• We're starting to work on in-context help w/r/t new editors. Doing mobile personas for iOS and for Android in India.

Slide 65

• On Wikipedia workflows, I've worked very closely with James F to gather all of the possible workflows in order to create a contribution taxonomy. In the end this will be made public for anyone to use.

Slide 66

• (the high-level framework)

Slide 67

• (zoom-in on some of the 88 workflows identified)

• TN: These are the proecesses by which content is created on the Wikipedias

• AR: Yes

Slide 68

• Scoring and weighting

• Red criteria are the abilities an individual needs to be able to complete the workflow
• Blue depicts opportunities for mobile

• Also looking at how much knowledge, social capital, etc. a user needs to complete various workflows

• There's also weighting so that teams can decide what workflows are most important to focus on

• TN: To draw the arc back to Jon's point, this is how we're going to assess to what extent mobile editing workflows can be supported on mobile

Slide 69

• (ex: workflow for copyvio detection)

Slide 71

• These will be publicly available for anyone to use. WMF teams are already using it.

Slide 74

• Lydia (WMDE, product manager for wikidata): Let's talk about Wikidata.

Slide 76

• We had two large annual goals:
  • Increasing number of volunteers who benefit from Wikidata
  • We are on track in usage; a bit behind in terms of data quality.

Slide 77

• . Making wikidata more useful for other open-data projects. Getting more people to install Wikibase outside Wikimedia. This is happening. We're a bit behind on success stories; this will just take time as people blog, etc.

Slide 78

• Goal: Wikdata use in the Wikimedia projects. Projects including English WP had questions about to what extent recent changes are shown. We've improved this by showing 2/3 less changes, consisting of the most relevant changes.

Slide 79

• Graphs show lowering load on the Job Queue.

Slide 80

• Goal: data quality

• We've been working on this for a long time. Last Q we rolled out constraint checks (rule-based checks for common mistakes).

Slide 81

• (depicts a dashboard providing for quick revision of vandalism)

Slide 82

• (depicts a constraint report) -- showing things that might be wrong/indicate mistakes in how the data should be modeled

Slide 83

• positive feedback.

Slide 84

• Goal: support for lexicographical data. We've had lots of discussions around roll-out and licensing with the community.

Slide 85

• Goal: wikibase adoption outside Wikimedia

• We're working on a Docker container to ease installation

Slide 86

• Goal: data partnerships

• We've had talks with various orgs, seen some good blog posts about their use

Slide 87

• For Wikidata query service, we made some small improvements to make it more responsive and so on. A GSoC student will make further improvements this summer. Usage is picking up.

Slide 88

• People are developing new tools to make it easier to add descriptions, make it easier for third parties to use, lots of other cool stuff. WDCM Journal—highlights interesting things we found while analyzing how Wikimedia projects are using

Wikidata's data.

• We got our own DB servers, yay, thanks SREs!

• A lot of work was done on search integration; thanks to the search team.

Slide 90

• Showcase: Wikidata-powered infoboxes on Commons

Slide 91

• (interesting example of WD data)

Slide 92

• (interesting example of WD data)

Slide 93

• (example of WD data)

Slide 94

• Example of an actionable use of Wikidata: providing statistics about the gender gap in the Wikimedia projects. Can see the global gender gap (in article subjects);

Slide 95

• ... or can see this broken down by project!

• TN: I asked Lydia for example of something actionable and this was a good one. We are working with Research on contributor diversity but hadn't seen data on content diversity; this is a good last demo.

• Tilman on chat: http://wigi.wmflabs.org/ ("Wikidata Human Gender Indicators") has already existed for a while

Slide 97

• Scorecards for ongoing work (showing metrics about usage/coverage and changes therin over time)

• Note: There's only so much we can do in terms of telling people what kind of data they can put into Wikidata.

• TN: That turns around a trend/is an uptick, right?

• Lydia: Yes.