Wikimedia monthly activities meetings/Quarterly reviews/Mobile/January 2015

The following are notes from the Quarterly Review meeting with the Wikimedia Foundation's Mobile Web and Apps teams, January 29, 2015, 14:30 - 16:00 PST.

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Present: Bahodir Mansurov, Bernd Sitzmann, Leila Zia, Carolynne Schloeder, Adele Vrana, Jon Robson, Daisy Chen, Sam Smith, Kaity Hammerstein, Ryan Kaldari, Rob Moen, Kevin Dubuc, Brian Gerstle, Corey Floyd, Monte Hurd, Erik Moeller, Tomasz Finc, Dario Taraborelli, Dan Garry, Maryana Pinchuk, Jon Katz, Tilman Bayer (taking minutes), Moiz Syed, Terence Gilbey, Damon Sicore, Howie Fung, Adam Baso, Vibha Bamba, Dmitry Brant, Max Semenik, Toby Negrin, Kevin Leduc, Joaquin Hernandez

Participating remotely:

Presentation slides from the meeting

Welcome, team intro

Maryana:
will cheat a bit and cover some of January too
hiring, welcome new members
Brian, Corey, Adam

slide 2

[slide 2]

What we said
Apps: focused on improving reader experience
we are behind industry in app usage
enable readers to browse quickly, get to facts they want
Mobile Web: piloted radically new contribution mechanism
big q: will all (most/more) readers be able to contribute?

Apps

Dan:

slide 4

[slide 4]

(What we did)
search on Android app before and now:
added Wikidata description to search results
also, new search backend helped
Lila: what about empty descriptions, are we working with community on populating them?
Dan: not explicitly, no
notified Wikidata development team of course
Jared: worked on introducing hard (length) limits on these descriptions so that they fit in UI
Lila: but we could stimulate community activity on this, working with community team
Erik: with its staff of like 4... :(

slide 5

[slide 5]

Dan: What we delivered: search

slide 6

[slide 6]

(What we learned)
reduced "no results" ratio, which is great
but clickthrough rate went down
Damon: sample size?
Dan: about a million searches, see appendix
Lila: is there a "see more" option at end of search?
Dan: can keep clicking, yes
Lila: but no search options? no

slide 7

[slide 7]

slide 8

[slide 8]

lead image
absolutely transformative for UX
make sure first sentence is above the fold
e.g. first page of Berne article didn't actually say it's in Switzerland
Erik: i18n for this?
Dan: works in all languages
Lila: this is great
from my experience as reader though, there aren't enough good images
sometimes just a logo
Dan: did some community outreach on this
tell editors how to swap out suboptimal images
Erik: when scaling this from app to desktop, such issues (useless/wrong/offensive image selected as lead) will become more severe
Dan: TheDJ pointed that out at Dev Summit (create awareness among editors that top image selection matters more now)
Lila: option to see entire Wikidata item beyond description? no
Dan: doing face detection, so that e.g. not just Obama's tie visible
that would be more difficult on web (using built-in face detection in Android now)
Monte: but can use data we gained from app for that now
Dario: how about using Wikidata image?
Dan: not possible right now

slide 9

[slide 9]

users love lead image feature

slide 10

[slide 10]

worked on an image viewer
pulls same data as Media Viewer
Lila: how many hours did that feature take? ;)

slide 11

[slide 11]

Dan: lead image feature meant we could no longer rely on file description page
worked with Luis from Legal to get attribution/licensing data right
answer: just three days for a single engineer :)
Tomasz: because there was an API
we got lucky, but this success story is an important takeaway

slide 12

[slide 12]

Dan: most users only view one image in gallery, but some view lots
Jared: counting views in image viewer as mobile page views?
Toby: ...yes.. working on such things
RobLa; for MV we needed to add that specifically
Lila: give feedback to services team about such things
Monte: have a Phabricator entry about our dream API that would give us all in single request (article data but also image metadata)

slide 13

[slide 13]

Dan: three "read more" suggestions at end, driven by "deep learning" use case

slide 14

[slide 14]

naively generated suggestions
[slide 15]What we learned:
25% of our installs reach end of article at least once
that's a really interesting result about long form reading
Lila: they didn't necessarily read the whole article inbwetween though
are you measuring session time?
that should be the success metric
if this is driving it, you won
Dario: ...
Erik: keep in mind this is the most basic version imaginable for this feature
could imagine more immersed UX, like slider
Dan: and this has only been up for a little while, could look at many other aspects
Lila: not asking for more metrics
Toby: yes, drive that one success metric
Dan: cannibalizing blue links? open q, can test that

Mobile web

Maryana:

slide 17

[slide 17]

one motivation for WikiGrok:
the community adding information on Wikidata is tiny
Lila: how does DBPedia compare to other structured data?
Dario: Knowledge Graph is two orders of magnitude larger in terms of items[?]

slide 18

[slide 18]

Maryana: WikiGrok is one possible solution to that

slide 19

[slide 19]

tested two versions
right one is a bit more complex

slide 20

[slide 20]

minimal, no frills (badges etc.), wanted to get it out quickly

slide 21

[slide 21]

will readers engage even without rewards?

slide 22

[slide 22]

engagement is high
3x as many wikigrokers than mobile editors in that period
Lila: apples to oranges (prompting users directly)
percentage out of total uniques?
Leila: 6% of those who saw it interacted
Lila: Toby, funnels would make this much easier to understand
Toby: OK

slide 23

[slide 23]

Maryana: quality is high
better than mobile editors: 70% good, 30% test edits/vandalism
Lila: did we target particular type of content?
Maryana: yes, actor bios, and studio albums
quality from readers was higher than for logged-in users(!)
Leila: caveat about data
Moiz: quality isn't that important because this will get aggregated
Lila: keep in mind that current efforts by FB and Twitter about flagging content definitely get gamed
Toby: different layouts had impact
Kaldari: TV/film actors, version B has "none of these", can also choose both
Lila: tried different language, e.g. framing it as game? not yet

slide 24

[slide 24]

(Maryana:)
Other findings:
Even without any gamification, 26% respond again
Brian[?]: what about when it's difficult to tell difference between studio and live album?
Maryana: yes, can be an issue
50% of users never saw the widget
current questions are basic, could probably answered by machine
more sophisticated/higher value: suggest possible occupations that can be extracted from article automatically, but are hard to decide automatically

slide 25

[slide 25]

user reactions
users didn't like quiz aspect (being challenged), in particular women
Lila: breakdown happy/neutral/bad?
Maryana: sample too small, this was in-person testing on the streets
Toby: this is significant, unlocking new contribution mechanism that hadn't been thought of before

What held us back

slide 26

[slide 26]

Maryana: not sending Wikigrok data to Wikidata yet; held back by infrastructure
reset expectation of what Wikidata could do
conflicting tests from Fundraising
Lila: heard that from other teams too
ask: what is fundamental so we need to do ourselves, and what can we buy elsewhere?
Leila: had to restrict A/B testing to 16% of users because of infrastructure capacity limits
Brian[?]: how is beefing this up going to affect shared hosting? (others:) let's take this offline
Lila: how many engineers do you have?
Dan: I went from 1.5 to 4 engineers, still complaining ;)
improvements benefit everyone: infrastructure like API, search is reusable, also outside WMF

Other trends and metrics

slide 28

[slide 28]

Maryana: topline trends
not a lot of news since December readership update
overall trend flat
Lila: in US, decrease, not offset by mobile growth
Toby: Beyond (search referrals), engagement is important too
Erik: not counting app traffic in these yet
Lila: think about session length
Dario: it's because we don't have notion of unique client yet, working on that

slide 29

[slide 29]

Dan: ...except for apps, where we do have session lengths
medians change more than means[?]
Android still biggest, 70/30 with iOS
Lila: what is the second biggest search term after "Wikipedia" by which people find the app?
Dan: should ask Google

slide 30

[slide 30]

third-party apps
many of these scrape instead of using API

slide 31

[slide 31]

Endless

slide 32

[slide 32]

Wikiwand

slide 33

[slide 33]

Maryana: wider trends:
iOS, Facebook search
I think we need to enable users to (better) search in our app
Lila: hearing that especially in non-Eng languages, peer to peer communication is important
and startups doing work there
Maryana: in many emerging markets, chat is most important
Erik: making connections with other people is important
single focus on search doesn't work any more
Tomasz: but should also not spread ourselves thin across these areas

slide 34

[slide 34]

search for "next billion"
working with Zero team on this
a lot of our work has been very Global North focused
Toby: setting up group with Zero team, marketing
Dan: don't know anything about app market in China, how people get our app there
e.g. Facebook has an app for some emerging market that has a much smaller memory footprint than their main app
Lila: how important is Offline?
Dan: FB doesn't have any, our app has that function
Carolynne: ...

slide 35

[slide 35]

What's next

Maryana: next 3 months
"shape": entirely new tool to create subjective content
Lila: these are experiments, good to do them and see which one will hit it out of ballpark
but need to define success
without that, I'm worried about churning
Toby: need to do some pencil sharpening about that
Erik: new contribution streams are much trickier
might find out machine learning gives better results than human microcontributions, and put this on hold
Toby: but we should decouple Wikidata from mobile microcontributions
users want to work on Wikidata anyway
Leila: also, seeing that we can do many different things with WikiGrok, e.g. also translations
Toby: reaching out to Duolingo

slide 36

[slide 36]

Dan: more value to apps readers
e.g. tweet a fact
Erik: see this subjective sharing etc. as natural extension of "objective" core of WP

slide 37

[slide 37]

Maryana: WikiGrok at full scale
Lila: but not for 100% of views? no
Maryana: never engaged at size of 100K editors before
Toby: is this (100K) a goal for this q?
Maryana: no, can't make that prediction yet
Lila: should test neither for 100% of articles nor for 100% of users
Toby: only just getting data and analysis, still a lot to be done
Kaldari: could do 100% of users, but with only 100s of articles
Lila: just want to be careful about setting perceptions [?] for users on brand

slide 38

[slide 38]

Maryana: could campaigns self-generate?

Collections

slide 39

[slide 39]

Jon K: Collections
still in baby stage

slide 40

[slide 40]

for entry point, co-opting the watchlist star so as not to crowd UI
Lila: can you enter content that is not on WP? no
Lila: can you enter content that you just created? not quite

slide 41

[slide 41]

JonK: experimenting with curation
in future the data from this could also inform browsing
Moiz: tool for telling story
Lila: not seeing yet how this affects [interacts with] user identity
don't have ID (user page) on mobile ("I am Lila")
Moiz: have very bare-bones user page on Mobile
also, could have notifications...
Lila: aggregate people around knowledge, topics (e.g. castles), facilitate collaborations
Vibha: should look at data from e.g. Nearby feature
how navigate collections?

slide 42

[slide 42]

JonK: only six weeks left to deploy, get some data
Jared: user will not be able to share collection publicly (unless enrolled in beta)
Brian: this is the playlists we introduced at Spotify; expect growing pains
Lila: flexible on timing

Asks/blockers

slide 43

[slide 43]

Dario: Analytics:...
Maryana: don't just grow engineering, also need design, research to keep pace
Lila: for future, get a sense of this ratio
Maryana: frontend moves very fast, but backend slows us down
Lila: build APIs yourself?
Maryana: this will slow us down
Lila: not if we hire
Brian: yes :)
Toby: concern - this needs to be managed
Lila: yes, would be a mental model change that takes time
Erik: from Product perspective, biggest issue is that there is no ownership for Services yet
Maryana: help us innovate - need to be held to goals, but also given room for experiments
Lila: thanks all, this is great, I like what you are doing. Please do sharpen pencils
naming is important, eg. "collections" - that's a marketing exercise
Jared: should rely on Communications team, but I could also recommend outside expertise
Lila: as soon as you put a project name on it, it will stick