Wikimedia monthly activities meetings/Quarterly reviews/Mobile and Wikipedia Zero, April 2015
Notes from the Quarterly Review meeting with the Wikimedia Foundation's Mobile Web/Apps and Wikipedia Zero teams, April 14, 2015, 10:00 - 11:30 PDT.
Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material
Present (in the office): Garfield Byrd, Jon Katz, Dan Garry, Carolynne Schloeder, Tomasz Finc, Maryana Pinchuk, Luis Villa, Lila Tretikov, Damon Sicore, Dan Foy, Terence Gilbey, Yana Welinder, Tilman Bayer (taking minutes), Kourosh Karimkhany; participating remotely: Geoff Brigham, Jorge Vargas, Smriti Gupta, Adele Vrana
Erik: Welcome
each of the Engineering/Product quarterly reviews this week has an owner, for this one it's Jon Katz
Jon: welcome, over to Dan
Apps
edit
[slide 3]
Dan:
iOS app: had an app store submission we were happy with, Apple too
but then we did testing and discovered issues (freeze while file transfer), submitted again after fixing this regression
could still have shipped by deadline, but wanted to coordinate with Comms for launch
Lila: you did the right thing
Terry: why was the regression not discovered earlier?
Dan: did not do enough testing
QA for Apps is still new
tested all the expected problems, but this one was unexpected
need more rigorous QA, are contracting more regularly with external testers for this
they found this particular issue independently, so I'm optimistic this will not happen again
(Lila, Dan, others: further discussion on crashes etc.)
(Dan:)
"Read more" feature:
hypothesis was: single link (instead of three) with more images and metadata drives more readership.
test did not confirm this hypothesis, so we didn't ship this
[slide 4]
successes
Android app got featured (after incorporating quite a bit of feedback from Google)
Lila: do you expect monthly uniques to go up as result? (they seem flat now)
Dan: yes, but not that much
--> Lila: provide dashboard with top level KPIs for apps (about 3 numbers)
Dan: a lot of data transferred to the app is not used, this is an issue for people without unlimited data, and for performance
with new prototype service, sometimes 20-30 roundtrips are reduced to 1
memory overhead: for a long time, had been receiving vague out of memory crash reports
Damon: memory problems can be hard to solve, does team have enough capacity for it?
Dan: yes. doing well on that now
Tomasz: (cites current crash report numbers)
Damon: that's really low
Lila: can you talk about the number on slide 9 and 10?
--> Lila: in general, teams should accompany the data they include in the quarterly review decks with their insights/interpretations
[slide 9]
Appendix - Apps search metrics
Dan: people type in different queries before tapping on the result they like
--> Lila: we should measure first time conversion for app search
Dan: OK, have that data
Lila: it is interesting that conversion dropped
Dan: also, added Wikidata description, so if looking for a definition, reader may already get they information they want from the search results only
Erik: a lot of low-hanging fruit remaining, eg. "Obama" still doesn't yield "Inauguration of Barack Obama" article[s]
[slide 10]
Appendix - Read more vs Read next
Dan: "Read more" looks like search results (image, Wikidata description, title)
"Read next" looks similar to lead image view
Lila: conversion is much higher for read more
Dan: "shown" higher for "read more", tells me that people purposefully scroll for that to end of article [after having learned previously they can find it there]
Lila: this is a learning opportunity: can test what people actually looked for (in article)
Lila: how is the share feature doing?
Dan: in first 4 days, 20k users and 40k tweets ...
important: can't record if users successfully tweets (after tapping share button in app)
Erik: Moiz' feed at https://twitter.com/wikishots is kept busy ;)
Dan: hiring meant significant increase in velocity
[slide 5]
Misses
average search attempts: 4 :(
Lila: fixing that should be #1 priority
Dan: have reached limits on what we can do in frontend, need search backend improvements
API account creation outage
Lila: why are we not monitoring this?
Dan: we seem to be the only users of account creation API
--> fix account creation API monitoring
Tomasz: very little monitoring infrastructure, at best it's only data [numbers on usage]
Dan: on the plus side, another API outage was detected and fixed very quickly
Dan: (remarks on content service)
Erik: service does all the reformatting for the app
still a somewhat immature approach to using services
Dan: app gets much more data than it ends up displaying
e.g. stripping out navboxes
painful slowdowns especially on older devices
[slide 7]
Appendix - Monthly unique users
[slide 8]
Appendix - Android daily downloads
getting featured had more impact than releasing app
Lila: think about that next time
Tomasz, Dan: also, there's "editor's choice" (one level above featured)
Lila: need to talk about driving up uniques
of course can put link on site, at some point we should do that
Dan: Google search (?) already offers to open result in app
Mobile (WikiGrok)
edit
[slide 11]
Jon: I will be driving this, but so far it has been Maryana
[slide 12]
successes
engagement 3x higher than on French fundraising banners
[slide 13]
Maryana: Wikidata community follows same trajectory as Wikipedia - after initial period of freedom, more and more rules
Lila: how about a duplicate database, from which community takes those entries they find appropriate?
Erik: Kaldari posted example edits and asked community to evaluate them, before asking for bot approval
response is mixed, some excited about it, some concerned
bot request still in limbo https://www.wikidata.org/wiki/Wikidata:Requests_for_permissions/Bot/WikiGrok [PS (Tilman): "approved with reservations" just today]
Lila: how many participants? about 10
Dan: a typical bot request has only about 1 person look at it [and decide]
Terry: we prioritized submitting data over more reader experiments [?], was that because of usage numbers?
Erik: I asked for this to find out how hard it would be to fully implement it
if we don't know what the end of the pipeline is, it does not make sense to experiment much further
not a lot of existing relationships we can leverage
Maryana: NB, Kaldari is admin on Wikidata [or was until 2012]
Terry: underspent in engineering, so if we could have done both [further experiments and submitting data], this would mean opportunity lost
Damon: if we could prove that data has a certain degree of reliability, would that have helped?
Maryana: we did some of that
Lila: we should have a meeting to discuss how we move forward with Wikidata/structured data.
[slide 15]
Jon: Wikigrok-> readership team
[slide 16]
Q4
Lila: sounds like backend is not ready
[slide 17]
mobile pageviews (logarithmic scale)
[slide 18]
mobile pageviews (linear scale)
(Jon:) mobile web growth is still tremendous
(discussion about overall pageviews)
Mobile Web: Gather
edit
[slide 19]
Jon: ...
[slide 20]
Realized within first two weeks into quarter that we would not reach objectives
so we revised objectives
[slide 21]
Successes
beta has all features you would expect from a brand new product
needed more resources, got Yuri from WP0
Lila: what are you doing this q?
Jon: need moderation tools. not the same level as encyclopedia content, but...
--> Lila: should review this [concept for Gather], a lot of complexity here which could cause problems
concept is important, just want to make sure (we do it right)
Jon: many find feature, but few find collection again
fixing these issues iteratively
Lila: find out how many users it brings back, compared to baseline
Jon: we don't have that data - are not tracking users (readers) for privacy reasons :(
Dan: only have that data for apps
[slide 22]
Misses
Jon: don't have culture of innovation, e.g. for rapid prototyping
our existing systems that block bad code are a hindrance here (prototypes always tend to contain bad code)
also, resistance from community and fellow staff
Damon: need to find out what possible "playgrounds" are
Luis: need to improve innovation
some of these issues strike at core of how we scale
can't talk about eliminating roadblocks without...
Lila: we don't want break fundamental ways of how Wikipedia works
find out what the things are we should not touch
Luis: and how to feel more bold when not touching them
Damon: ...
Jon: ultimately, converge watchlist and saved pages
Jon Robson integrated lots of watchlist features already
Dan: held off on apps side because Gather is still experimental
saved pages are private collections
Erik: would caution about judging Gather based on short term numbers
even if initial numbers aren't blowing us away, a holistic strategy could lay the foundation for lots of effective engagement methods (e.g., content feeds)
Dan: also it's now duplication of...
Lila: how does this feed into curation?
Jon: these pages are public...
need a way to hide "bad" collection
Lila: but it doesn not threaten [existing] workflows? [no]
Luis: needs to be curatable by community (staff can't do it all)
Jon: aim for flagging mechanism like on Facebook
Luis: ...
- Appendix slides
-
-
-
-
-
-
Wikipedia Zero
edit
[slide 29]
Erik: Zero team has been transitioning to Kourosh
Carolynne: we were concerned about impact, pageviews
hence proposed deep dive. It was put on hold due to restructuring
IP migration for HTTPS on good track
Loaned Yuri (see above), which disrupted engineering work
[slide 30]
objectives:
Android preload
[slide 31]
some of the reds actually have successes behind them
e.g. Russia
[slide 32]
Misses
Deep dive on hold. We need to get more insight into Global South users
Luis: is my team involved?
Carolynne: To some extent, e.g. Asaf's and Rachel's areas
Terry: planned outcome of deep dive?
Carolynne: growth in usage (uniques)
[slide 34]
Damon: what about awareness in these markets?
Carolynne: Comms is an important partner in this
could work with research consultants
needs to scale, e.g. develop templates
Damon: will this come out of deep dive?
Carolynne: deep dive is on hold, an opportunity to look at this
In India, Aircel terminated because not reaching their objectives, no growth, policy shift against content-based pricing
basically English-only usage
affordability isn't biggest issue in India any more, lack of non-English content is
Damon: so we have lots of options, but not enough insight which to prioritize?
Luis: limitations of deep dive approach: areas differ a lot, insights from one might not apply to other
--> Kourosh: Damon, let's meet on this [Zero/Global South]
Lila: want to understand past boots on the gound efforts and their problems
Erik: one problem: avoid triggering people you want to work with
Kourosh: preload will be center of strategy that I'm going to present
[slide 35]
pageview chart
Carolynne:
dip in February is probably seasonality
[slide 36]