Wikimedia monthly activities meetings/Quarterly reviews/Mobile contributions/2013-03-18

The following are notes from the Quarterly Review meeting with the Mobile Contributions team on March 18, 2013, 12:30-3:30pm.

Agenda: http://www.mediawiki.org/wiki/Mobile/QuarterlyReview

Present: Howie Fung, Maryana Pinchuk, Brion Vibber, Tilman Bayer (taking minutes), Erik Moeller, Sue Gardner, Tomasz Finc, Juliusz Gonera, Jon Robson, Adam Baso, Dan Foy, Brandon Harris (from 2:40pm)

Remotely: Michelle Grover, Arthur Richards, Max Semenik, Yuvaraj Pandian, Yuri Astrakhan

Please keep in mind that these notes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material.

Slides: Mobile Contributions Quarterly Review, 3/2013 (additional data)

Team intro and Annual Plan goals

Tomasz: Welcome
Today focus on contributory features, not pageviews etc.
Goal was: 1000 unique photo uploaders/month
Rationale for chosing this: active community on Commons, ...
Work divided between Web (majority of efforts) and Apps teams (more experimental)
Reason for split:
Majority of readership on web, rather than apps (exception: WLM app)
Matrix: Which product (web/app) to reach which user group?
Coordination tools (team is very spread out): email (also important for later reference), IRC = water cooler, Hangouts (of central importance), wiki, Mingle

Q1: WLM
Q2: readership improvement, contrib experiments
Q3: photo uploads and account creation

Alpha --> Beta (400k opt-ins) --> Stable (what everyone sees)

Recap of Q1 (Jul.-Sept.)

Arthur Richards:
almost entirely focused on WLM app, after having been approached by WLM organizers in spring
shows nearby monuments, allowed photo uploads for the very first time
very short deadline, changing requirements
adapted lessons from fundraising on agile development
hoped for 10k uploads during contest (September), reached 3K (~1% of total WLM uploads)
3.5% of uploaders used app
successes: 1st mobile uploads, process changes to adapt to short turnaround time, learned valuable first lessons
challenges: no dedicated QA, Phonegap framework (that was used in earlier WP app) very hard to debug --> moving away from it now, complications with WLM data (inconsistencies, changing requirements...)

Recap of Q2 (Oct.-Dec.)

Arthur Richards:
UI improvements: navigation, improve table display, collapsed maintenance templates (into "this page has some issues"), interlanguage links (frequently demanded. previously missing, now present in collapsed form)
login and account creation (beta), critical for contributions
watchlists - low-hanging fruit, incentive to log in
introduced alpha mode ("here be dragons mode")
experimental contrib features:

section-level editing (pushed to Beta in November), very first mobile text edits. form factor is very restricting
photo uploads, directly adding to article (to Beta in December)

measuring impact, started to collaborate with analytics team / E3
successes:

Better teamwork improved our efficacy,
as did the alpha version
collaboration with E3 and Analytics teams, hope will do so with VisualEditor team too

challenges:

collaboration with Ops team
no decent testing environment
team changes

Recap of Q3: January-March 2013

Maryana:
main focus: photo uploads
1000 unique uploaders goal
also worked on moving other features to beta or production
watchlist star as incentive (to logged-out users)
significant impact - mobile account creations ~10% of desktop account creations
4% went on to edit within 24h (compared to 24% on desktop)
Mobile watchlist: offer toggling between both all pages view for new users (reading list) and usual list of recently modified pages (for experienced user)
Impact: YOY comparison 2012/2013: seem to see a rising trend within 2013
"add an image to this article" feature
results: not always high photo quality (some of the usual experimentation), but lots of interesting contributions, quite international, very few inappropriate ones
Sue: this could have serious implications for future nature of encyclopedia, more contributory/on the ground/as it happens
no quantitative analysis of quality (coding) yet
little gamification element: # of uploads displayed
quite a few self-portraits; contributors who seem very proud of contributing, even if not yet clear about educational content etc
Sue: Commons right now full of high quality encyclopedic aiming work. Could become much broader, democratized; could be transformative for the kinds of material we store and offer
Maryana: Apps: more for existing, experienced users
responsive design adapting to screen sizes
impact: currently 50 uploaders/week for web feature (projected 200), 30/week for apps (projected 100).
Brion, Tomasz: currently more testing/participation on Android, iOS will catch up

Plans for Q4

Maryana: with Geodata database, productize "nearby" feature , "articles without image" call to action
more deliberate marketing for mobile features
discuss role of mobile more thoroughly (acquiring new users?, engaging/retaining them?, empowering existing users?, closing loop for involvement by e.g. notifications?)
Learned in Q1-Q3:

lots of readers curious about participation features. will get at least some usage
experienced users prefer apps, but far more traffic on mobile web

successes:

generated excitement about mobile features
got new users to sign up and try out features
team building

challenges:

CentralAuth
getting deep community engagement (experienced editors ask about apps, dont look at mobile web)
market strategy

Erik: who did community engagement? / Was there support for liaison type work?
Maryana: mostly me / no

Asks

Tomasz:

video conferencing hardware. Sue: main issue maybe that Chip's setups in conference room are tampered with?
connection with Ops team: changing points of contact, need dedicated site reliability engineer
need better QA infrastructure
- better testing environment
- convert Michelle to full time
Business analyst - so far split between various team members. Arthur: this would be translator between parts of team
Analytics: made strides with e.g. Eventlogging (Ori), but still need lots more infrastructure in place, e.g. distinguish different types of users

Erik: is there an agreed upon alternative approach which would solve this?

CentralAuth: talk to platforms team

Questions

Sue: On track for 1K mobile uploaders target? Tomasz: Yes
Sue (summarizes): Team does lots of different things, seems to have a plan, is productive, has some fundamental questions about direction (which of the many opportunities to focus on), says that division into two groups is working. Is hitting its targets. Is it true that this is one of the most effective/productive/happy engineering teams?
Erik: Yes.
Sue: Reasons for that? Lots of great experienced members with good tenure in the movement/organization (e.g. Brion, Tomasz, Arthur); includes "shit has to get done" experience from Fundraising team. Fundraising got effective quickly because it had to raise the money to fund everyone's work, now mobile is learning from those experiences. Another factor: On mobile, fewer community expectations (e.g. to solve problems for experienced editors), more leeway for experimentation, less entrenched user base.
Tomasz: Yes, but this is a window (perhaps 1-2 years).
Howie: This team has not plugged into existing workflows much, will see more friction as mobile contributions expand.
Sue: Three things: 1) I have a few follow-up mop-up questions; 2) let's talk about Maryana's question for direction, and then 3) about your resource asks (notwithstanding Annual Planning process)
Sue: First a personal question: why do I constantly get logged out?
Juliusz, Brion: Don't have "keep me logged in" checkbox on mobile, token can only be on one device.
Erik: Legal required to set limit back to 30 days due to conflict w/ privacy policy.
Sue: We should change this. Keeping users logged in is beneficial for them. I want to be logged in; I am annoyed when I get logged out. It impedes me from participating in the beta/alpha, knowing about new functionality, etc. Getting logged out makes it harder for me to have the Wikipedia experience I want: it's not user-supportive.
Tomasz: This is an example for the opportunities the mobile team has to modernize MediaWiki.
Sue: Found the UI of WLM app not great: was that because it was a hurried project, or new ground for us?
Tomasz: had just one person focusing on design, have expanded that since. Now have learned more, and have better infrastructure.
Howie: Designers didn't have much mobile experience, had to learn. Have two new upcoming design positions.
Sue: user testing?
Maryana: hard to emulate real person's experience with user testing companies. Can't ask them to walk around in city to test "nearby" feature.
Michelle: Trying to collect suitable testers, evaluate feedback comments
Sue: Hardcore Wikipedians will not help us in that, because they are *too motivated* to overcome even difficult technical issues ;)
Tomasz: App feedback has a much higher signal to noise ratio than the feedback for the mobile site.
Erik: Eventlogging etc. gives us more specific data about success/failure. Direct user feedback still useful if we can get the signal/noise issues worked out, esp. for surfacing edge cases.
Jon: ...
Sue: I have always assumed people will never will write whole articles from phones, based on my own experience and what Wikipedians told me. It's the form factor, but also the mobile situation (not sitting at home with cup of tea and lots of books). But also that there is opportunity for light editing: fixing typos, adding citations or a quick fact. Can you confirm that from your experiences so far?
Maryana: Actually, some experienced editors will do complicated edits, e.g. fixing infoboxes
Erik: Look at typical edit size byte ranges in https://en.wikipedia.org/w/index.php?title=Special:RecentChanges&tagfilter=mobile+edit ; mostly small edits
Maryana: Mobile editing might be good for e.g. writing about things while traveling.
Sue: long-term effect might be that notability bar comes down, which would be positive.
Tomasz: Shift to mobile devices
Howie (showing slides with additional data):
New comScore data, multi-platform now including mobile stats. Intersection mobile/PC
Time spent on PC: slowly rising, but not much. PC+mobile: much higher
Big question was/is: Mobile incremental or displacing? E.g. tablets could be cannibalizing.
For now, it looks like most mobile is incremental, not cannibalizing. Would mean mobile is opportunity, not threat.
different forms of editing
compare Yelp: full reviews (long form) vs quick tips (mobile)
for WP, e.g. page curation might be suitable for mobile
Erik: challenge with this: some of this could be less light than it seems (e.g. checking edits sometimes requires inspecting sources).
Initial focus has been uploads as low hanging fruit
want more quality coding analysis
features may be bringing in people who shouldn't contribute, don't add value

Sue: back to Maryana's questions on long-term planning (slide)
New user acquisition is a positive surprise.
Maryana: did comparison with other websites' approaches
examples: WikiHow tips: focus on engaging as many users as can, not worry about quality, or registration. Open the funnel wide, hope that a few more high quality contributors emerge at the other end
registration hooks: e.g. YouTube likes (offered to non-logged-in surfers, but require login when clicked). WMF does this with watchlist star.
Sue: Youtube's incentive is monetizing user data (i.e. benefit of keeping users logged in is not so much to generate contributions, but to be able to track viewing)
ex: Quora
mobile-specific user engagement
ex: Yelp draft feature (draft on mobile, later on desktop get reminder to edit and publish the draft)
Sue: yes, this is great. Use case for WP (apart from photos)?
Maryana: e.g. read article on NYT app, want to start WP article. Or: saw a new beer brand while traveling, took photo of bottle ;)
Sue: Often I want to type up notices from reading a book in a sandbox. But that is a private use, not a public use. The benefit of the Yelp app is that the draft is private.
(Tomasz and Sue discuss differences of private vs public drafts)
Erik: this could be a good trial. find out if people would like to use such a feature
Maryana: ex: Facebook. Can do everything on mobile that one can do on desktop. but for FB, it's not an acquisition tool.
Erik: same for Twitter. Tomasz: yes
Sue: Most contributions to Facebook are simple, less of a stretch to do it on mobile
Erik: some of our interactions are like that, can be moved to mobile.
Maryana: should have Flow on mobile.
Sue: that was mobile web. What about apps?
Maryana: A lot of existing user tasks could be pushed into apps (like the drafts feature).
Tomasz: A lot of things simpler to do in apps.
Erik: e.g. cropping images while uploading
Tomasz: ...(mobile web to apps)
Erik: Maryana started some conversation with Communications team on how to market features. Broad CentralNotice campaign might not be suitable for all. Give context on what kind of uploads we want. We are not used to doing marketing work in Engineering.
Sue: Journalists are getting spammed by all those companies, and there are junkets and freebies. We come from a different perspective and we'd be received differently.
Maryana: ...
Howie: Can categorize contributions into: 1. editing, 2. patrolling, 3. uploading, and 4. peripheral contributions. Haven't given much thought to #4 peripheral contributions, need to map to overall WMF strategy, active editors goal
How much should mobile team focus on editor acquisition?
Sue: general question: why participation? Reason: want to build a fantastic encyclopedia. E.g. Yelp quick tips may just serve to generate pageviews, but Yelp drafts aim to drive content. We don't want to just drive acquisition number. (E.g. AFT's initial goal was to improve article quality.) "Nearby" feature is great in that regard: focus on improving content.
Maryana: Peripheral contributions: e.g. adding categories is interesting candidate, dabsolver - worth exploring these
Sue: tasks that existing editors do that could be better done on mobile

Tomasz: Roadmap now until Q4, next year?
Erik: moving away from too fixed too long term planning. Therefore look at combined goal like mobile edits + uploads, so as not to set too many details in stone
Sue: That is fine.

Sue: Let's go to the team's asks: What's the path to improving video conferencing?
Tomasz: Team felt it has to nudge, support process instead of just specify needs.
Sue: Could ask Finance and Administration department to prioritize deployment.
Erik: A lot of this has already been mapped out, should be happening. Doable fast with contractors if necessary. Will sync with Tomasz to flesh out requirements, write to F&A.
Sue: Connection with Ops?
Tomasz: e.g. Jeff Green as contact person for FR.
Sue: Fundraising team's model is being copied because it is successful. That's good.
Tomasz: other teams like E3 and VE work similarly with Ops.
Erik: This team's work has lots of impact on architecture, e.g. caching. Other projects have more restricted impact on Ops
Tomasz: Everybody at Ops wants to help, but those who have not worked with mobile before need to be brought up to speed.
Arthur re QA: Michelle has been really valuable. Testing infrastructure
Michelle: need to balance between mobile and QA team
Erik: Permanent Mobile QA position is already in the 2013-14 plan
Mobile is unusual in that is has a high need for QA: lots of devices, difficulty of debugging
infrastructure. Is the mobile team in alignment with Antoine's and Rob's thinking on beta labs?
Michelle: talking with them. they need resources (bandwidth) for this
some of our major issues are with regard to caching
In the longer term, need testing environment where we can merge code.
Erik: Rob has asked for testing infrastructure resources, but not currently getting them in plan;
not planning full-time now, but could do contractors
Erik: justification for business analyst position?
Arthur: Right now split among team, resulting in significant context switching and overhead. In FR, we had a dedicated full-time BA and benefited greatly from it. Purpose is to distill and refine work to be done, in form understandable by all in team, act as buffer between moving parts
Sue: Analytics needs?
Maryana: two big ones: 1. measure global contributions. Currently very expensive to measure other projects. (Above stats were enWP only.) 2. mobile/desktop distinction
Maryana: Had some successes talking to Analytics, but had some troubles getting what we need, compared to shiny large scale stuff/dashboards etc
Sue: Reasons for this: 1. Had no head of analytics, which made it hard for the team to survey people's needs and prioritize their requests. 2. E3 team was lucky because they had analytics embedded, also Grantmaking &Programs department. Mobile has not had that.
Erik: This is being addressed e.g. by new head of analytics and other positions
Maryana: Some legal issues e.g. for unique users.
Sue: Privacy review going on right now. We are generally very conservative on data protection, but there are some good uses of data we should allow.
Erik: In last analytics review, mobile has been a priority.
Sue: CentralAuth?
Erik: for Rob, CentralAuth and OpenID are priority.
Arthur: We need mechanism for flagging priorities across teams.
Sue: In a normal organization this would happen via the directors' meeting. But that doesn't work for us -- because we are not much hierarchy focused as an organization, we need to develop communications mechanisms that will work for us.
Erik: Varying approaches to process management within Engineering (Scrum/Agile, ...). More rituals were introduced recently, e.g. these quarterly reviews. Ops started to bring in tech leads from other teams to their meeting.
Michelle: Information sharing with other teams (e.g. to find out that they work on similar things)?
Erik: Tell us if existing things are working, should not add new processes gratuitously.
Sue: E.g. Luis should hear directly from mobile team on needs (not just from analytics team).

Sue: This was a great meeting. Team is doing really well. Partly luck and circumstances, members with e.g. FR experience, good hires. But also good work, good management, skilled team.