Wikimedia monthly activities meetings/Quarterly reviews/Reading and Community Tech, April 2016
Notes from the Quarterly Review meeting with the Wikimedia Foundation's Reading and Community Tech teams, April 12, 9:30 - 11:00 AM PT.
Please keep in mind that these minutes are mostly a rough paraphrase of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material
April 12, 2016, 9:30am-11am
Present in the office: Katherine, Jon Katz, Danny, Anne, Zhou, Stephen LaPorte, Tilman, Sherah, Nirzar, Josh, Maggie, Geoff, RobLa, Joady, Jaime V; remotely: Adam Baso, bd808, Dan Garry, Dmitry, Kevin, Michael Holloway, Niharika, Ryan Kaldari, Trevor Parscal, Wes, Rita Ho, Volker Eckl
Jon: Welcome broken into 2 areas: 1) goals and progress 2) how do we build our muscles as a team, build what we should be building?
jk: YoY decrease in readership, actually a little misleading. There were two 1-time drops leading to this result. But actually about flat over 3-year period.
Slide 2 - Objective: Lang switchingEdit
Adam: first goal for reading web team was making enhancements to language switching one of the most clicked buttons, thought worth spending time making interaction more fluid goal: roll out changes to beta -- we were able to do this, ship some stuff to prod even
we've taken lang-switching button, used to be on bottom, requiring scrolling, now on top doing some research, anecdotally it's confusing for users used to the old way with button at bottom of article
On the right is the language overlay people now get when they tap the btn Old display was fairly minimalist, just some lines of text Now, structured! See a list of preferred langs, user tapped on DE at one point, also see ES which is the user's keyboard lang
We've applied this in other contexts like search and works pretty well, decide to do here too Still working on sorting routines for the langs in the 'all languages' list
Seems like users get less frustrated with new overlay, engage more. We think it helps users look up synonymous term, because scrolling has decreased while language selection is strong. Tends to drag down tap-through (language selection), but numbers are still good and wethink it's a net good for users
So looking pretty good, tho we're still working on how to expose lang switching button (call to action maybe?)
jk: web team's focus is really on new readers, particularly in emerging markets/communties
Slide 3 Objective: Web SpeedEdit
(jk cont from 2) we see that we have a goal of reducing page size sent to improve speed reduce image size (went from retina-grade to another v. high grade) -- reduced image size 60%
also shipping less HTML (some of which wasn't being shown anyway)
This improves cost, but does it make it faster? We prob didn't hit our goal, but made some improvement. Need to measure better.
Q4, will roll out lazy images (and references, probably some other odds & ends) -> reduce size, let page load faster.
Slide 4 - Objective: Web Speed / Mobile Article SizeEdit
Shows impact on size (ex: Barack Obama page on mobile.) Images are huge % of total article size. Thx to performance team for their work here.
Question: do we know how many images are seen?
- jk: with lazy loading in beta, based on results expect to see 75% reduction in total image size.. which suggest that 75% are not seen
- jk: also, something like 40% of users never open section on mobile
Slide 5 - Collections Feature removedEdit
jk: Removal of collections feature was popular with readers, but NOT editor community intro'd moderation concerns
Community RFC went out jan, near unanimous vote to remove the feature Made some friends in comm, but ultimately had to remove
Hurt Android team's quarterly goal since dependent on the Gather backend
Learning: need to address comm concerns in timely fashion; also, our notion of 'beta' not necessarily == the community's.
Geoff: What was the biggest pushback?
jk: Prob moderation. There were ways but prob outside the existing system. There was a misconception also that we didn't think the feature through. In fact we saw that traffic was low enough that existing mech's would be fine.
Geoff: My experience is it's always a learning experience working with communtiy in rolling things out.
Slide 6 - iOS: Objective: Data LayerEdit
Josh M: Our goal this Q was last Q's goal be completed -- to release major new update (5.0.0)
Goal of the update, measure of success was retention. Not use just 1-2x but keep using after 7 days (pretty standard app industry measure)
Big hit, great press, low crash rate, moved retention #s significantly
misses: accessibility (visual impairments in particular), user education/support, analytics Our analytics infra is not designed around apps, so kind of working around what it can support
Good relations with Apple, app was quickly approved
Wes : They (Apple) also discovered they had a problem internally b/c of the app?
Josh: Yes, they needed us and some other apps to address the bug uncovered in iOS 9 (i.e. on their side)
Slide 7 - retention, true “app-y” experienceEdit
What was 5.0 all about? -Added feed of main page, most read content from pageviews api -Privacy-friendly reading recommendations -- browsing history not persisted in server profile -Deeplinks -Previous version looked like a browser, this version is much more app-y --Scrolling feed is a common pattern --App-specific things, like working with the OS on search, ...
Slide 8 - Comms workEdit
Thx to comms team Juliet worked with us about how to talk with people about new features, press was v. much on message, responded well
Slide 9 installs, active devices graphEdit
Impact: red line is active device count Blue line new installs Nice pop from the press, apple Active device count has risen significantly and stayed Goal wasn't so much to grow a huge audience as retain the audience we have
KM: what is done about getting it featured in the app store?
Josh: unfortunately they had already selected a different WP app to be feature around the time of the launch JM: A contact at Apple has asked about our future release schedule, so that we can be featured going forward Currently still featured under 3D touch apps
KM: They do great promotions, back to school was a big one last year, would be nice to see about more general purpose promotions
Slide 10 - retention numbersEdit
Geoff: Why is there a downward trend in active devices? JM: Becuase of the recent issue with deeplinks not loading Also, download rate not much higher than previous version
KM: We were comfortable with the idea of being sort of experimental this Q, do you have a plan to increase downloads? Jm: still have a ?? JM: Trying to hire an audience manager as well
How to read these charts: 0-day (download day), then % of people who downloaded still using after 1, 2, ... 7 days Retention is a hard metric to move becuase youre really trying to change someone's habits
GB: Good % by app industry standards? JM: 10-15% is generally good for reference apps, more like 40% if you're a social game People are using Wikipedia every day, fact we're not capturing them is a shortcoming
Prob will not be getting to 40% but we can do better Feed is simplistic, can get better
Slide 11 - Android: Objective: Sync across devicesEdit
DB: Strategic pillar was "community of readers": cultivate community of sharing content, make it easier to discover Ultimate goal: fully social collections of articles, can comment on, suggest additions to We knew Gather was already beta feature in mobile web, seemed like a good idea to plug in, use their backend
Goals this quarter were steps toward that larger goal:
- -Sync saved pages so they are available when logged into another device
- -But Gather was shut down
- -Explored other options, but they weren't sustainable
- -We still want to achieve this goal, but it's on hold now
- -Talking about how it should be impolemented in MW, work with community to do
Slide 12 - Android: Reading listsEdit
We did press forward with "reading lists", a generalization of the saved pages feature -User can organize pages in multiple lists with unique names & descriptions, for better integration into the reading experience -Believe it can improve retention even if not synchronized -Build in such a way that it can be plugged into a syncable backend when one becomes available -Also substantially delayed by Gather removal, but should be finalized and in beta next week or two -Then we'll have some updated #s on retention
Km: Earlier it sounded like the challenge wasn't synchronization but community issues? DB: Reason sync was the problem is that Gather was disabled, we lost the backend KM: So there's no inherent issue with synchronization? db: no
KM: How do you see moving forward on Comm of Readers work, what to do next? DB: Exploring features like multiple watchlists, maybe have those be sharable; but will require lots of comm input
JK: We're also engaging with community on a more co-creation level for any further features in this arena
Slide 13 - Community Tech - Objective: Prioritize needsEdit
DH: Did big survey, our job for Jan was to report back with our results on what people think we should do We're planning such a report every few mos. First one at the dev summit, then one at Hackathon/Wikimania, then one in fall
Also have goal of on-wiki discussions start to finish on projects; we've had such discussions on small project but no big ones yet Discussing 3 different approaches to cross-wiki watchlists
Reporting back done, on-wiki discussions starting soon
Slide 14 - Community Tech - Objective: Wish fulfillment - core contributors' productivityEdit
Big goal for q3: complete development and ship:
- 1) migrating dead external links to archives
- -Replace links marked as dead in template to links to Internet Archive*
- 2) pageviews analysis tool
Volunteers are already working on both of these and we don't want to discourage them, we want to work with them
Slide 15 - Community Tech - Migrate dead external links to archivesEdit
RK: #1 request on wishlist survey was migrating dead ext links to archives Vol named max was already writing bot to do this We built a logging system for it, hosted on tool labs Multiple bots can interface with it, also has API so bots can see last time it was assessed/fixed and can avoid duplicating work Also lets bot know last article it fixed was so it doesn't lose its place
Also contributed advanced detection code. Initially it relied on volunteer hand-marking dead links by template. We wrote code to allow bots to detect dead links.
Good news: first phase (fixing all marked deadlinks on enwiki) is almost finished. There were (#?), now only 60,000 left. Next phase: Deadlink detection phase
GB: What has been community reaction: RK: enthusiastic. People have been asking for this for a very long time. Other (non-english) wikis actually had solutions for this but enwiki never did.
??: Was there a scale issue on enwiki? RK: Probably, enwiki is the largest and has by far the most references Other issue is that the enwiki community wanted to Do Things Right, have good long-term solution
DH: Tool is being/can be internationalized. But there are concerns about convincing other language communities to use our tool rather than what they're already doing
KM: Is this solution as robust for other langs? Does Internet Archive have the same depth for other langs? RK: We worked with Internet Archive to help them extract all the external links from all the wikis, so they could queue them up and try to archive all of them; should be good coverage across wikis.
GB: What was error rate/case requiring human intervention? RK: We don't have that data (b/c involves human interaction)
DH: Template will show when somebody has manually changed the status
Slide 16 Community Tech - Pageviews Analysis toolEdit
RK: Before, most commonly used tool was stats.grok (offsite), it was notoriously unreliable and people were frequently upset Features were limited, usually just static graphs of pageviews Worked with a volunteer (MusikAnimal) to build a nice new tool on tool labs using API developed by analytics team
Currently getting about 15,000 hits/day We worked on multilang support (was initially english only) And data exporting And various UI/bug fixes
DH: Interestingly, every project that we started, someone was already working on it. It'll probably turn out that most of our projects are collaborations. What people want most are usually things people have been thinking a lot about and at least starting to work on.
Also had some staff issues this Q (down a person).
Slide 17 - Community Tech - other successes and missesEdit
Other projects before wishlist survey:
- Gadgets 2.0
- PageAssessments extension -- basically to provide easy interface for projects to keep track of articles and their importance
Both had blockers and took longer than expected, so we depriortized in favor of comm wishlists projects, but hope to finish next Q
DH: Also working on cross-wiki watchlists; will be exciting Q.
Slide 18 - Reading infrastructure - SecurityEdit
BD: Two projects to report on:
- 1) AuthManager -- steady and slow progress but has taken a backseat to other time-critical issues
- Goal is a more modern, flexible auth interface
2FA for high-risk accounts Password verfication service so we can move them out of primary db and to more isolated area
We also deployed SessionManager -- change to how we handle cookies/session state Rollout issues took more time than estimated. Hoping to be done w/ AuthManager this Q, now looking like mid-late next Q Also based on that experience, feature-flagging AuthManager so can be disabled in case of deployment issues Currently waiting on security review so we can merge the core code into trunk
Hope to get into the next 1.27 tarball release (otherwise will have to support the old auth code for next 3 years)
Slide 19 - Reading infrastructure - API EngagementEdit
Started work Oct last year, but stalled b/c the discovery project we were assisting with had some hiccups, needed fixes This Q, the data pipeline got completed, allowing us to measure all Reqs to Action API, including data never captured before in pageviews table:
- -params posting w/ API requests
- -which API modules are being used
- -whether internal/tool labs or 3rd party users
Allows analysis re: user-agents (we recommend that every user/bot use unique user-agent) Data flowing into Hadoop as of end of last month
Raw usage of Action API has volume roughly equal to enwiki pageviews KM: Were you expecting this #? BD: We suspected so based on the size of the log files, but didn't really know until we got this data
Slide 20 - Reading infrastructure - Other successes and missesEdit
-Non-free image switch: enwiki found they didn't like fair use images as hero images, we changed code to surface this -Got people one-time snapshots of their Gather collections before removal -Major rewrite of API sandbox, folded it into core --In doing this, extended OOjs.ui with new date/time widget -Invented security feature allowing quick expiration of user sessions. Previously took 48h.
Adam B. taking over as manager as I (BD) transfer to comm tech.
Slide 23 - StrategyEdit
JK: Q2 was about strategy, this Q more about execution --Greater reliance on design research, working more closely together, focusing on collabs across the org
Slide 24 - Overall pageviewsEdit
Pageviews still flat to negative --Important finding to add clarity: If we only know pageviews, don't know why: disintermediation, move to mobile? ---recently got device #s from analytics and it turns out people view 40% fewer pages on mobile devices, so move to mobile is likely in large part driving the drop
GB? : So this could be consistent with increased # of readers JK: Y
Slide 25 - strategy assignments by platformEdit
JK: Three themes, divided by platform so each team can have focus Not sure which will pay off first or fastest
Improve the encyclopedia experience: new experience: extend the experience, add new use cases new readers: extend to places where access has been an issue
Slide 26 - operationalizing our strategyEdit
JK: How do we define success metrics? (see slide) There's overlap but these are the basic buckets.
Slide 27: Community (cover slide)Edit
JK: Lack of relationship with community is one of the biggest problems we faced (#1 for web team) -Not just can't get approval, but aren't even sure what approval looks like
Slide 28 - Developing Useful Engagement ModelsEdit
We want to move to the right here. Initial sense was adversarial; mostly stuck in the 'customer' bucket now, moving toward partnership/multiplier model
Slide 29 - Mapping Engagement SuccessEdit
When/how to talk to community? The more early and the more specific the terms, the better the results
Slide 30 - Progress on Engagement (then)Edit
Slide 31 - Progress on Engagement (now)Edit
JK: Significant inroads this Q in moving convos upstream rather than presenting comm with finished products and asking how they like them
e.g.: Language switching co-creating RFCs with community members
JK: Haven't yet shipped anything with this model Nirzar: User pages will be
Slide 32 - Progress on Engagement - Reading / Community Tech (long list)Edit
JK: Our on-wiki consultation methods aren't working on mobile. KM: Who owns these methods? JK: in one case I am referring to idealab. I don't know who owns it now, but it was created awhile ago by Jonathan Morgan and HEather.
- : Might be helpful to contextualize some of these charts (e.g. "Adversary --> Customer") could be upsetting without context
Slide 33 - Understanding our Users (cover slide)Edit
JK: Right away when started reading team we realized we didn't know a lot about our users.
KM: Impressed by depth of thinking that's gone into comm engagment, great to see
Slide 34 - Understanding our Users (list)Edit
JK: Now we know about numbers, starting to do ethnographic reserach (Mexico); usability lab running every other week Working on analytics for mobile usage
Slide 35 - Mexico Reader’s ResearchEdit
AG: Currently working through the data, figuring out presentation
Slide 36 - Looking aheadEdit
Will be heading to Nigeria (May) and India (June)
Slide 37 - Research team work - depth researchEdit
JK: Teasers from research team:
- Asked readers: what are you looking for? (in-depth info, overview, specific fact)
- ~25% come for in-depth learning -- about what we thought
Slide 38 - Research team work - motivation researchEdit
JK: Also asked: what is your motivation for coming to WP:
- Top answer: Media (I saw a TV show or read a book) -- most often mobile
- 2: work or school -- most often desktop
Geoff : What's the implication in real terms? -Better ways of thinking about, e.g., which users are getting what they need (quick fact) from Google -How to think about session behavior (what they do while they're here)
Slide 39 - Reading Q4 Goals (cover slide)Edit
Slide 40 - Reading: Q4 Goals: Web, AndroidEdit
Web: Hovercards feature -- allows mousing over link to preview article. Much better for users on slow connections; don't ahve to wait for full page load. 75% of users don't want full article. So better for them. KM: Will we be tracking hovercard hovers? Can we capture: JK: Y, working closely w/ Analytics to capture. Not exactly pageviews. Tricky b/c some will be accidental. KM: Prob not useful for overarching product decisions but good data nonetheless. JK: Data previosly showed pageviews didn't decline that much w/ previews.
DB: Android project for Q4 is feeds; it's the kind of feature that helps drive retention (as Josh's numbers shown) -- all about providing great first impression to the user. Currently just the front page. Want to be able to show variety of interesting content from the start.
JM: There's no backend for the iOS feed, it's all the OS. Would be nice to incorportae backend. DB: Android team owns the mobile content service; we'll be working to encapsulate some of these features.
KM: Sounds like two goals in there...
JK: Sometimes people wonder why platforms don't all look the same all the time; this is an example where we're testing on one platform then scaling
Slide 41 Reading: Q4 Goals: iOS, Infrastructure, Community TechEdit
JM: Launched partial support for univeral links --Now support top 6 langs --SWAT deploy is scheduled for thurs to do what remains to be done on universal links
Apple reached out re: handling all searches on the device in standardized way
Out of time.