Wikimedia monthly activities meetings/Quarterly reviews/Editing/October 2014

The following are notes from the Quarterly Review meeting with the Wikimedia Foundation's Editing (formerly VisualEditor) team, October 1, 2014, 12:30PM - 2:00PM PDT.

Attending

Present (in the office): Trevor Parscal, Rachel diCerbo, Rummana Yasmeen, James Forrester, Erik Moeller, Tilman Bayer (taking minutes), Kaity Hammerstein, Timo Tijhof, Abbey Ripstra, Jared Zimmerman, Gabriel Wicke, Damon Sicore, Daisy Chen, Dario Taraborelli, Terry Chay, Howie Fung; participating remotely: Sherry Snyder, Erica Litrenta, Moriel Schottlender, Bartosz Dziewoński, Subramanya Sastry, C. Scott Ananian, Arlo Breault, Ed Sanders

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

JamesF:
Welcome
Trevor: we changed the team name
(Participant intro round)
JamesF:
slide 3
since last q
hired Bartosz
Converted Moriel to full-time
potentially more changes coming up

slide 6
Personae mapping
500m casual readers, some of them power readers
Draft Wikimedia-level personae don't currently capture logged out editors, which is a big gap compared to our focus in Editing (about a third)
my personal mnemonic: E for editors, second letter for role (e.g. Enya - editor who is new)
Lila: this is just one dimension - edit count
what about depth? (e.g. writing huge article vs. fixing commas)
need to start that other dimension
what motivates people, what do they edit
JamesF: indeed, I had another column here that I took out with percentage of edits / byte count (instead of percentage of editors)
Lila: needs to be analyzed, probably not by this team but across Product
JamesF: yes, rough mapping, definitely imperfect

slide 7
This shows when VE becomes most useful to each persona

slide 10
Abbey: doing a card sort for editing tools
Lila: sounds simple, any reason we don't announce this more widely?
Abbey: planning tweet, etc. working with Comms, CE teams
Lila: target banners to e.g. particular edit counts

slide 13
goals of user survey: find out how frequently used
Trevor: we did click analysis on toolbar, made us think it's biased towards proximity ;)
Dario: emergent theme in several teams' quarterly reviews: need a scalable way to do this; with handcoding etc
Lila: agree we need to be (proactive)
Gabriel: can analyze diffs wrt DOM [e.g. to identify formatting changing edits]
JamesF: found that bolding and italics were rarely used, so we pushed them down
feedback buttons: large part of feedback was about content of the article
Abbey: that's just a problem of survey framing
Kaity: need opportunity to provide feedback about particular task, not general
Lila: yes, don't need to solve it now though
Abbey:
slide 15
Guerrilla testing basic editing tools
found that people confused x button with something destructive
arrow for next, check mark for save - these icons confused people (they are used to "save" "next")
Lila: when you do this kind of change, take it across board, include in style guide
JamesF, Trevor: yes, we killed the icons elsewhere too

slide 16
Abbey: test to compare both tools for same users
Lila: I know how to add citation in wikitext, but was not able to do that in VE - did not connect to mental model I already had
might be different for new person
prompt people as they type - "where did this information come from? could you provide a ref?"
Abbey: got feedback from one user who really hated VE, he showed us why: difficult to do disambiguation edits. but he wasn't aware of recent improvements
keyboard shortcuts very useful for some, but VE is mouse and click only
JamesF, ErikM: actually, VE has keyboard shortcuts too - in fact more than the wikitext editor
Abbey: ah, that user wasn't aware of that
JamesF: OK, so that still means VE failed here
Erik: we already have that slightly annoying first use dialog
opportunity to offer guided tour
[discussion about possibilities to offer new users guided tours for VE]
Lila: let's not go too much into this now

slide 18
Kaity: I do design iterations, focus on common edit tasks

slide 19
people did not know intuitively, so clicked through all tools from left to right
so move more important stuff to the left
found a lot of confusion around linking
users are often confusing "edit source" with adding references (sources)
on mobile, it's different: one edit button + switch

slide 20
I did a test on linking
people see links as just words, not aware of linked page
(shows usertesting video)
hard to link word to differently names article
people then go to citation tool
making external links is hard
demo design mockup (we use these to test and iterate quickly)
link dialog shows small thumb + excerpt from articles to be linked
Lila: does it auto-suggest while I type?
Kaity: yes
Lila: goal?
Kaity: being able to link exact title and differently titled article
and change existing links
Lila: quantitatively?
Jared: We don't have instrumentation for that yet
Dario: having to type in something was my first usability blocker
Lila: who in the org owns tooling for this?
Erik: tooling exists: Eventlogging
JamesF: it's still live for clicking edit, save
Trevor: but no one does this for other things
Erik: the technology exists though
Timo: From the user testing videos we can measure the time spent. For larger data sets, we can instrument the application in production using the EventLogging infrastructure.
JamesF: sometimes broken, pinged Ori
Dario: Analytics dev maintains Eventlogging infrastructure, monitoring
Research & Data team can help with instrumentation
but actual implementation done by teams themselves, shared best practices with teams
Damon: (my preliminary thinking:) need role on each team for looking at funnel points, find features that are clearly broken
Erik: could be learned by someone on each team
Trevor: yes, but takes them off some other task
Lila: someone who owns success or failure of feature
JamesF: currently have 100s of MBs of data about VE [but no time to analyze it]
Lila: if you need, take off 2 weeks and stop working on other things just to look at this
Roan: I put in instrumentation points, but that's step 2.
step 1 is to figure out where to put it, step 3 is analysis
Dario: we [Analytics] can do that lower-level instrumentation for you
Damon: there are data scientists who can do both low-level implementation and later analysis, they were really useful [in my previous job]
Lila: e.g. do we have a higher save rate in VE vs wikitext?
Erik: should consolidate (current plans)
Lila: avoid flying blind
Trevor, Abbey: not entirely blind- we do have qualitative data. but agree we need quantitative data too
Lila: qualitative analysis is valuable, of course. quantitative is scaling better though
I'm willing to wait until we figure this out, but we need to do it

slide 21
Kaity: mobile
Lila: my guidance: spend more time on mobile than desktop
Trevor: did a lot more conversion work in this q, now develop much for both
JamesF: not total feature parity, 90% desktop..
Lila: use cases for mobile and desktop (and audience) might often be separate

slide 23
JamesF: added IE support
if your browser is not supported, tells you in preferences [where one can activated VE]
and, doesn't show [VE] edit tab

slide 24
(demo auto-fill cite dialog, paste in a NYTimes link, auto-fill fields)
Roan: it knows NYT, but not BBC ;) needs lots of fixes

slide 25
JamesF:
right now in wikitext, terrible but works
Parsoid built something, need to give them feedback
Roan: also, convert template fields to VE - but don't want to have toolbar for each
Trevor: needs some UX
Roan: also for other situations where there are lots of mini VEs on one page
Gabriel: and need to remove from DOM

slide 27
implemented compression etc., but didn't yet get enough impact
Damon: do we have a list of roundtrip[?] perf?
Trevor, JamesF: not quite yet
Damon: interested in "press key" --> "see text change" times [?]
JamesF: it's much faster in Chrome, btw
no hooks in our code
Roan: not quite (there are some things that slow typing)
Gabriel: is typing perf what users complain most about?
(team:) when it happens, it's really annoying
Damon: native code means what?
JamesF: browsers' own code that we have no control over
JamesF: demos labs tools for widgets

slide 30
re-enabling: postponed

slide 32
graph with percentiles (desktop and mobile, but skewed to desktop due to volume of edits)

slide 33
top line (percentile): can take 16-25sec
Lila: why did perf degrade?
JamesF: we don't know
Trevor: we do most of the work during edit summary dialog, so if one spends time on that, it doesn't take as long
Jared: how about pre-saving more granularly?
JamesF: indeed, most changes only touch one part of article
Damon: ...
Lila: need instrumentation so we see what causes problems
JamesF: some pages are 8MB of HTML
Erik: also need better testing environnment, which we've been working on for MV
e.g. connection quality, network lags around the world
Roan: page size/complexity and connection speed are the 2 main factors
Damon: "simple" can be measured
Jared: ...
Trevor: re incremental saving - no work on that yet, but have been thinking about it. these things are already roadmapped

slide 34
JamesF: differences between projects
Damon: ...
Jared: looked at some with Kaity

slide 36
JamesF:
collaboration with Wikia
they do their own thing
(demo)
Damon: why does theirs pop up so quickly?
JamesF: ours would be as quick - this is a very small page
we talk to them most weeks, but they don't submit as much code these days
Damon: anybody else submitting code?

slide 37
JamesF: Yes, PLOS!
they paid dev
upstreamed code for table editing
(rushes through table editing mockups)
we have not decided about this yet, might do some user testing before deploy

slide 49
demo auto-fill citation
Lila: what is the error message if website is not recognized?
Roan: gives just URL in that case
Lila: but did not see alert to user. should then ask users to fill out
Kaity: working on this

slide 50
JamesF: other than that, our focus this q is perf
also, need to be able to show it
technical rather than UX speed (might expand to that later)
Erik: bounce rates
Kaity: does that include mobile? yes
JamesF: now have single frontend widget library
Lila: value for user?
JamesF: consistency
Erik: even for gadgets and user scripts
Lila:...
JamesF: this q we'll convert probably 90% of our code, less of the rest [gadgets and user scripts]
Jared: it's affecting VE, but people from my team work on that too
May and Jon

Asks

slide 53
JamesF: principal area of concern is Analytics
also, some bus factor and hiring issues in team
e.g. I am the only product person
community engagement support: great team, but not enough capacity to support large deploys because of other work
Dario: need to break Analytics down: we can offer research and instrumentation support, and dev
Jared: ...
Erik: VE as test case for full instrumentation
Damon: for Firefox for Android, we actually sent teams out to interact with users (10 locations, 15 in team)
Abbey: here, we are only just building team :/
Damon: this is blue sky thinking of course

Lila: don't see strategy for mobile
is this team going to own editing experience for mobile?
JamesF: yes, we own that
Lila: I'm onboard with the objectives
next time we should have more specific [?] goals /progress