Wikimedia monthly activities meetings/Quarterly reviews/Editor engagement experiments/2013-04-12

The following are notes from the Quarterly Review meeting with Editor Engagement Experiments (E3) team on April 12, 2013.

Present: Dario Taraborelli, S Page, Terry Chay, Steven Walling, Ori Livneh, Howie Fung, Sue Gardner, Tilman Bayer (taking minutes)
Remotely participating: Erik Moeller, Matthew Flaschen
Not present: Ryan Faulkner, Munaf Assaf

Agenda

Short reminder of team's goals regarding editor engagement
Review of our plan for the quarter
Timeline of what we accomplished on a per-project basis with associated data, and discussion of successes, failures, and blockers encountered
Overview of our upcoming work from the perspective of the new user experience, discussion
Discuss projections for E3 contribution to Total Active Editors, by June and for the 2013-14 Annual Plan
Any final feedback or comments

Slides

Minutes

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Steven: welcome

team changes this quarter, e.g.:
Ryan Faulkner just left
Maryana moved away from E3 to Mobile

Goal here: move edits from 0 to 1st edit, then..
Productization of account creation and login changes
Getting Started
Guided tours

Review of goals

Account creation and login

Login changes were merged into core yesterday
Opt-in launch first
This is our first major project in core, took half month longer than expected
Mid quarter additions: review / support for volunteer patches, or new features (CAPTCHA refresh)
Ori: opt-in is for a limited feedback period, several weeks
Dario, Steven: clarify feedback is about UI elements like translations
Steven: This is first project in core which uses Agora design style (Translate extension also does now)

Guided tours

Reached all 3 goals: build and test one tour, productize and i10n, support volunteer created tours
Currently on 11 Wikipedias and Commons

Getting Started

Improved 0->1 conversion rates, but not after that. Launched replacement for SuggestBot, which means we're 1/2 way to being able to release GettingStarted on other wikis.

Eventlogging

Goals were:

campaign support
realtime monitoring and alerts (done)
support for funnel analysis and statistical testing
documentation (done)

Many teams in different departments use it already. Campaign support and automated analysis/testing deferred. Other accomplishments this quarter include migrating to dedicated log database, full puppetization, and adding support for server-side event tracking.

User Metrics API

goals were:

visualization, with analytics team
public release (done)
user tag repository redesign (done)

With Ryan F leaving, Evan is probably going to take up a lot of his work. This departure also necessitated accelerating work on stability, security, and collaboration with Analytics.

Dario: also going to be used for programs review
Sue: replacement for Ryan?
yes, JD being prepared

Review of plan, timeline

Steven:
January (around previous quarterly review): Getting started + guided tours
February-March: testing guided tours alone (lost some time due to faulty logging)
March: redesigned whole page

Big achievement during that period: completely replaced Suggestbot - this was maybe about half of the work toward productization
this is a minimal version of a recommender system, could do lots of more complicated stuff (if data justifies it)
Getting started increases first-time editors,
effect of adding guided tours: +3.9% in first 24h article edits (within cohort who went back to getting started page)
pretty clear positive effect on first-time editors, but not effect on further edit frequency (1..5 edits)
Dario: at time of test, did not yet have extended suggested articles list
Steven: too much choice in suggested tasks is detrimental
grammar and spelling fixes more popular task than linking or "improve clarity"
Sue: anecdotally, many users say they started with grammar and spelling fixes - it's a risk-free task
Ori: users don't want to leave a mess
Sue: this is reminiscent of conversations that Fundraiser team had
Steven: did much work on article selection, but not a lot of effort before launch to ensure tasks would be understood

Our contribution to total active editors?

conservatively, we can add 400-500 unique editors per month
(caveat: statistical significance of results so far)
Howie: concern was that additional editors drawn by GS+GT would have lower propensity to continue editing, but looks like not much lower
Dario: ..(?)

Steven:
recommend annual plan goal change: for total of 2400 more active editors per month through E3 by end of FY
Howie: http://reportcard.wmflabs.org/graphs/active_editors fluctuates in similar dimensions
correlation between active editors
Sue: seasonal effects?
Dario: known
Sue: retention also varies seasonally
Dario: yes, even by day of week
maybe even shouldn't launch important feature in summer ;)
Howie: slight decrease in number of registrations is already anticipated in above estimate
Dario: recall readership still goes up
Howie: incremental vs. cannibalizing? hard to determine because of fluctuation
Howie: in overall effect, how to separate new user acquisition improvement from other work for all editors?
Dario: have better tools for that now with Eventlogging

Short term roadmap

Steven: next goals
UX goals:

simplify task flows
from experience with new landing page: need understand user's decision making better
learn more about what best newbie tasks are
encourage flow state

for 0 -> 1s

(more) rapid prototyping of Getting Started
apply guided tours more proactively
campaign support (differ by how users were acquired)

S: do we know of guided tour usage by others?
Steven: nobody did comprensive tour that is deployed yet. Wikipedia Adventure project by Ocaasi is exciting.
Matthew: could do more outreach
Sue: anybody can use guided tours, without tracking, right?
Steven: yes
Ori: announcement of GT seemed well-received by community

for 1-5s

Steven:
rapid prototyping of GS
persistent link back to GS
use Echo notifications. first time we use email
on enwiki, some users keep coming back to GS
need to find better way of tagging their edits (right now still as new editors)

Sue: could Echo break user's "flow", by interrupting with notification?
Steven: disruption should be pretty minimal, e.g. welcome notification comes right after signup

expand Guided Tours
what if we give "1st edit" GT to all new users?

Ori: Many 0 edit people do stay on, but site kind of fades back into reader mode again.
Sue: One difference between us and Facebook: FB doesn't care what users do, goal is to keep them on site.
Steven: Echo needs action triggers currently, e.g. automatic notifications after 24h not possible
Ori: experienced editors have hacked notification tools, e.g IRC bots
Steven: ...e.g. CTA to register in semi-protection message - don't ask them to edit a page that they can't edit
This is also why we didn't extrapolate our conversion rate (above) to all editors.
Dario: could also filter out areas that are likely to create friction
Steven: e.g. removed BLPs from recommendation on community feedback - thanks to productization this was just ~1 day of Ori's work
re rapid prototyping: put wireframes / mockups / JS/CSS protoypes in front of users asap, e.g. on usertesting.com
example: remove article list from GS page, encourages clickthough to icon --> GT
people tend to read maintenance (cleanup) notices on top of article a lot; GT tells them they don't need to.
Steven: let's watch highlight reel of prototype user testing
Dario: could use this to test more direct CTA in cleanup templates?
Ori: possible regarding our infrastructure, but don't have capacity to analyze resulting data
Steven: Erik suggested doing such rapid protypes (e.g. language engineering team does this a lot)
Erik: what about highlighting game-like elements like "you have completed x edits"?
Steven: could make sense

Conclusion: reversing active editor decline is within reach

questions:

resources needed?
reduce distractions for team

Discussion

Steven: even with Faulkner in, we were stretched in analytics area. knowledge transfer to other teams (e.g. evan) has needed time
want more autonomy
(Sue and Steven discuss how much work is needed for onboarding new team members, e.g. designers)
Ori: only now hitting our stride in how to process community feedback
Steven: Dario's and Ryan F's work has had huge positive impact outside team.
(discussion about the changing Analytics landscape in general)
Steven: central design role in org might also be beneficial
Sue: targets look conservative to me, which is good. thinking ahead of new ED's first months...
can team give input to people who hire for positions with community involvement, on what kind of candidate is most likely to flourish there?
Ori: in our world, can take months to make tiny change, but then it stays and impacts many millions of people
Terry: there is no gradual path to community interaction
many product managers are from community, but e.g. designers not
Steven: lots of anxiety in community about possible inconsistencies / breakage introduced by design
Matthew: community feedback sometimes very constructive, sometimes not
Howie: does not think it's down to hiring, team needs to support each other
Erik: is optimistic that UX director position could address most of concerns regarding new design members and community feedback
Sue: even with short-term resourcing issues, if you feel we're on track, this sounds good to me
Ori: team still held back on analytics, not so on engineering
(more discussion about the changing Analytics landscape)
Dario: team is seriously affected by analytics needs
Steven: example from this week: User Metrics API down for a day, had much higher impact on E3 than on other teams who also use it (Echo and VE)
Erik: this is complex, the way API was developed first was not sustainable
Terry: Sue, your general input for team?
Sue: It all looks solid, recall time last year when team was a bit in disarray, really hit its stride now. In retrospect feels that that difficult "storming" phase was beneficial, even necessary
Erik: (on chat) Happy to see rapid prototyping
Steven: This review felt much less high pressure than first, since we're doing it the second time around. Frequent reviews with more free-form discussion feel more beneficial.