Wikimedia monthly activities meetings/Quarterly reviews/Research, Design Research, Analytics, and Performance, July 2016

Notes from the Quarterly Review meeting with the Wikimedia Foundation's Technology I: Research, Design Research, Analytics Engineering, Performance teams, July 14, 10:00 - 11:30 AM PT.

Please keep in mind that these minutes are mostly a rough paraphrase of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Present (in the office): Dario, Ellery, Leila, Michelle, Ori, Tilman, RobLa, Jaime, Madhu, Grace, Katherine, ; participating remotely: aotto, Dan, Darian, Faidon, Gilles, Greg, halfak, Jonathan, Maggie Dennis, Marcel Ruiz Forns, Mark, nuria, Samantha, Wes

Backup Datacenter

Wes: Great rollout. We had to make adjustments, moved out a quarter do be prepared and do this well. Good learning and improvements.
Katherine: reiterating last session; huge accomplishment for the team, community members were very positive; confidence in the results; board appreciated the work

Research and Data

Objective: Revscoring in production

Dario: first goal, ORES

most important milestone for this project. Both service (API) and client interface in the production cluster and available to users. Big thanks to Amir Sarabadani and others from Release Engineering, Operations and our volunteers.

Objective: Revscoring in production - Successes and Misses

Dario: related updates on ORES. Wikimania session to socialize the changes in ORES by Amir.
Dario: 14 new models, substantial performance improvements

Objective: Discussion modeling

Dario: Also related to the application of machine learning methods. Discussion modeling project (Detox) has the goal of providing insights into talk page discussions when it comes to interactions that may drive people away. This is a collaboration with different stakeholders, the community department, research-and-data, and Jigsaw. We have the first results of it available this quarter. Data-set released, for now there has not been an announcement. We will make an announcement one all diffs are scored by the algorithm. Ellery also did an analysis of users that are blocked to understand why users are blocked.

Other successes and misses

Dario: It divides in 3 steps. Understanding harassment, designing interventions, [what is the third step? --Leila]

Objective: Research outreach

Dario: hosted 2 pretty major outreach events. Co-organized by Stanford University, Wiki Research in WWW and ICWSM conferences. We also cohosted WikiCite with WMDE. A lot of effort for doing these outreach. The results have paid off for all the effort that we put in for organizing them.

Learnings: how to organize funding we get for events such us this one? We had no network for the events we organized.

Other successes and misses

Dario: Wiki workshop largest concentration to date of Wikimedia research scientist. A report for WikiCite for the community and the funders in the coming quarter.

Objective: Reader segmentation

Dario: Carry-over goal. Concluded that this was a miss, as not yet completed.
figure out next steps, still a lot of interesting results that could be drawn from this data
Katherine: partnership with Reading team?
(Dario:) yes, started as collaboration on technical level, in past 2 quarters shared findings with them

Other successes and misses

Dario: Wikistats 2.0. Had been maintained quietly by Erik Z, but he'll focus on other tasks now. As of this quarter, the official maintenance of wikistats is handed over to the Analytics team.
Dario: Wikipedia navigation vectors data release which we expect to increase the research on understanding Wikipedia readers and logs.
Dario: published Research FAQ after request by FDC

Core workflows and metrics

Dario: we did not host a showcase in the last quarter. You should see them coming back and happening on the regular basis starting this month.

Appendix

Katherine: workshop (about harassment at Wikimania[?]) went really well. q: other than attendees, what do you see as outcomes from that? conversations, research, projects?
Leila: conversations around three projects. empowers when we want to start other projects, people get together and they may start something that comes up a year from now. we can announce maybe 6mos year from now. hard to quantify concrete outcomes of such workshops, but the relationship built are really important, enable collaboration
Dario: we org dev summit every year. research community is as important as developer community. tighter integration between researchers' priorities and community needs
Maggie: we heard from German community just last week that harassment research project is helpful work
Katherine: appreciated documentation, so people know who is doing what in research. I know that's not nearly as fun as working on projects, but it's important

Design Research

Jonathan: Filling in for Abbey who is in a workshop.

Objective: Personas

Jonathan: not completed for several reasons: most importantly loss of partners in org, in particular Kaity (design) also, had to make room for New Readers research (Mexico etc.)

Personas

Jonathan: going forward, emphasis on integrating these personas into product research and design (rather than creating new personas or refining existing ones)

Objective: Evaluative Design Research

Evaluative Research

Screenshot credit: Pau Giner

Objective: Deep dive / contextual inquiries

Collaboration with Reboot; we're prioritizing getting the information published

New Readers Contextual inquiries

Each deep dive took approx 2 weeks

Objective: Collaboration with UW on Survey

research prompted by questoin asked by Trevor; how do people learn? collaboration deployed to students to find how they find informatino. summary provided at last month's metrics. this was design research's first

Collaboration with UW on Survey

noting UW's publication of data

Objective: Benchmarking / Tooling

Benchmarking / Tooling

new user testing platform.
Jonathan: relationship with previous vendor was not working for several reasons. primary goal was to get something in place to meet product team's needs right now. Re-scoped to just securing a contract; meets our current needs and we think our future needs.
Wes: thanks to Legal team
Jonathan: Manprit and (?)
Jonathan: we have 3 seats for 3 researchers. we can use this for any research, not just 3 verticals. our previous contract
Jonathan: we're phasing out some of my old responsibilities; Chris Schilling taking over some of them
Katherine: thank you

slide 25

Analytics Engineering

Nuria presenting
our team : Madhu is transitioning to Ops. We work quite a lot with ops
we use velocity as KPI, went down during this q - several of team took vacation, Wikimania

Objective: Public by default

also track data by country, but only release aggregate publicly, for privacy reason

(data)

Nuria: WIkistats transition (old version has been up for the better part of a decade) so far relied on XML dumps

Objective: Wikistats 2.0

Public events stream

Druid (screenshot)

Nuria: tool for e.g. some people in Reading (JonK, Tilman), much easier way to consume our data, queries takes seconds in Druid instead of minutes in Hive. WMDE loves this too
Dario: for Wikidata?
Nuria: yes
Dario: e.g. Russia is top traffic source for Wikidata

Objective: Better Data Access

Objective: Operational Excellence

our team has big operational (component), e.g. Cassandra scaling issues

Other successes and misses

Nuria: Varnish/Varnishkafka upgrades
Nuria: Launched http://analytics.wikimedia.org : one place for our tools
Michelle: I'm so impressed with your emphasis on privacy. curious about API response rate. Practical impact on API users?
Nuria: impacted by cache. only heavy users of the tools. those people see the impact. overall if you request data, you aren't hitting storage
Dan: most people accessing via PageView tool by MusikAnimal, he's handling from community view
Dario: it was all really great this quarter. please demo Druid
Katherine: making unique dataset public is fantastic. have it for all projects?
Nuria: yes, did it originally for enwiki, but then for all. Also projects that are too small to yield
Katherine: overall de-duplicated?
Nuria: yes, it's something Reading/Tilman have requested. investigate performance/traffic considerations of additional cookie https://phabricator.wikimedia.org/T138027
Katherine: is that a goal currently?
Nuria: not a goal this q, perhaps next q, depends on ops/perf feedback
Katherine: recognize it's not something we would use in product decisions, but eg important for board (important vs urgent vs informational)
Nuria: banner consultation[?]
Katherine: interest to external audiences

Dashboards and Data analysis

Performance team

KPI: First paint time

Ori: theory: traffic moving to mobile, but don't know

KPI: Page save time

Ori: regressed during quarter. one major regression (Authmanager)

KPI: Page save time

Ori: graph that goes to today; we're actually better now

Objective: Thumbor

Ori: tight coupling to mw. goal was prod deploy. we have VM deployment. team decided to package for Debian. coming along quite well. Filipo provided enormous time and assistance

Objective: Performance Inspector

Ori: not deployed; asked Peter to help Reading team with lazy loading. looking good for next q

Multi-DC

Ori: talked a lot about that earlier.

other successes and misses

Ori: Running MediaWiki from secondary datacenter without writing. reducing slave lag (read slide)
Ori: Aaron Schulz made optimistic saving work for majority of edits
Ori: expect substantial impact of inlining CSS on first paint time
Katherine: so optimistic saving is live?
Ori: yes
Katherine: slave lag: that's in addition to multi-DC work?
Ori: yes. don't want to serve stale content, this imposes a lag tax
Katherine: does this (metric) mean team will work further on this
Ori: it's a huge win already, but.. we've made a lot of progress in past 3 mos. 5 secs of slave lag still a problem for modern web application
Dario: as top KPI?
Ori: it's a longer bet, switchover to Dallas was one of biggest payoffs of continuously serving from both DCs is that caches remain warm in future, should allow automatic swtichover improve performance e.g. in south America (closer to Dallas) but bigger payoff: can open more DCs in this way
Katherine: interested in progression on lag time
Ori: mostly social rather than technical. a lot of stuff exclusively Aaron Schulz. this is going to be problematic. needs to be widely understood in org, otherwise this will be difficult to work on. Aaron has been collaborating with Stas, documenting the work on mw.org; ideally we would reallocate
Katherine: when we first started; paint time increases may be transition to mobile. in next q, do you feel like you have resources to do what you want/tradeoffs?
Ori: work that Timo is doing will make substantial impact. but even if metrics substantially improve by eg. including CSS next q: it will be great to have compensated for that, but will still be not fully satisfactory; doesn't answer the question why regression happened in the first place. don't feel adequately resourced
Katherine: what is your purpose for next quarter?
Ori: jumping between projects; drop in project management
Wes: Ori, thanks for pushing forward performance and making the site more efficient; heard lots of compliments from other teams about your help supporting their teams.
Katherine: I really appreciate time we have together. helps me understand arc of the org, appreciate learning since this isn't my core experience, so thank you very much

General

Katherine: comment about presentation structure: makes sense to rotate order so VE doesn't always get smushed at the end.