Wikimedia monthly activities meetings/Quarterly reviews/Infrastructure, July 2015
Please keep in mind that these minutes are mostly a rough paraphrase of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material
Present (in the office): Terry Gilbey, Dan Duvall, Chris Steipp, Grace Gellerman, Dario Taraborelli, Katherine Maher, Lila Tretikov, Kevin Leduc, Madhu Viswanathan, Geoff Brigham, Gabriel Wicke, Tilman Bayer (taking minutes), Abbey Ripstra, Daisy Chen, Leila Zia, Garfield Byrd, Stephen LaPorte, Kim Gilbey, James Forrester; participating remotely: Marcel Ruiz Forns, Yuvaraj Pandian, Aaron Halfaker, Arthur Richards, Mark Bergsma, Jonathan Morgan, Faidon Liambotis, Dan Andreescu, Chase Pettet, Chad Horohoe, Joseph Allemandou, Ori Livneh
Kevin: had one goal:VE
dashboard's pageviews = measure of usefulness
Terry: do we know how much it cost to build?
Kevin: started in Q3, could look up #points... but maybe 1 person for 2 months
core workflows and metrics
EventLogging: measure of success should be capacity, increased that
now can handle well over 1000 msg/second
Terry: incident defined as?
Kevin: data lost
Ori: clarify 6x perf increase means ...?
DanA: consumer itself got faster by that factor. See appendix for a link to the test.
before that, data was dropped at slower rates.
Kevin: reduced failed Wikimetrics reports
Terry: what's the value of that?
Kevin: shown in mailing list feedback: number of ppl who report problems about failed report and need support
Release Engineering edit
Dan D: I'm filling in for Greg, who can't be here today
released MW 1.25
deployment tooling: avoid wheel-reinventing
hard to give metrics since it was all research, but have a plan now
staging cluster: closer to production, but not affected by side effects
team formed mid last year, lot of siloing
so worked to share knowledge more between us, via pairing sessions
also helped morale a lot
about 1 person on each project, perhaps 2 for staging cluster, but needs broader participation
Terry: staging means?
DanD: relates to CI too, more deterministic deployment pipeling
Terry: so decision driven by resources, or complexity?
DanD: resources, remaining work seems straightforward
Terry: still think decision was right? yes
Chad: and work from this q was not in vain, e.g. already reduced some tech debt in puppet etc.
so it's not dropped forever
Terry: so just paused? yes
Gabriel: top goal was support for mobile VE
section editing (just retrieve section instead of entire page)
built that API, but VE team disovered they need additional metadata e.g. for refs (which rely on entire page)
remove metadata: VE was not ready to consume
streamlined services deployment using Docker and Ansible; reduced code deploy steps from 11 to 3 by automating rolling deploys
working with Rel Eng on more general solutions, this might be something they could use
created a new /api/rest_v1/ entry point in the main project domains, which brings big perf improvement for new requests by avoiding DNS lookups, separate HTTPS handshake
Terry: resource issues for mobile VE
VE team has 3 engineers now, busy with rollout, bug fixes, not a lot of time for new things (even though they may have anticipated)
Dario: I still don't understand why we devote a lot of resources to mobile VE
Gabriel: API is independent of VE, can be useful [in other ways]
Lila: starting to break up article structure in general
assume VE best for new editors, and new eds are on mobile
Terry: shifted resources between parts of org, might need better coordination (on goals setting)
Terry: sounds we spin up stuff in one part of org without [perfect synchronization in other parts]
James: IIRC this was stretch goal in VE with hard dependency on Services. Services did great
Technical Operations edit
e.g. people outside Ops can/could not easily pool servers
and even for Ops people, workflow is smoother now
had discussion with Services
goals process used to be done a bit last minute, with only one goal, which didn't suit us well. now better
had actually thought uptime goal was not very ambitious
for most of quarter, we hit it
but then had a major almost 2 day outage some weeks ago
a lot of improvements, but not enough, because of remaining problems with underlying layers in Labs
trying to make it more redundant actually impacted perf negatively (some decisions should have been made differently)
backup of large filesystem takes weeks on NFS, I (Mark) insisted on doing 2 backups (which saved us in case of the large outage), but means that latest backup is always old
Terry: had a SPOF here, had a SPOF earlier because of systems being in same cabinet
do we have SPOF analysis across systems?
Mark: per cluster yes, not for all systems
design was to have hot spare in place, but that too added to complexity
some bad design decisions were made, not a lot of ppl involved
Lila: alternative storage mechanisms?
Mark: yes, when I started working for WP 10 years ago I removed it [NFS] everywhere I could
Lila: might want to look at specialized hardware
Mark: assumption here [Labs] was that it's a (simple use case)
also thinking about offering different storage solutions to users
Lila: need to think ahead, e.g. in case we'll ...
do we have standby for NFS?
Mark: this was actually already redundant..., high availability setup
Lila: is it hot-hot?
Mark: it is hot-cold
Lila: I know, that setup can create more problems itself
Ori: had one goal only because team still being put together
focused on page save time this q
1.5 years of waiting time saved every month
--> Lila: please send me that slide together with what you just said [about waiting time saved/month]
wish I had some project management support(?)
Research and Data edit
successful communication about that VE test
still analyzing implication of French WP test (on article translation recommendations), but report is complete
revscoring also feeding into Q1 goal
Lila: what is the tangible effect on editor or reader of this?
Dario: internal customers are Search - surface quality of articles
for Editing team - surface quality of edit, e.g. revertable ones [in e.g. recent changes page]
Lila: next time explain this in a tangible way so it's understandable (eg. in case of article creation recommendations I understood it lead to an explosion of new article creations)
Aaron: have two on-wiki tools based on rev scoring that help ppl combat vandalism, and bots using it
Lila: quantify e.g. how much time it takes to process edits
this is amazing work, but I need to [be able to demonstrate it quantitatively]
--> Lila: quantify value we get from revscoring
Dario: could produce separate interim report about this
DanD: why does it need to have numbers? I understood it fine
Abbey: how about quality of experience for editors, that's important too
Lila: absolutely, it can also be stories to illustrate quality
Dario: team does a lot of end user affecting work, needs communications support
VE was probably most important research project of year, and a complex one
one of the largest ever WP research workshops at ICWSM
Lila: Stanford project is about what?
Leila: 3 students extracted facts from Wikipedia, the goal is to digest to Wikidata
will talk about this in July research showcase
Lila: evaluate quality?
Leila: yes, pretty good
--> Lila: Dario: define pipeline research -> productization
reorg was particularly taxing for Research team
still not completely settled
--> Lila: for all teams: assess how much of your time goes to:
- supporting others
- new projects
- prototyping / research
want to see a pie chart at end of q
Lila: great work
Dario: yes, also comms legal next
Terry: so that's a menu of services you provide?
Dario: pretty much, also self-service for some
Design Research edit
Abbey: obj 2
refocused because of lack of funding for software
focus where we have survey data and academic papers to work from
migrated from Trello into Phabricator
Lila: list of Flow workflows we are investigating?
Abbey: not list, but examples. Jonathan?
Jonathan: just wrapped up interviews this week
about 12 different workflows, e.g. AfD, dispute resolution, deletion on Commons
more at Wikimania
Lila: that's great, I've been asking for this for a year
--> map superset of all editor workflows (everything we know of, does not need to be quantified / tested), and the subset that we focus on [with more research data]
Lila: there are so many things the current system does (because it can do anything), chip away, can't improve / substitute everything at once
to Aaron's comment on pipeline: a lot of research will lead not to productization, that's ok but need to map it still
new team: me and Darian who joined in May
released risk assessment - if you want to scare yourself, read it ;)
metrics: basic problem: "when we do our job, nobody knows"
core workflows and metrics
Terry: how bad is backlog?
Geoff: you work closely with Michelle on Legal, understand it's a good collaboration
Chris: yes. several people worked with me on privacy, takes up lot of my personal time. I think org needs a more centralized privacy role
Terry: new hire should help with sec reviews?
Chris: yes, but should also have automated reviews, which were cut out of the budget
Terry: thanks everyone,
these teams are on the backend
small groups of people with potentially very large impact
Lila: and thanks for stepping up, many of you have stepped up into leadership roles