Wikimedia monthly activities meetings/Quarterly reviews/Research, Design Research, Analytics, Performance, January 2016

Notes from the Quarterly Review meeting with the Wikimedia Foundation's Technology I:  Research, Design Research, Analytics, Performance teams, January 22, 2016, 8:00 - 9:30 AM PT.

Please keep in mind that these minutes are mostly a rough paraphrase of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Attendees: Dario, Kevin, Chris, Leila, Faidon , Guillaume, Lila, Ori, Greg G., Madhumitha, Abbey, Stephen LaPorte, Abbey, Rob Lanphier, Trevor Parscal

Remote Attendees (BlueJeans attendees): Tilman, Aaron Halfaker, aotto, Dan, dchen, Faidon Liambotis, Gregory Varnum, Joseph Allemandou, Luca Toscano, Mark Bergsma, mforns, Moushira, Nathaniel Schaaf, Nuria Ruiz, QuimLaptop, Samantha,Wes, Chase Pettet

Presentation slides from the meeting

Research and DataEdit

slide 1

Slide 1Edit

Dario: This quarter was really important. Many initiatives that came to fruition and depended on a large group of contributors and fellows.

slide 2

Slide 2Edit

Dario: Request to clarify who owns what in data and research. Talked to team members to clarify roles and streamline ownership. I've made a list of responsibilities ("Research fingerpost") to see who to go to for what. The map is published on the office wiki and would like to move to a public wiki. This isn't a replacement for individual team processes

slide 3

Slide 3Edit

Dario: Goal was to bring revscoring to users as Beta Feature. Missed because volunteer time became scarce.

slide 4

Slide 4Edit

Dario: Achieved actual 90-99% reduction in curation workload. This was a massive gain. Nice blogpost on Wikimedia Germany's blog.

Dario: Communications team announced revision scoring. Received positive community reaction and good media attention.

Lila: Amazing work. How does this connect to previous slide?

Dario: Service is available in many languages via the API.The goal of enabling it to production as a beta feature is to come up with standardized way to integrate it into the user interface (e.g. Recent Changes, etc.)

Lila: Is this an official goal?

Dario: not an official goal but confident we should be able to release in Q3

Lila: This is the first show of what is possible with machine learning. Allows editors to focus on important tasks. Really excited about the capabilities of this.

slide 5

Slide 5Edit

Dario: Productization refers to underlining service. We started seeing the first third party tools adopting the service.

slide 6

Slide 6Edit

Dario: Saw up to 16% of all saved translations come from article recs. Good confirmation of early results.

Lila: how many are there?

Leila: About 280??

slide 7

Slide 7Edit

Dario: This is partly a miss because work that went into other projects. First part was presented at Wednesday's research showcase. I would encourage everyone to watch the presentation.[1]

Dario: Exploring working with other service providers.

slide 8

Slide 8Edit

Dario: First time we invested into this kind of research . We are trying to understand who our readers are from both a qualitative and quantitative perspective.

Dario: Second part of project is starting now. Had a lot of contributions from the reading team.

slide 9

Slide 9Edit

Dario: Lila asked if we could come up with template to make services replicable. Came up with first draft of proposal.

Dario: Hired first dedicated software engineer, Nathaniel. Very competitive position.

slide 10

Slide 10Edit

Dario: 3 papers and 1 peer reviewed poster accepted at major conferences in the field. 3 workshop proposals accepted. Trying to bring results to fruition and engage with academic community to secure new collaborations. Expanded on last year's success of a dedicated Wikimedia workshop with 50 attendees and 21 submissions.

slide 11

Slide 11Edit

Dario: Worked with Discovery Team to support Big English fundraising

Lila: Look at the impact your team is having on users and so grateful of what the team has been able to accomplish. So grateful

Kevin: Do you still get data analysis requests?

Dario: we do; sometimes we provide some advice, but it's nothing compared to requests in previous quarters

Robla and Quim: It would be great if everyone is as active as the research team is in conferences.

Design researchEdit

slide 12

Slide 12Edit

slide 13

Slide 13Edit

Abbey: Realized as we tried to coordinate with project teams that people did not understand about about process. Decided it would be good to do a little design thinking and try to understand other teams. Conducted workshop where we laid out generic product development process. All of the people in the workshop from various disciplines the outlined all their activities as part of that process.

Abbey: About 40 people attended.

slide 14

Slide 14Edit

Abbey: There were 4 workshops total. Talked about programatic personas and inspired people to use those as they build products.

slide 15

Slide 15Edit

Abbey: Question if this should be red or green. Started prototyping lab at the end of December. Going to have more regular sessions of the reading team to understand users. Since we are a small team, want to rely on mentoring and collaboration with other teams to broaden our reach.

slide 16

Slide 16Edit

Abbey: We need to talk to the people we are building for. It is important for the team to have a dedicated person to find the right people and schedule them. Made a lot of improvements to process this quarter.

slide 17

Slide 17Edit

Abbey: Worked on integrating tools - database and mass emailing tools. Worked with IT to navigate through that. Abbey: Made efficiencies in how we communicate with participants. For example, May made a gif on how to screenshare on google hangout. We are also using talk pages and querying databases to find the right participants.

slide 18

Slide 18Edit

Abbey: Working on integrating design research into how we build products. Conducted 11 research products. Need a lot of QA.

Lila: Within the product team?

Abbey: Yes. This may be more of a conversation regarding when something is ready for usability testing. How will be people know when something is ready?

Lila: Quim and Wes - can you make a note? (offline note by RobLa: I have an idea or two on how to resolve the QA problem Abbey brought up...feel free to ping me)

slide 19

Slide 19Edit

Abbey: Did 5 heuristic reviews and some focus groups to understand users needs?

slide 20

Slide 20Edit

Abbey: Have a set of programatic personas built from gathered secondary research. Workshopped them with engineers, designers.etc. Iterated them from that workshop and used them to help build products around users. Now we are doing deeper research around users. Conducted about 5 interviews.

Also, working on the recruiting process -important part of the work. We are continually learning about users to understand their needs and how they navigate around software we and the community are building. When we go to Mexico in couple weeks, learning about users in person and how technology is part of their social fabric.

slide 21

Slide 21Edit

Abbey: Going to do a survey of higher education to understand how they learn and tools the use. Working with them on a weekly basis and developing relationship with this organization. This is another way of expanding reach.

slide 22

Slide 22Edit

Abbey: All participants have a point person to reach out to if they have questions.

Abbey: The design team is also working with the L&E on implementing their surveys and mentoring on how to do design research. Step back only once team is confident that they have the right tools.

Abbey: Delivered several presentations this quarter. We don't go to conferences at rate that the research team does but would like to in the future

Wes: Really great work this quarter.

Lila: The way you built esteem and input into team has been great - so impressed. Need to make sure every engineer understands design thinking. How do we keep that going? There are some things that have enormous impact on product. Pick one flagship product and use that to understand how users really engage with learning. Part of changing internal engineering thinking is how it is perceived externally. Let's talk about it.

Guillaume: Regarding the work with U of W, students are conducting surveys that will inform possible decisions in editing.

Lila: Those are the things that are important.

Dario: Really want to see if you can take role in consulting/mentorship and to carve out time to figure out what products need support.

Lila: Would like to train product teams.

Abbey: We're luck to collaborate . It's a unique situation we should be able to take advantage of.

Abbey: Bringing two engineers to Mexico. Want to collaborate toward process that we all think will work.

Robla: There's a cultural norm of people not doing code/"real work". We need to remove that barrier.

Wes: There is a nice momentum between design and research team.


slide 23

Slide 23Edit

slide 24

Slide 24Edit

Madhu: Pageview API was a big ask and a carry on goal from the last quarter. The team met the goal - APIl was launched and blog post released. Also had a number of requests come through phabrictor and received some great community feedback.

slide 25

Slide 25Edit

Madhu: Scale is logarithmic. Graph show that the per-article endpoint gets several orders of magnitude traffic compared to the top endpoints,

Lila: External or internal?

Madhu: mostly external

Lila: Wes - Is this API usage presented part of those numbers?

Wes: No


Slide 26Edit

Madhu:Erik Z has been maintaining.  Monthly pageviews with breakdown reports have been updated and are available.

Lila: is the URL still the same?

Madhu: Yes

Lila: Great

slide 27

Slide 27Edit

Madhu:As of now this is done, but missed the goal as of the end of the quarter.

Lila: It would really help..

Kevin: 800 hours saved using wikimetrics. This will greatly simplify.

slide 28

Slide 28Edit

Madhu: Stretch goal was done by services team abut the deploy freeze kept new code being deployed before the end of this quarter.

slide 29

Slide 29Edit

Madhu: Much better docs for users doing self-service analytics. Now Close to over 400 million devices?? [Madhu - can you confirm]

Lila: Just English?

Madhu: Yes

LIla: Would like to schedule meeting to discuss more. (Lila asked Wes for this)

slide 30

Slide 30Edit

Madhu: Purple on top is the production hdfs user. Ellery runs a close second - runs a lot of jobs for fundraising. 30 uses running jobs on the cluster in December compared to 15 last June. A lot more users.

Lila: We need to think about capacity.

Kevin: Will need even more servers.

Lila: Great progress.

Dario: Would like to stress the level and quality of documentation. Pretty remarkable and want to thank you for that.

Lila: Want to thank you for all of the work you have been doing - you are the instruments the product team is using to navigate. Great work!


slide 32

Slide 31Edit

Ori: Unable to calculate the percent difference - there were no reliable metrics for several weeks.

slide 31

Slide 32Edit

Ori: We did not meet goal because we kept finding low hanging fruit - hard to anticipate how much we will be able to accomplish in a quarter. Should have set a more modest goal in hindsight.

Ori: There's been a lot of work by the ops team on the caching layer. The blue line is backend response time and green line is the user-perceived response time.

Lila: we use our own CDN ,right?

Ori: Use varnish but a lot of our own custom code.

Ori: Ashburn, Amsterdam, SF, Dallas,

Lila: Think about distribution next year?

[Unknown]: Going to include in annual plan which we are working on.

Lila: Need to think about splitting the measurement.

Ori: Its fairly stable over time.

Lila: Think we need to introduce first to read? metric for performance.

Ori: This is the time anything appears on the browser.

slide 33

Slide 33Edit

Ori: Availability project. Some systems are born distributed; others become distributed; others have distribution thrust upon them,

Ori: All the high level things we set out to do are done but can't say we can flip a switch. There be a lot of bugs resulting in unknown additional work. Will work with the dev ops team.

Ori: The team is still responsible for lion's share in Mediawiki - Over 50% of all commits - largely due to Aaron and Timo

slide 34

Slide 34Edit

Ori: Image manipulation set of requirements is complicated . Proposed solution is to move out of MediaWiki so people can run locally. Want to get it formally vetted to make sure we are good to beign rollout next quarter.

Ori: A little tricky because there is a dependence on varnish 4. Tech ops has reviewed. There is block but they hope to rollout soon.

Ori: Put a lot of work into job queue.

Lila: Can you explain the bottom of the graphs?

Ori: We currently have access to actual Iphones and androids that are submitting metrics to us. Mobile performance still has a way to go.

Ori: For the next quarter, both Petr and Timo will be collaborating with reading team on base load of images.

Dario: Many practices that editors follow. Many changes that are invisible to community that we could advise upstream editors on

Ori: Onwiki performance inspector is a major goal for next quarter. Planning to have a dashboard that will appear and show a lot of the performance data. This will give editors a sense of how article will load in different parts of the world.

Lila: Should make our editor tools smarter. Lets start thinking about editor's experience. Conversation with James will definitely be helpful.

Lila: Great session. Very informative and great results. Thanks everyone.

Wes:Split into two groups so we can go through review in regular pace seems to have worked out much better and we should continue this format.