Talk:Experiments

Revised lead

I just want to say that I love the revised lead. --MZMcBride (talk) 23:15, 19 August 2012 (UTC)

Setting expectations

Thanks for starting this MZ. I think both interested community members and e3 staff could use this page to try and develop more common ground. If we are going to try and meet the entirely commendable goal of making experienced community members feel like partners instead of customers, then we need to talk more to try and get a sense of what kind of compromise we can make between editor needs to not be mucked with, and the desire to shake things up and try new ideas (a desire not limited to the WMF.) Steven Walling (WMF) • talk 03:39, 21 August 2012 (UTC)

A good portion of me is wondering how you all were able to start these experiments without first setting some ground rules. What's off limits? What's within bounds? What are the grey areas?

Twitter has started sending out these "what's happening on Twitter" e-mails to mostly (or completely) dormant accounts. Is it acceptable to spam users? It's certainly an easy way to get people to re-engage with the site (though sometimes in a very negative way). This is, of course, just one limited, specific example. There are thousands of variations of problems that experimentation can create when done on a volunteer community such as Wikimedia's.

I'd definitely like to see this page expanded and contributed to by all involved. And assuming these experiments are to continue in the months and years ahead, eventually I'd like this to turn into something more than a brainstorming page. We'll see what happens. --MZMcBride (talk) 04:51, 26 August 2012 (UTC)

Partly what you're asking wasn't done because the possibilities are endless with a brand new team and a fairly open remit. Currently we've done experiments separately targeting lapsed experienced editors, readers, anonymous editors, new registered editors, and people making their 1,000th edit. What is acceptable is partially based on who we're talking about delivering experiments to. In any case, what we have done is write the FAQ that's posted on English Wikipedia, commit to working in public, and announce experiments publicly. We are currently trusting that our own common sense as fellow Wikipedians (there are more editors than non-editors on this team), the input of Foundation leadership, plus public feedback on our work from the community will keep us from turning into Dr Frankenstein here. ;) For example: I also hate those fucking emails from Twitter. Wikipedia currently has enormously higher rates of people who open and click on stuff in email we send to them, and we would like to retain that goodwill by not spamming people. Steven Walling (WMF) • talk 18:35, 26 August 2012 (UTC)

Examples of collaboration and community feedback?

If you look at E3 you can see we designed an experiment with community members during the Wikimania unconference day. It's not perfect by a long shot, but that and the posting of plans here and on MediaWiki.org should be models we should continue to try I think. Are there other ways we might work more in partnership with volunteers early on? Steven Walling (WMF) • talk 03:43, 21 August 2012 (UTC)

Comparable organizations

In my discussions of these issues, it's been difficult for me to find comparable organizations to Wikimedia. We can definitely say we're different from Google, Facebook, Twitter, et al. But who are we similar to? Mozilla? EFF? Does anyone have a similar governance model? WikiHow, maybe?

I think finding comparable organizations may be beneficial. Maybe. --MZMcBride (talk) 04:36, 26 August 2012 (UTC)

Movement roles studied this, but I don't know how helpful it is in this context. --Nemo 06:31, 26 August 2012 (UTC)

It would probably be most useful to compare to the different kinds of A/B testing that are pretty widespread in big Web companies and startups now. That's the closest kind of operation to what E3 is doing, even if overall the WMF is more like Mozilla or the EFF. Another good example to compare to would be the testing done by the fundraiser. Steven Walling (WMF) • talk 18:20, 26 August 2012 (UTC)

I would have liked to include a comparison in my last edit, for instance "dozens tests, compared to thousands Facebook does every year". But I'm not sure the comparison makes sense and I'm not sure we have reliable numbers. --Nemo 07:49, 31 October 2014 (UTC)

Consent

The "colleague vs. customer" dynamic quickly approaches the issue of consent. How much consent is required for experiments? Does the level of consent change between brand new users and established users? Between brand new users and anonymous users? And does providing an opt-out option absolve some of the sin of experimenting without consent? --MZMcBride (talk) 04:38, 26 August 2012 (UTC)

To answer your question in more bureaucratic way (I thought about just saying this but it really doesn't give an answer), we consulted with WMF legal before running any experiments, and confirmed that the terms of use, privacy policy, and other current policies cover the purpose of experimentation to improve the service the Foundation provides. When you think about whether experiments should be opt-in or opt-out, regardless of whether we have to from a legal perspective, there are huge disadvantages to performing these in an opt-in way:

1. It's not any less disruptive to be regularly asking a sample of Wikipedians (new or old) whether they want to participate in an experiment. The number of people who actually have complained and asked for an opt-out option in relation to the number of people who have been impacted by our experiments is tiny. But providing some kind of new ask about trying an experiment would be annoying for the majority of readers and new editors who see what are experiments to us and nothing new or strange to them.
2. We want to make a wide impact, and asking for an opt-in group would probably limit the size of the groups in any given experiment.
3. We have and probably will continue to run experiments that impact a subset of readers or anonymous editors, for whom the mechanics of opting in and opting out are much more complex and don't work very well at the moment. Creating some kind of prior informed consent system is not only unnecessary in relation to the Terms of Use, but would require not insubstantial hours in infrastructure work (Ori or someone could expand here, or tell me I'm wrong and it would actually be easy).
4. Self-selecting for people open to experimentation within a target group biases the results of experiments, thus making it worthless to experiment in the first place, because you produce biased answers to a hypothesis.

Also: experimentation to improve things is not a sin. It's our job, as the organization that is entrusted with donations, donations which overwhelmingly come in to improve the technical side of Wikipedia. To not try new things in an experimental way, and instead go full bore on untested new features, would be the sin. Steven Walling (WMF) • talk 18:17, 26 August 2012 (UTC)

Customer satisfaction

Now, «use a short survey to track customer satisfaction».[1] Will this set a trend? What's the purpose? --Nemo 09:43, 2 May 2013 (UTC)

You should ask the Echo team directly about that, it's not super relevant to experiments, or the E3 team in particular. FWIW, mobile has also run surveys, as have I, but not really "customer satisfaction" surveys. We tend use surveys more for gathering qualitative data about how people use a feature, not if they generally feel positive or negative about it. Steven Walling (WMF) • talk 22:01, 2 May 2013 (UTC)