Research talk:Necromancy

Excellent naming

Is there any other industry which uses the term "necromancy" to refer to strategies to reclaim a lost userbase? If this was coined in-house then someone is very clever for choosing the best word to describe a common action. Blue Rasberry (talk) 04:46, 12 April 2012 (UTC)

We actually stole this from StackOverflow. They have a badge called "Necromancy" that they give out to users who revive dead Q&A threads. Steven Walling (WMF) • talk 05:21, 12 April 2012 (UTC)

w:Thread necromancy. It doesn't make it any less awkward. Killiondude (talk) 07:19, 12 April 2012 (UTC)

oh lawdy, what's next Poltergeist, Goblins or Warlocks? Sounds very professional. Anyway, more witchcraft and dark magic! Theo10011 (talk) 07:37, 12 April 2012 (UTC)

Knocking Reaper Eternal's username, are we? That's low :-P Maryana (WMF) (talk) 16:43, 12 April 2012 (UTC)

Results time period

Are you going to track edits over a longer period of time? I'm curious if any of the users who logged in in response to the e-mail will edit in the coming weeks and months. WhatamIdoing (talk) 01:46, 21 April 2012 (UTC)

We could quite easily, but the effect you can see and reasonably attribute to the email contact is over in about 48 hours. We capped it at a week so far because there was not really any further activity from the test group after that. There was one person who made hundreds of edits with Twinkle after returning from the 3 month lapse, but that's a single exception. Steven Walling (WMF) • talk 19:13, 22 April 2012 (UTC)

What I'd like to know is whether reminding the user about WM sites in March increases their activity in April or May. So I receive an e-mail message in March, and I respond to it. Let's say that I login, but I don't edit anything at that time. Do I login again next month? Do I edit anything next month? Or later this year? Or if you wanted me to take some kind of action every month, would you have to send me an e-mail message every single month? WhatamIdoing (talk) 17:42, 23 April 2012 (UTC)

It's a good question, but it's really hard to establish a causal relationship in that time frame: how do you say (without asking people) whether they came back because of the email reminder or not? Steven Walling (WMF) • talk 18:09, 23 April 2012 (UTC)

You're running a randomized controlled trial with a single intervention. You assume that the e-mail caused any statistically significant differences between the two groups in exactly the same way that you assume an experimental drug is responsible for the differences between the drug-receiving and the placebo-receiving groups in a clinical trial. You don't have to ask: you already know that the only difference between the groups is that one received the e-mail and the other didn't. (Of course, given the response rates, your groups might not be sufficiently large to identify these differences.) WhatamIdoing (talk) 14:07, 24 April 2012 (UTC)

It's not that controlled though. We don't know what the usual rate of return is over time, and there are a lot of potentially confounding effects, such as emails from other Wikipedians, talk page notification emails, etc. We also know that when it comes to email, the number of new opens and clickthroughs dies down dramatically after 48 hours, which is a big sign that we shouldn't stretch to try and create a relationship between any actions by editors far in the future. Steven Walling (WMF) • talk 16:29, 24 April 2012 (UTC)

You started with one big pool, and you randomly assigned individual users to either "treatment" (e-mail) or "no treatment", right? That's as controlled as it gets, Steven. "Typical" rates don't matter. The groups might be underpowered to account for the potential confounding factors, but you don't really need to worry about what's "typical". Proper randomization is really all you need.

I'm not proposing a mechanism by which people come back three months later to read old e-mail messages. I'm curious whether even seeing the message listed in the user's inbox reminds the user that he can edit Wikipedia if he wants to. It might be weeks or months before he wants to, but the reminder might make him more likely to do so when it occurs to him. WhatamIdoing (talk) 13:13, 25 April 2012 (UTC)

Some Questions

I came across this page as I brainstorming strategies for user recruitment/engagement. But there are many open questions which you might be able to answer - especially if you have access to older surveys. It seems that not all retired users are similar. I would be interested in four dimension.

• in the sense of what value they have actually contributed in hundreds of edits/patrolling/
• what value they are expected to contribute in the future based on the level of articles they worked on.
• what caused them to leave? (burn out, edit wars, personal vendettas, stalking, etc)
• an estimate of the difficulty of bringing them back.

Finlay there is an issue of merit. There are policy pages/essays about this subject and they recommend not to pursue certain classes of users that have departed. I'm worried that since there is no discrimination in the groups you might be encouraging the undesirables to return -- trolls or drama queens etc. OrenBochman (talk) 14:00, 23 April 2012 (UTC)

When you're emailing on the large scale it's pretty much impossible to screen out trolls or drama queens, except that we studiously filtered all blocked/banned editors (I believe including editors who were blocked any time in the 90 days prior to departing). Based on the responses we got back: the drama queens just emailed us back with further complaints, most editors did nothing, and a few read the email and clicked through.

In any case, you're correct that we need to play with different metrics: the challenge of course is you have to turn insights into what classes of editors we want back into some kind of query we can make of the database. This was our first stab at it, and I think we learned some important things. But there's definitely more to try! Steven Walling (WMF) • talk 17:23, 23 April 2012 (UTC)

Oren, you might consider another layer of complexity: Do they believe that they have left the site? I believe that past experience indicates that a significant number of users do not consider a lack of activity over the course of several months to actually mean that they have left. WhatamIdoing (talk) 17:44, 23 April 2012 (UTC)

Absolutely agreed with the above. We tried to assume that the 3 month group was on a WikiBreak rather than having left "forever", and write the email accordingly. Steven Walling (WMF) • talk 18:10, 23 April 2012 (UTC)

Good ideas all.

• If I was to extend this project as a MW feature I would indeed trigger it using a query that exracts the mean and standard deviation of edit interval. Whenever three times the standard deviation of edit interval elapses I'd send the user a we miss you messege. I'd also angineer the message using a complex template with a more emotional message.

I was fairly inactive for a years before getting more involved with wikipedia. If some one would ask me for a specific task in that subject domain and told me I was one of the few contributrs who could do it, I might be more moritvated to do the one edit.

My interest in this research is in a large part due to work I am doing on a proposal for Simone an agent for coaching new users. Simone is also planned to provide executive coaching for senior editors to extend editor life and provide transition to more senior roles. Simone would detect activity patterns and contact users proactivly. If this sounds interesting. you can have a look at the proposal at More on Simone OrenBochman (talk) 15:59, 25 April 2012 (UTC)

I generally like your "statistical" approach, but with a dose of common sense. Without some hard limits, then I suspect that a four-day vacation would result in a "We miss you" message to me. I'm not convinced that "senior editors" are really the target here anyway. If I go inactive and you want me back, then the message needs to come from a user that I know and respect. WhatamIdoing (talk) 23:51, 25 April 2012 (UTC)

Comments on phraseology

These are, of course, "form letters" as we used to say. And the recipient will know this. For that reason I suggest not being too over-the-top with the language - "fantastic contributions" rings false to me (maybe this is a British thing). Unless we are actually going to look at their contributions and say "Your fantastic contributions in identifying and using pictures of Mid-west railway stations" - then this language (if the contributions were indeed fantastic and not, say, mostly wrong) is justified, and will be very motivating. Rich Farmbrough 23:55 23 April 2012 (GMT).

Yes, the language we used is not typical for communication on Wikipedia. We believe that's a very good thing. ;) Anyway, I agree these emails would be vastly improved with some dynamic content based on some things about the editor themselves, but currently that's not really within our reach. I think if we wanted to try more personalized messages with manual analysis, then I'd prefer to ask community members (even if it was just OTRS folks or something) to get involved. Probably even more powerful than email content personalized to your edits would be an email from someone who actually collaborated with you. There's only so much an email from a faceless person with a wikimedia address can do. Steven Walling (WMF) • talk 00:15, 24 April 2012 (UTC)

Exactly! We could trial a manual mail to 100 users. I already did a similar data extraction for the Arabic Wikipedia and left messages for about 100 users to attempt to form a discussion on barriers to contribution, as part of my work with OII (Arabic WP uses captchas on email). Maybe this is something for early May. Rich Farmbrough 00:22 24 April 2012 (GMT).

Comment on analysis

• "One year is too long a period to contact lapsed people. It's safe we can ignore this group in the future, and try to reach them before it gets to be that long."

This is based on comparing the 1 year and 3 month figures. However there will be some who would have been in the 3 month group 9 months earlier but who have edited in the succeeding 9 months (the natural resurrection rate). The comparison needs to take that into account before assuming that there is a difference between two groups. Rich Farmbrough 00:05 24 April 2012 (GMT).

It's not only based on comparing the two. They had a very similar open rate and clickthrough, but the fact that not one person edited from the one year group isn't very promising. We'd have come to that conclusion even without having emailed the 3 month group. Steven Walling (WMF) • talk 00:09, 24 April 2012 (UTC)

The 1 year group is 53% more responsive than the 3 month group. Very strange. It would be useful to have some user stories here. Rich Farmbrough 00:15 24 April 2012 (GMT).

Where are you getting the 53% from? If you mean the different clickthrough rate (7% vs. 11%), that's not really a statistically significant difference. Steven Walling (WMF) • talk 00:22, 24 April 2012 (UTC)

Are you sure? If it was a sample of 10 I'd agree, if it was 1000 I'd disagree. Round about this sample size I'd want someoen to do the math. Rich Farmbrough 00:24 24 April 2012 (GMT).

(ec):Note that the higher click-through for the 1 year test group compared with the 3 month test group is very interesting, and if statistically significant, needs investigating. Rich Farmbrough 00:13 24 April 2012 (GMT).

Yeah, we could test that by getting a different landing page link to send them to and see if that makes a difference. Steven Walling (WMF) • talk 00:17, 24 April 2012 (UTC)

rename please

Necromancy is a fictional black magic occult practice of reactivating the dead. Reactivation is the normal uncontentious name for the process of reactivating former customers or members. Please rename to reactivation as more accurate and respectful to the people involved. WereSpielChequers (talk) 17:11, 7 December 2013 (UTC)

exclusions and tests

Are you excluding blocked and banned editors from this, and are you checking whether the editors involved are inactive across the whole project and not people who have refocussed from say Wikipedia to Wikidata or Wikimedia Commons? WereSpielChequers (talk) 17:15, 7 December 2013 (UTC)

control group

Is your control group also of editors with Email set on? If not there could be a skew as I'm assuming that the people who scramble passwords and delete email accounts have a different reactivation profile. WereSpielChequers (talk) 17:20, 7 December 2013 (UTC)

Alt test

I've been doing some reactivation emails over the last few months, but with a different offer. I've been emailing former editors with a geographic connection to a venue where I am planning to organise a GLAM event. So far the results have been mixed, but I'm planning to do some more robust testing next year. I have not mentioned that we think the editors have lapsed and want them back, this saved me the trouble of checking whether they have simply shifted to another project. Nor have I yet been checking subsequent edits. But when I emailed people who have taken photographs in a particular town and told them about an event in that town most of them replied, one chap said he lived at the other end of the country and had been taking photos there on holiday, another was keen but couldn't make the date. So far only one of my respondents has said she has not been editing recently. In another test I emailed ten inactive editors who had signed up to a relevant Wikiproject and got no response, but some of them had not edited for four or five years. I haven't checked if any of them have resumed editing. It would be great if we could integrate your program with GLAM and do such emails to promote local events, I appreciate you only keep IP info for a few months, but there should be occasions where someone can be emailed about an event near them 1-3 months after they edited at an IP in that area. WereSpielChequers (talk) 17:39, 7 December 2013 (UTC)