Research:Donor engagement/Thank You campaign

Overview

The campaign ran 3 different pairs of banners tested concurrently. During each test, visitors randomly received one of the two banners up to a maximum of 5 impressions, after which the banner was automatically disabled. Visitors also had the ability to manually disable banners via the close button. Each banner included a direct link to the account registration landing page as well as thumbnails pointing to a series of landing pages with editor stories, which in turn pointed to the account creation page (indirect link). Referrals from the direct link where logged with a fr12b-{banner_id} campaign id, while referrals from the indirect links embedded in editor stories where logged with a fr12c-{page} campaign. The campaign was launched at 2012-12-27T21:15:00Z and ended approximately at 2013-01-01T00:10:00Z.

Banners

2012-12-27: B12_1227_ThankYou_5pillars; B12_1227_ThankYou_Short
2012-12-28: B12_1228_ThankYou_Nonprofit; B12_1228_ThankYou_Thrilled
2012-12-31: B12_1231_ThankYou_Createacct; [1]

Results

How many new users registered as a result of this campaign?

The campaign resulted in a total of 1,107 successful account registrations referred from the banners. The vast majority of these registrations (N=1,057, 95.4%) were referred via the direct link, while only a tiny fraction originated from the landing pages (N=50, 4.5%) . During the period of the campaign (as defined by the time of the first and the last account registration referred by the banners), a total of 20,958 users registered an account via the regular Create Account link, in line with the expected signup rate for this period of the year.

How productive are these users compared to organic registrations?

Proportion of users with 1+ ns0 edits within 24 hours of registration in the two groups, plotted with 95% confidence intervals

To compare the productivity of the two groups of referred and organic registration we looked at the proportion of users in each group who completed at least 1 edit in the main article namespace within 24 hours of registrations. A total of 183 users (17.3%) from the referred group completed 1+ main namespace edits in 24 hours compared to 4,633 users (22.1%) from the control group of organic registrations (+4.8%). A chi-square test rejects the null hypothesis that the true proportion of editing accounts is the same in both groups (X² = 13.24, N = 22,015, p < .001). Referred users were significantly less likely to complete an edit than the control.

Which banner had the highest conversion rate?

Banners had fairly low conversion rates (account creations by raw account impressions), with mean conversions ranging from 0.00037% to 0.00056%. We compared the raw clickthrough rate, completion and conversion rate for each pair of banners that ran concurrently (we didn't include the last pair of banners (B12_1231_*) as they ran for a substantially shorter amount of time compared to the previous ones). The plots below compare the campaigns against these three metrics.

Click-through
Signup completion
Conversion
Click-through
Signup completion
Conversion

Conversion rates for each banner campaign (with binomial proportion CI)

      Estimate        Lower        Upper  Campaign
1 0.0005489544 0.0004825809 0.0006244566 B12_1227_ThankYou_5pillars
2 0.0005655095 0.0004927165 0.0006490568    B12_1227_ThankYou_Short

      Estimate        Lower        Upper  Campaign
1 0.0004575854 0.0004052429 0.0005166885 B12_1228_ThankYou_Nonprofit
2 0.0003737361 0.0003332572 0.0004191317  B12_1228_ThankYou_Thrilled

Conclusions

Funnel conversion data

The following data was collected at the end of the test (events logs imported as of 2013-01-02T23:30:00Z, raw event counts, non-deduplicated).

Banner impressions

(1/100 sampled impressions)

420800	B12_1227_ThankYou_5pillars
357200	B12_1227_ThankYou_Short
568200	B12_1228_ThankYou_Nonprofit
781300	B12_1228_ThankYou_Thrilled
76300	B12_1231_ThankYou_Createacct
23200	B12_1231_ThankYou_Getstrtd

Account creation: impressions

(excluding error impressions, i.e. form impressions logged when an error is returned after submitting the form)

2298 fr12b-B12_1228_ThankYou_Thrilled
2145 fr12b-B12_1228_ThankYou_Nonprofit
1517 fr12b-B12_1227_ThankYou_Short
1332 fr12b-B12_1227_ThankYou_5pillars
 623 fr12b-B12_1231_ThankYou_Getstrtd
 288 fr12b-B12_1231_ThankYou_Createacct

Landing page

 110 fr12c-Poongothai-Balasubramanian
  92 fr12c-Thank-You
  84 fr12c-Ken-Thomas
  73 fr12c-Mei-Jiun
  57 fr12c-Jimmy-Wales
  55 fr12c-Ruben-Hilare-Quispe
  53 fr12c-Oarabile-Mudongo
  15 fr12c-Abhishek-Hingnikar
  12 fr12c-Melisa-Parisi
  12 fr12c-Benjamin-Chen
   8 fr12c-Peter-Gallert
   7 fr12c-Ravan-Jaafar
   6 fr12c-Zinaida-Tebaykina
   6 fr12c-Peter-Coti
   6 fr12c-Noopur-Raval
   6 fr12c-Dumisani-Ndubane
   6 fr12c-Aliona-Bogdanova
   5 fr12c-Q-Miceli
   5 fr12c-Netha-Hussain
   4 fr12c-Viswa-Prabha
   4 fr12c-Elena-Strelnikova
   4 fr12c-Anastasia-Lvova
   4 fr12c-Amina-Duale
   3 fr12c-Susanna-Mkrtchyan
   3 fr12c-Ganesh-Paudel
   3 fr12c-David-Shankbone
   3 fr12c-Andrea-Zanni
   2 fr12c-Tony-Santiago
   2 fr12c-Judy-Tuan
   2 fr12c-Juan-Ignacio
   2 fr12c-Ivan-Meetin
   2 fr12c-Howard-Morland
   2 fr12c-Chanitra-Bishop
   2 fr12c-Benoit-Rochon
   2 fr12c-Amit-Kapoor
   2 fr12c-Alejandro-Linares
   1 fr12c-Srikeit-Tadepalli
   1 fr12c-Mina-Nagy
   1 fr12c-Leslie-Carr
   1 fr12c-Gideon-Digby
   1 fr12c-Gereon-Kalkuhl
   1 fr12c-Emmanuel-Engelhart
   1 fr12c-Deryck-Chan
   1 fr12c-David-Thomsen
   1 fr12c-Brandon-Harris
   1 fr12c-Adrianne-Wadewitz

Account creation: error impressions

(form impressions logged when an error is returned after submitting the form)

Banner

1735 fr12b-B12_1228_ThankYou_Thrilled
1314 fr12b-B12_1228_ThankYou_Nonprofit
 981 fr12b-B12_1227_ThankYou_5pillars
 900 fr12b-B12_1227_ThankYou_Short
 239 fr12b-B12_1231_ThankYou_Getstrtd
 164 fr12b-B12_1231_ThankYou_Createacct

Landing page

 52 fr12c-Poongothai-Balasubramanian
  40 fr12c-Oarabile-Mudongo
  31 fr12c-Thank-You
  31 fr12c-Amina-Duale
  25 fr12c-Ken-Thomas
  19 fr12c-Ruben-Hilare-Quispe
  17 fr12c-Jimmy-Wales
  15 fr12c-Abhishek-Hingnikar
  11 fr12c-Mei-Jiun
   4 fr12c-Ganesh-Paudel
   4 fr12c-Deryck-Chan
   3 fr12c-Viswa-Prabha
   3 fr12c-Peter-Gallert
   3 fr12c-Dumisani-Ndubane
   3 fr12c-Amit-Kapoor
   2 fr12c-Zinaida-Tebaykina
   2 fr12c-Ravan-Jaafar
   2 fr12c-Noopur-Raval
   2 fr12c-Benjamin-Chen
   2 fr12c-Adrianne-Wadewitz
   1 fr12c-Elena-Strelnikova
   1 fr12c-Andrea-Zanni

Account creation: submits

(excluding multiple submit attempts after the form returns an error)

Banner

 814 fr12b-B12_1228_ThankYou_Thrilled
 765 fr12b-B12_1228_ThankYou_Nonprofit
 590 fr12b-B12_1227_ThankYou_Short
 557 fr12b-B12_1227_ThankYou_5pillars
 117 fr12b-B12_1231_ThankYou_Getstrtd
  91 fr12b-B12_1231_ThankYou_Createacct

Landing page

  35 fr12c-Poongothai-Balasubramanian
  19 fr12c-Thank-You
  17 fr12c-Ken-Thomas
  12 fr12c-Jimmy-Wales
  10 fr12c-Ruben-Hilare-Quispe
   9 fr12c-Oarabile-Mudongo
   7 fr12c-Mei-Jiun-Kwek
   5 fr12c-Abhishek-Hingnikar
   3 fr12c-Zinaida-Tebaykina
   3 fr12c-Amina-Duale
   2 fr12c-Viswa-Prabha
   2 fr12c-Peter-Gallert
   2 fr12c-Ganesh-Paudel
   2 fr12c-Dumisani-Ndubane
   2 fr12c-Benjamin-Chen
   1 fr12c-Tony-Santiago
   1 fr12c-Ravan-Jaafar-Altaie
   1 fr12c-Peter-Coti
   1 fr12c-Noopur-Raval
   1 fr12c-Juan-Ignacio-Iglesias
   1 fr12c-Howard-Morland
   1 fr12c-Elena-Strelnikova
   1 fr12c-Deryck-Chan
   1 fr12c-Andrea-Zanni
   1 fr12c-Anastasia-Lvova
   1 fr12c-Amit-Kapoor
   1 fr12c-Aliona-Bogdanova
   1 fr12c-Adrianne-Wadewitz

Account creation: error submits

(submit attempts after the form returned at least one error)

Banner

1035 fr12b-B12_1228_ThankYou_Thrilled
 875 fr12b-B12_1228_ThankYou_Nonprofit
 725 fr12b-B12_1227_ThankYou_5pillars
 595 fr12b-B12_1227_ThankYou_Short
 165 fr12b-B12_1231_ThankYou_Getstrtd
 119 fr12b-B12_1231_ThankYou_Createacct

Landing page

  31 fr12c-Poongothai-Balasubramanian
  31 fr12c-Amina-Duale
  29 fr12c-Oarabile-Mudongo
  18 fr12c-Thank-You
  15 fr12c-Jimmy-Wales
  11 fr12c-Ken-Thomas
  11 fr12c-Abhishek-Hingnikar
   9 fr12c-Ruben-Hilare-Quispe
   5 fr12c-Mei-Jiun-Kwek
   3 fr12c-Viswa-Prabha
   3 fr12c-Thank-You
   3 fr12c-Ganesh-Paudel
   3 fr12c-Deryck-Chan
   3 fr12c-Amit-Kapoor
   2 fr12c-Poongothai
   2 fr12c-Peter-Gallert
   2 fr12c-Noopur-Raval
   2 fr12c-Benjamin-Chen
   2 fr12c-Adrianne-Wadewitz
   1 fr12c-Zinaida-Tebaykina
   1 fr12c-Ravan-Jaafar-Altaie
   1 fr12c-Dumisani-Ndubane

Account creation: success

Banner

 292 fr12b-B12_1228_ThankYou_Thrilled
 260 fr12b-B12_1228_ThankYou_Nonprofit
 231 fr12b-B12_1227_ThankYou_5pillars
 202 fr12b-B12_1227_ThankYou_Short
  38 fr12b-B12_1231_ThankYou_Getstrtd
  33 fr12b-B12_1231_ThankYou_Createacct

Landing page

   9 fr12c-Poongothai-Balasubramanian
   7 fr12c-Jimmy-Wales
   6 fr12c-Thank-You
   4 fr12c-Oarabile-Mudongo
   4 fr12c-Ken-Thomas
   3 fr12c-Amina-Duale
   2 fr12c-Viswa-Prabha
   2 fr12c-Ruben-Hilare-Quispe
   2 fr12c-Mei-Jiun-Kwek
   2 fr12c-Abhishek-Hingnikar
   1 fr12c-Zinaida-Tebaykina
   1 fr12c-Ravan-Jaafar-Altaie
   1 fr12c-Peter-Gallert
   1 fr12c-Noopur-Raval
   1 fr12c-Ganesh-Paudel
   1 fr12c-Deryck-Chan
   1 fr12c-Benjamin-Chen
   1 fr12c-Amit-Kapoor
   1 fr12c-Adrianne-Wadewitz

Raw signup completion rates

Referral	Raw account creation impressions	Account creations	Raw completion rate
fr12b-B12_1227_ThankYou_Short	1517	202	13.3%
fr12b-B12_1227_ThankYou_5pillars	1332	231	17.3%
fr12b-B12_1228_ThankYou_Nonprofit	2145	260	12.1%
fr12b-B12_1228_ThankYou_Thrilled	2298	292	12.7%
fr12b-B12_1231_ThankYou_Getstrtd	623	38	6.1%
fr12b-B12_1231_ThankYou_Createacct	288	33	11.4%
fr12c (landing pages)	674	50	7.4%

Notes

This data is interesting to compare to the results of our previous account creation experiments, where conversion rates were between 28-32% for both the control and test versions of a new account creation interface. It should not, however, be necessarily used to decide which of the two banners was more successful. To do so, the conversion rate would be account creations over total banner and landing page impressions.