Fundraising 2012/We Need A Breakthrough
May 2012 edit
| 11 May, 2012: More "Facts" testing |
|---|
|
Hi everyone, this is Megan and Zack on May 11. Megan's in Paris, Zack is in Missouri. We're doing another one of our weekly tests and this week we're trying another approach to presenting "The Facts". At 13:50 UTC we launched a three way landing page test and now we're placing our bets before looking at the first results: Control = 100 Now we'll look at the initial results to make sure donations are coming in properly. The first results make us feel smart. But we can't trust them -- we're only minutes into the test and the sample size is still too small: +-----------------------------+-----------+---------+----------+ | landing_page | donations | amount | amount25 | +-----------------------------+-----------+---------+----------+ | Test_120511_Brandoncont | 75 | 1655.00 | 1055.00 | | Test_120511_Brandonfacts | 67 | 1653.98 | 1103.98 | | Test_120511_Brandonfactswap | 95 | 2220.00 | 1515.00 | +-----------------------------+-----------+---------+----------+ "Amount25" is a normalized representation of the total amount raised. We convert all donations of 25 and up to 25 in calculating the total. This removes the effect of a few large donations. OK, the test is done. We left banners up for a little more than one hour, hoping to get samples big enough to have confidence. Unfortunately, in the end we did not get a statistically significant result. We should have stuck to running only two landing pages -- then we might have gotten a better result. But, we still learned something: It's unlikely that any one of these three was a big breakthrough. And this test reinforces what we saw last week: That putting "The Facts" in place of an appeal can be as powerful as the appeal in the primary position. This is a good thing, because it gives us hope that we can feature Jimmy, editors, staff, donors and others and make as much as with our standard money-maker, the Jimmy appeal. But more testing will be necessary to see if this is true. If it is, then maybe we will always lead with the facts in the primary position and try to find new combinations of facts that work even better. Here are the almost-final results (some donations will trickle in). +-----------------------------+-----------+----------+----------+ | landing_page | donations | amount | amount25 | +-----------------------------+-----------+----------+----------+ | Test_120511_Brandoncont | 376 | 7562.00 | 5462.00 | | Test_120511_Brandonfacts | 385 | 8579.99 | 6141.99 | | Test_120511_Brandonfactswap | 422 | 10706.77 | 7005.41 | +-----------------------------+-----------+----------+----------+ One more thing: Even though we're not getting a significant result for donations, it's possible that we have significance when it comes to total amount. There seems to be something about The Facts that raises the average donation. We will do more research on this later in the day and report back. (And later that night...) OK, so Peter Gehres did some work on analysis. He did something that we meant to do during the fundraiser by never found the time: He did running averages for our metrics. Then we graphed them. This was something suggested to us by the good folks at Optimizely who Steve Geer brought by for a visit. This is a really great visualization. But now we have to bring this back and connect it to the students t-test that we've been using to calculate confidence. Our fancy t-test calculating system is currently in dry dock for repairs. But I plugged the numbers into an online t-test calculator (in a not-perfect way) and it noted 95% confidence for this test for both donations per and amount per view. In addition to that, you can see in the graphs below that a clear trend is visible by the end of the test. This just goes to show again why those common AB testing statistical significance calculators on the web (the kind I was using above) use the wrong method for this particular kind of testing. Here's the underlying numbers if you'd like to do some analysis yourself. |
| 16 May, 2012: Multivariate landing page testing! | |||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Good morning from Zack and good afternoon from Megan. Today we're attempting a multivariate test of three variables. We think we'll be able to get full sample results for each variable and -- assuming these variables are independent -- have valid results for each variable. We just launched the campaign and before looking at any early results, we have placed our bets:
We're betting on amount as well as donations now because after analyzing the results from last week, we think one or more of these variables are having a big impact on the average donation and therefore the amount raised. OK, we got some initial results...going to have a lot of fun picking these apart over the next few hours. Stay tuned. (Now it's May 21! That took a bit more than a few hours...) So this is a pretty exciting time for our fundraising program. Thanks to Peter Gehres, we've got a much more flexible system for gathering and analyzing our test data. That's why we felt confident to try out this multivariate test this week. The same as in the May 11 test above, we made the running average graphs. And they made me feel like we might have fairly solid results. But then I pulled the data in R and ran a wp:t-test on the data. It turned out, we didn't learn as much as I hoped. One test produced one statistically significant result (Avg25 in the Five/No5 test). The rest were not significant. One question I have now is this: Did cramming three variables into this test reduce or increase our chances for finding significance. I'm pretty sure it increased our chances. That's because by testing three variables, we had three shots at finding something significant. Because of the way we structured the test, each A and B of each of the three vars all had the same sample sizes they would have if we had only tested one variable. Moreover, when I ran the t-test against our previous two single-variable tests, I found a similar mix of confident and unconfident results. So first I'll show you the running average graph. I know that many testing programs rely on this kind of graph. But the analysis I did over weekend (if correct) shows that either this graph is misleading or that the t-test is not appropriate for this kind of data. The graphs below represent the adjusted cumulative average donation over the course of the test. Imagine that every minute we measured the average donation given on each landing page over the course of the test up until then. In fact, that is exactly what we did, and that's what you're looking at below! By nature, these graphs show high variability in the beginning, when the sample size is very small and each donation has a big effect on the average. Then they smooth out. Over time, a more "true" average would emerge. Statistics theory says that the mean of a large enough sample will be (very likely) very close to the eventual average of the whole population. In our case, the population would be all the minutes of a full five-week fundraising campaign. The first graph shows all eight of the landing pages separately. The next two group the four "Five" and "No 5" landing pages together to look at that one variable's effect on adjusted average donation and number of donations.
Those last two graphs look fairly convincing don't they? Doesn't it look like both are almost stabilizing at significantly different levels? But according to the t-test, we can only be confident in the adjusted average result. As far as the top graph is concerned, the t-test finds many statistically significant differences in the performance of different landing pages in that test, but only when mixing variables. In other words, AppealNo5Short beats SwapFiveLong consistently, and the t-test says the relationship is not random. But which variables are responsible for the relationship? For example, it's possible that Swap actually beats Appeal, but that its effect is overridden by the downward pull of Five and Long. Also, remember above when I said that we hoped these three variables were independent. Well, they're probably not. It looks like Long pushes the Five/No5 off the screen for many readers. That means that Long-Five/etc become variables in themselves. Learning how to interpret the effects of those kinds of dependencies -- or whether we just need to avoid them altogether -- is something we're going to work on in the coming months. So what did the t-test say about the previous two tests? Using the t-test, I went back and looked at our two previous tests. Happily, we had a very confident result that Brandon with the Facts in the primary position (Swap) beat the reverse (Appeal) by average. But again, we did not have confidence when it came to the number of donations. Applying the t-test to the Jimmy Facts vs Appeal test we did a few weeks ago, we find no confidence in any result. That is not surprising because the two pages we tested were extremely similar. Here's what we still don't understand though: If we test two things that seem very similar to us, and the results from each page are very close to each other, and we do not get statistical significance, then can we trust that there is very little difference between the two pages? Or do we have to keep running the test longer until we get confidence in a t-test or other statistical measure? To try to answer that question, as an experiment I applied the t-test to the distribution of landing page views between two pages that we (think we) know will get almost exactly equal numbers over a pretty short time (as short as an hour). In the hour-long sample, the numbers vary slightly from minute to minute but average out to almost the same number over the whole sample. The t-test showed almost the lowest possible confidence level. That leads me to believe that if we're testing two very similar pages against each other, that we can interpret low confidence as meaning "no difference" when the results are almost the same. PS: Here are the raw numbers for each of the variables tested, and all the landing pages separately. But, to repeat, we think we only have confidence in the average for Five/No5: +-------+-----------+----------+----------+----------+----------+-----------+-----------+-----------+-----------+-----------+-----------+ | Leng | donations | amount | amount18 | amount25 | amount50 | amount100 | avg | avg18 | avg25 | avg50 | avg100 | +-------+-----------+----------+----------+----------+----------+-----------+-----------+-----------+-----------+-----------+-----------+ | Long | 638 | 13628.00 | 8081.00 | 9688.00 | 11703.00 | 13178.00 | 21.360502 | 12.666144 | 15.184953 | 18.343260 | 20.655172 | | Short | 637 | 13788.00 | 7900.00 | 9443.00 | 11738.00 | 13138.00 | 21.645212 | 12.401884 | 14.824176 | 18.427002 | 20.624804 | +-------+-----------+----------+----------+----------+----------+-----------+-----------+-----------+-----------+-----------+-----------+ +--------+-----------+----------+----------+----------+----------+-----------+-----------+-----------+-----------+-----------+-----------+ | Swap | donations | amount | amount18 | amount25 | amount50 | amount100 | avg | avg18 | avg25 | avg50 | avg100 | +--------+-----------+----------+----------+----------+----------+-----------+-----------+-----------+-----------+-----------+-----------+ | Appeal | 666 | 14125.00 | 8227.00 | 9765.00 | 12100.00 | 13675.00 | 21.208709 | 12.352853 | 14.662162 | 18.168168 | 20.533033 | | Swap | 609 | 13291.00 | 7754.00 | 9366.00 | 11341.00 | 12641.00 | 21.824302 | 12.732348 | 15.379310 | 18.622332 | 20.756979 | +--------+-----------+----------+----------+----------+----------+-----------+-----------+-----------+-----------+-----------+-----------+ +------+-----------+----------+----------+----------+----------+-----------+-----------+-----------+-----------+-----------+-----------+ | Five | donations | amount | amount18 | amount25 | amount50 | amount100 | avg | avg18 | avg25 | avg50 | avg100 | +------+-----------+----------+----------+----------+----------+-----------+-----------+-----------+-----------+-----------+-----------+ | Five | 666 | 14203.00 | 8106.00 | 9643.00 | 11803.00 | 13353.00 | 21.325826 | 12.171171 | 14.478979 | 17.722222 | 20.049550 | | No5 | 609 | 13213.00 | 7875.00 | 9488.00 | 11638.00 | 12963.00 | 21.696223 | 12.931034 | 15.579639 | 19.110016 | 21.285714 | +------+-----------+----------+----------+----------+----------+-----------+-----------+-----------+-----------+-----------+-----------+ +--------------------------+-----------+---------+----------+----------+----------+-----------+-----------+-----------+-----------+-----------+-----------+ | landingpage | donations | amount | amount18 | amount25 | amount50 | amount100 | avg | avg18 | avg25 | avg50 | avg100 | +--------------------------+-----------+---------+----------+----------+----------+-----------+-----------+-----------+-----------+-----------+-----------+ | L12_0516_Appealfivelong | 186 | 3958.00 | 2326.00 | 2768.00 | 3458.00 | 3808.00 | 21.279570 | 12.505376 | 14.881720 | 18.591398 | 20.473118 | | L12_0516_Appealfiveshort | 174 | 3532.00 | 1954.00 | 2287.00 | 2882.00 | 3382.00 | 20.298851 | 11.229885 | 13.143678 | 16.563218 | 19.436782 | | L12_0516_AppealNo5long | 158 | 3604.00 | 2017.00 | 2404.00 | 2929.00 | 3454.00 | 22.810127 | 12.765823 | 15.215190 | 18.537975 | 21.860759 | | L12_0516_AppealNo5Short | 148 | 3031.00 | 1930.00 | 2306.00 | 2831.00 | 3031.00 | 20.479730 | 13.040541 | 15.581081 | 19.128378 | 20.479730 | | L12_0516_SwapFiveLong | 136 | 2725.00 | 1691.00 | 2025.00 | 2325.00 | 2575.00 | 20.036765 | 12.433824 | 14.889706 | 17.095588 | 18.933824 | | L12_0516_SwapFiveShort | 170 | 3988.00 | 2135.00 | 2563.00 | 3138.00 | 3588.00 | 23.458824 | 12.558824 | 15.076471 | 18.458824 | 21.105882 | | L12_0516_SwapNo5Long | 158 | 3341.00 | 2047.00 | 2491.00 | 2991.00 | 3341.00 | 21.145570 | 12.955696 | 15.765823 | 18.930380 | 21.145570 | | L12_0516_SwapNo5Short | 145 | 3237.00 | 1881.00 | 2287.00 | 2887.00 | 3137.00 | 22.324138 | 12.972414 | 15.772414 | 19.910345 | 21.634483 | +--------------------------+-----------+---------+----------+----------+----------+-----------+-----------+-----------+-----------+-----------+-----------+ |
| 25 May, 2012: Editor+Facts vs our best Jimmy |
|---|
|
Morning everyone. We're testing a few interesting variables today. What we're really hoping we find is that using "The Facts" Susan will hold her own against Jimmy -- or even do better. More multivariate testing going on today. What variables are we testing?
Bets! Facts+Jimmy: Avg25 = 100; Donations = 100. Five: Avg25 = 100; Donations = 100. Old facts: Avg25 = 100; Donations = 100. Comments: Zack: BUT...I don't think we're going to get statistical confidence on anything but Susan vs Jimmy. Megan: Brighter banners will do a tad better.
More statistical analysis to come over the weekend, those are just a few preliminary results!
Round 1 Banners:
Round 2 Banners: Landing Pages:
|
June 2012 edit
| 1 June 2012 5 Facts about Wikipedia vs. 5 Reasons to support Wikipedia | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Mornin' and Evenin' to you from Megan at the Berlin Hackaton and Zack in his backyard in Missouri. Today we're testing:
Links here: banner: http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_0601_JW 5 reasons: http://wikimediafoundation.org/wiki/L12_0601_ReasonsRed/en 5 reasons twin: http://wikimediafoundation.org/wiki/L12_0601_ReasonsBlue/en 5 facts: http://wikimediafoundation.org/wiki/L12_0601_FactsRed/en 5 facts twin: http://wikimediafoundation.org/wiki/L12_0601_FactsBlue/en We've placed our Bets!
We're spending a lot of time these days studying statistics and trying to improve our ability to detect small differences (e.g. 5%) accurately. We think we'll need this ability in 2012 and the future because the big advances are getting harder to find. Here's why we're doing "twins". Look: minutes into the test, and Reasons are winning! +---------+-----------+---------+----------+-----------+ | cell | donations | amount | amount25 | avg25 | +---------+-----------+---------+----------+-----------+ | Facts | 208 | 3602.00 | 2777.00 | 13.350962 | | Reasons | 221 | 4610.11 | 3285.11 | 14.864751 | +---------+-----------+---------+----------+-----------+ Or are they? Here are the results for "Red" vs "Blue" (each cell is 50% Reasons and 50% Facts -- so there should be no difference): +------+-----------+---------+----------+-----------+ | cell | donations | amount | amount25 | avg25 | +------+-----------+---------+----------+-----------+ | Blue | 223 | 4419.00 | 3144.00 | 14.098655 | | Red | 206 | 3793.11 | 2918.11 | 14.165583 | +------+-----------+---------+----------+-----------+ Blue team winning! ;-) Here is the breakdown of the actual landing pages: +----------------------+-----------+---------+----------+-----------+ | cell | donations | amount | amount25 | avg25 | +----------------------+-----------+---------+----------+-----------+ | L12_0601_FactsBlue | 104 | 1975.00 | 1350.00 | 12.980769 | | L12_0601_FactsRed | 104 | 1627.00 | 1427.00 | 13.721154 | | L12_0601_ReasonsBlue | 119 | 2444.00 | 1794.00 | 15.075630 | | L12_0601_ReasonsRed | 102 | 2166.11 | 1491.11 | 14.618725 | +----------------------+-----------+---------+----------+-----------+ There are statistical methods that are supposed to tell us how to do all this. We have found that they work to detect large differences with our 2 hours tests, or small differences with much longer tests (which we can only do during the fundraiser). But we've found that everyone starts disagreeing about methods when you want to push the envelope on sample size. This is a big deal with drug trials and other high-stakes statistical problems. Debates are raging among statisticians. Luckily, we're not putting drugs into people, we're just showing banners. But we really want to figure this out so that we can make more discoveries outside of the time of the fundraiser. A few minutes later, and here are some more (still highly unreliable) results. We're just showing these so that you can see how wildly the results fluctuate when the sample size is small: +---------+-----------+---------+----------+-----------+ | cell | donations | amount | amount25 | avg25 | +---------+-----------+---------+----------+-----------+ | Facts | 340 | 6138.00 | 4588.00 | 13.494118 | | Reasons | 339 | 7026.11 | 4976.11 | 14.678791 | +---------+-----------+---------+----------+-----------+ +------+-----------+---------+----------+-----------+ | cell | donations | amount | amount25 | avg25 | +------+-----------+---------+----------+-----------+ | Blue | 346 | 6766.00 | 4791.00 | 13.846821 | | Red | 333 | 6398.11 | 4773.11 | 14.333664 | +------+-----------+---------+----------+-----------+ +----------------------+-----------+---------+----------+-----------+ | cell | donations | amount | amount25 | avg25 | +----------------------+-----------+---------+----------+-----------+ | L12_0601_FactsBlue | 169 | 3181.00 | 2206.00 | 13.053254 | | L12_0601_FactsRed | 171 | 2957.00 | 2382.00 | 13.929825 | | L12_0601_ReasonsBlue | 177 | 3585.00 | 2585.00 | 14.604520 | | L12_0601_ReasonsRed | 162 | 3441.11 | 2391.11 | 14.759938 | +----------------------+-----------+---------+----------+-----------+ OK -- we're a little over an hour into the two-hour test and look where we are: +---------+-----------+---------+----------+-----------+ | cell | donations | amount | amount25 | avg25 | +---------+-----------+---------+----------+-----------+ | Facts | 438 | 8213.00 | 5963.00 | 13.614155 | | Reasons | 418 | 8589.11 | 6089.11 | 14.567249 | +---------+-----------+---------+----------+-----------+ +------+-----------+---------+----------+-----------+ | cell | donations | amount | amount25 | avg25 | +------+-----------+---------+----------+-----------+ | Blue | 429 | 8524.00 | 5974.00 | 13.925408 | | Red | 427 | 8278.11 | 6078.11 | 14.234450 | +------+-----------+---------+----------+-----------+ As we would expect: the identical Blue and Red groups are harmonizing. But what if the sample size is still too small and they are only harmonizing by chance at this moment. Well...here's what we're trying to figure out: Let's say we did this every day and the twins always harmonized after an hour. Does that mean we could trust the difference between Facts vs Reason (or whatever we're testing on a given day) at this point in the test? And could we trust the size of the difference or just that the winner is the real winner? And at what confidence level -- i.e. how often would we be falling victim to a false positive or a false negative? If you can answer these questions and explain the answers (rather than just say "Someone else figured this out and I trust them!") we want to talk to you! In the mean time, we'll keep running this test until 2 hours have passed, which should be about 80,000 landing page impressions. The test is over now. Reason did 91% as well as Facts and Blue did 98.8% as well as Red -- by # of donations. But how sure can we be that this reflects what would happen if we ran this test for several days with a much, much larger sample? In other words, how confident can we be that we should stick with Facts? And then there's amount: by adjusted amount, Facts wins by only 4%. But Red also beats Blue by 4%. So even after 70,000 observations (landing page impressions for us), we don't see a difference we can trust between Facts and Reasons in the statistic that really matters. I know we still owe you more analysis of last week's test too. We've been working on it. We hope to have some good statistical analysis of both this and last week's tests up here soon. Here are the final raw results of the test: key:
/lpi = donations per banner impression +---------+-----------+----------+----------+-----------+-----------+-----------+-------+--------+----------+----------+--------------+ | cell | donations | amount | amount25 | amount100 | avg25 | avg100 | lpi | d/lpi | t/lpi | t25/lpi | m25/lpi | +---------+-----------+----------+----------+-----------+-----------+-----------+-------+--------+----------+----------+--------------+ | Facts | 891 | 26992.00 | 12223.00 | 17123.00 | 13.718294 | 19.217733 | 37563 | 0.0237 | 0.718579 | 0.325400 | 0.0003652076 | | Reasons | 811 | 16772.11 | 11586.11 | 16022.11 | 14.286202 | 19.755993 | 37245 | 0.0218 | 0.450318 | 0.311078 | 0.0003835737 | +---------+-----------+----------+----------+-----------+-----------+-----------+-------+--------+----------+----------+--------------+ +------+-----------+----------+----------+-----------+-----------+-----------+-------+--------+----------+----------+--------------+ | cell | donations | amount | amount25 | amount100 | avg25 | avg100 | lpi | d/lpi | t/lpi | t25/lpi | m25/lpi | +------+-----------+----------+----------+-----------+-----------+-----------+-------+--------+----------+----------+--------------+ | Blue | 846 | 17925.00 | 11649.00 | 16125.00 | 13.769504 | 19.060284 | 37319 | 0.0227 | 0.480318 | 0.312147 | 0.0003689677 | | Red | 856 | 25839.11 | 12160.11 | 17020.11 | 14.205736 | 19.883306 | 37489 | 0.0228 | 0.689245 | 0.324365 | 0.0003789308 | +------+-----------+----------+----------+-----------+-----------+-----------+-------+--------+----------+----------+--------------+ +----------------------+-----------+----------+----------+-----------+-----------+-----------+-------+--------+----------+----------+--------------+ | cell | donations | amount | amount25 | amount100 | avg25 | avg100 | lpi | d/lpi | t/lpi | t25/lpi | m25/lpi | +----------------------+-----------+----------+----------+-----------+-----------+-----------+-------+--------+----------+----------+--------------+ | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 1 | NULL | NULL | NULL | NULL | | L12_0601_FactsBlue | 440 | 9999.00 | 5884.00 | 8349.00 | 13.372727 | 18.975000 | 18732 | 0.0235 | 0.533792 | 0.314115 | 0.0007138974 | | L12_0601_FactsRed | 451 | 16993.00 | 6339.00 | 8774.00 | 14.055432 | 19.454545 | 18831 | 0.0239 | 0.902395 | 0.336626 | 0.0007463986 | | L12_0601_ReasonsBlue | 406 | 7926.00 | 5765.00 | 7776.00 | 14.199507 | 19.152709 | 18587 | 0.0218 | 0.426427 | 0.310163 | 0.0007639483 | | L12_0601_ReasonsRed | 405 | 8846.11 | 5821.11 | 8246.11 | 14.373111 | 20.360765 | 18657 | 0.0217 | 0.474144 | 0.312007 | 0.0007703870 | +----------------------+-----------+----------+----------+-----------+-----------+-----------+-------+--------+----------+----------+--------------+ |
| June 18 2012 -- Thinking about stats (in Moscow) |
|---|
|
We've been in Russia for 5 days, listening to donors and others who might be able to help us improve our fundraiser Russia and Russian. We're hoping to launch a test with a new translation of the Jimmy appeal and some new banners based on what we've learned. We've been studying stats pretty intensively over the past few weeks and I'm trying to use what I (think I) have learned to determine how many banners and landing pages we can test in Russia per day. Traffic to Wikipedia in Russia is much lower than in the U.S., therefore our sample is going to be smaller. We would like to be able to detect at least a 2% difference in click rate of banners. Looking back at last year's first fundraising day in Russia, we had an overall 2.5% click rate. With that info, using R, I think that this is how I predict our required sample size: > power.prop.test(p1=0.025,p2=0.0255,sig.level=0.05,power=0.8) p1 and p2 are the click rates of two banners we want to compare. 0.025 is a 2.5% click rate. A 0.0255 click rate would represent a 2% increase over 0.025. The output below from R tells me that we need 1.5 million banner impressions for each banner to know -- with 95% confidence -- that an observed difference of 2% or more can not be attributed to chance. Two-sample comparison of proportions power calculation
n = 1545439
p1 = 0.025
p2 = 0.0255
sig.level = 0.05
power = 0.8
alternative = two.sided
NOTE: n is number in *each* group
Using the same R function, I can also give it a sample size and see how big of a difference we'd be able to have confidence in. If we did a simple AB test of two banners, each would have about 7 million impressions over a day in Russia. Here's how I tell R that: > power.prop.test(p1=0.025,n=7000000,sig.level=0.05,power=0.8) And R tells me that we'd get close to being able to detect a 1% difference: Two-sample comparison of proportions power calculation
n = 7e+06
p1 = 0.025
p2 = 0.02528033
sig.level = 0.05
power = 0.8
alternative = two.sided
NOTE: n is number in *each* group
In other words, we should be able to have confidence in anything greater than a 1.12% difference: > 0.02528033/.025-1 [1] 0.0112132 If you're a statistician, please tell me if you think I'm doing any of this wrong! |
| June 22: Our first top banner tests |
|---|
|
We did a big multivariate test of a bunch of new top banners with "the facts" right in the banners. The only way to present the results is in spreadsheet format. So please take a look here: The bottom line: Top banners with these facts do work! Even the short banners beat the old inline Jimmy banner. Interestingly, putting the Jimmy banner up top does not help, but hurts. Granted it's an ugly looking banner. (Note: if you look into the future, down below, we did some nice looking top ones and they also did poorly. For some weird reason the Facts work on top and the Appeal works inline.) |
| June 29 -- Big Crazy New |
|---|
|
Megan's in Stockholm for focus groups with donors. I'm back in the shed in Missouri. We're testing some really new and different designs today thanks to the new Springfield, MO based designers we're working with now. We first tested banners with the form directly in the banner:
At first we thought the form banners did horribly. But it turned out we just had a tracking value set wrong. Because of that error, we took down the test too soon for significant results. But the test at least showed that the form banners hold their own. And what's significant here is that these banners held their own without any appeal letter on the landing page at all -- the form banners necessarily bypass the first step LP. Here are the results: +------------------------------------+-----------+---------+----------+----------+-----------+-----------+ | banner | donations | amount | amount18 | amount25 | avg25 | avg100 | +------------------------------------+-----------+---------+----------+----------+-----------+-----------+ | B12_061529_InlineWhiteAppeal_Jimmy | 129 | 2245.00 | 1481.00 | 1745.00 | 13.527132 | 17.403101 | | B12_062912_InlineWhiteFacts_Form | 95 | 2029.02 | 1165.02 | 1437.02 | 15.126526 | 21.358105 | | B12_062912_TopBlueFacts_Form | 154 | 2822.00 | 1808.00 | 2087.00 | 13.551948 | 16.376623 | +------------------------------------+-----------+---------+----------+----------+-----------+-----------+ http://en.wikipedia.org/wiki/Peace?banner=B12_061529_InlineWhiteAppeal_Jimmy Then we tested banners with the facts right in the banner -- but one with an ask and one with the traditional invitation to "read a personal appeal": +-----------------------------------------+-----------+---------+----------+----------+-----------+-----------+ | banner | donations | amount | amount18 | amount25 | avg25 | avg100 | +-----------------------------------------+-----------+---------+----------+----------+-----------+-----------+ | B12_061529_InlineWhiteAppeal_Jimmy | 355 | 6846.00 | 3995.00 | 4685.00 | 13.197183 | 18.861972 | | B12_062912_InlineWhiteFacts_Jimmy | 405 | 9237.00 | 5108.00 | 6123.00 | 15.118519 | 21.696296 | | B12_062912_InlineWhiteFacts_JimmyAppeal | 333 | 6012.82 | 3874.82 | 4544.82 | 13.648108 | 18.056517 | +-----------------------------------------+-----------+---------+----------+----------+-----------+-----------+ http://en.wikipedia.org/wiki/Peace?banner=B12_061529_InlineWhiteAppeal_Jimmy Finally, we tested colors... We thought that maybe the link blue color was magic, but maybe it's just extreme contrast. +------------------------------+-----+------------------------------+-----------+---------+----------+-----------+ | banner | lpi | banner | donations | amount | amount25 | avg25 | +------------------------------+-----+------------------------------+-----------+---------+----------+-----------+ | B12_062912_TopBlackFacts | 333 | B12_062912_TopBlackFacts | 66 | 2491.00 | 990.00 | 15.000000 | | B12_062912_TopBlueFacts | 344 | B12_062912_TopBlueFacts | 57 | 977.00 | 802.00 | 14.070175 | | B12_062912_TopLightblueFacts | 292 | B12_062912_TopLightblueFacts | 37 | 477.00 | 452.00 | 12.216216 | | B12_062912_TopWhiteFacts | 278 | B12_062912_TopWhiteFacts | 31 | 770.00 | 395.00 | 12.741935 | +------------------------------+-----+------------------------------+-----------+---------+----------+-----------+ http://en.wikipedia.org/wiki/Peace?banner=B12_062912_TopBlackFacts LPI, the landing page impression count, is a proxy for click rate. But we goofed by using the banner text with the ask. We should have used the text with the "please read a personal appeal" because that would get a higher click rate and we'd have a better senese of how color effects page views. The numbers in that third test are all too low to be reliable. |
July 2012 edit
| July 3 -- Settling a quick question |
|---|
|
Today we're just trying to answer one simple question with a short 15 minute test. We know that a face in a banner increases click rate, and we know that a deep blue or black background with white text increases click rate (over the old black text on white). But are the effects of each cumulative? So we're comparing these two: http://en.wikipedia.org/wiki/Peace?banner=B12_070312_FaceTopBlueFactsask http://en.wikipedia.org/wiki/Peace?banner=B12_070312_NofaceTopBlueFactsask We are both betting face does 45% better in click rate than noface and 22% better in donation rate. |
| July 18: Colors! |
|---|
|
We still owe you a full report from the facet test... the results we complex so I procrastinated in posting them. We did a short test today to see if dark grey would do as well as darkish but bright blue. It looks like it almost did. +----------------------------------+------+----------------------------------+-----------+---------+----------+-----------+ | banner | lpi | banner | donations | amount | amount25 | avg25 | +----------------------------------+------+----------------------------------+-----------+---------+----------+-----------+ | B12_070612_NofaceTopGreyFactsask | 2813 | B12_070612_NofaceTopGreyFactsask | 219 | 4154.24 | 3054.24 | 13.946301 | | B12_071912_NofaceTopBlueFactsask | 2971 | B12_071912_NofaceTopBlueFactsask | 270 | 6069.50 | 3857.00 | 14.285185 | +----------------------------------+------+----------------------------------+-----------+---------+----------+-----------+ LPI (landing page impressions) is basically click rate because both banners get the same amount of banner impressions. In these raw results, Grey has 94% of the click rate of Blue, but only 81% of the donations and 79% of amount25. But according to how we're figuring statistical confidence, it's not actually a weird result at all. The Blue is probably having a tiny effect, but we can't be sure: And we can't trust the difference between the donations at all basically: So the good news is that Grey may be doing about as well as Blue. That's good because the Blue really annoys some people who are strongly opposed to adding visual distractions to Wikipedia. |
| July 19: Top vs Inline |
|---|
|
We tested this: http://en.wikipedia.org/wiki/Pareidolia?banner=B12_070612_NofaceTopGreyFactsask vs this: http://en.wikipedia.org/wiki/Pareidolia?banner=B12_071912_NofaceInlineGreyFactsask and got these results: +-------------------------------------+------+-------------------------------------+-----------+----------+----------+-----------+ | banner | lpi | banner | donations | amount | amount25 | avg25 | +-------------------------------------+------+-------------------------------------+-----------+----------+----------+-----------+ | B12_070612_NofaceTopGreyFactsask | 7473 | B12_070612_NofaceTopGreyFactsask | 540 | 9542.00 | 7212.00 | 13.355556 | | B12_071912_NofaceInlineGreyFactsask | 7040 | B12_071912_NofaceInlineGreyFactsask | 483 | 10249.15 | 6933.60 | 14.355280 | +-------------------------------------+------+-------------------------------------+-----------+----------+----------+-----------+ We ran this test for longer than the one you'll hear about next and I think we can trust the results somewhat. But it's not a huge difference. Basically though, top gets a slightly higher click rate and donation rate because of that. |
| July 19 Part 2: Top appeal vs Top facts |
|---|
|
We tested this: http://en.wikipedia.org/wiki/Pareidolia?banner=B12_070612_NofaceTopGreyPersonalappeal vs this: http://en.wikipedia.org/wiki/Pareidolia?banner=B12_070612_NofaceTopGreyFactsask The results: +----------------------------------------+------+----------------------------------------+-----------+---------+----------+-----------+ | banner | lpi | banner | donations | amount | amount25 | avg25 | +----------------------------------------+------+----------------------------------------+-----------+---------+----------+-----------+ | B12_070612_NofaceTopGreyFactsask | 1399 | B12_070612_NofaceTopGreyFactsask | 111 | 1922.00 | 1547.00 | 13.936937 | | B12_070612_NofaceTopGreyPersonalappeal | 2580 | B12_070612_NofaceTopGreyPersonalappeal | 64 | 1466.00 | 861.00 | 13.453125 | +----------------------------------------+------+----------------------------------------+-----------+---------+----------+-----------+ When we plug those in to our confidence test, it says "Yes, very big difference, you can trust it!" But I'm not so sure. We need to go back and repeat this. We ran it a short time and it might just not be enough data. We've seen other tests split that far early on and then even out. Here's the explanation I just thought of: At this early stage in a test, the data is not "normal" and therefore the tests are not valid. That seems obvious in hindsight, I wonder if it's true. |
| July 26: Another top banner face test, with appeal instead of facts |
|---|
|
As seen above, Face seems to hurt click rate a little and donation rate a lot, independently of click rate, when used with the Facts banner. We know in Inline Personal Appeal banners that Face helps by up to 30% in donations. So we just want to make sure that's also the case in the Top banners. So we're testing this first today: http://en.wikipedia.org/wiki/Pareidolia?banner=B12_072412_FaceTopGreyPersonalappeal vs http://en.wikipedia.org/wiki/Pareidolia?banner=B12_072412_NofaceTopGreyPersonalappeal In very early results it looks like Face is killing Noface. We've still got a ton of little tests to go before we know exactly what's going on. When we tested Face with the Facts banner, we used a picture that didn't blend in as well as this one. So we'll have to go back and do this one again. We also tried a Facts banner with the last line being the "Please read a personal appeal from..." that should have gone well with the Face. But the face was on the left and the appeal invitation was on the right. So we'll have to try the better Face photo on the right before we rule out using Face with Facts. Here are those results: +----------------------------------------+------+----------------------------------------+-----------+---------+----------+-----------+ | banner | lpi | banner | donations | amount | amount25 | avg25 | +----------------------------------------+------+----------------------------------------+-----------+---------+----------+-----------+ | B12_072412_FaceTopGreyPersonalappeal | 8169 | B12_072412_FaceTopGreyPersonalappeal | 216 | 4092.00 | 2982.00 | 13.805556 | | B12_072412_NofaceTopGreyPersonalappeal | 8202 | B12_072412_NofaceTopGreyPersonalappeal | 181 | 2869.80 | 2200.80 | 12.159116 | +----------------------------------------+------+----------------------------------------+-----------+---------+----------+-----------+ Can't we trust it? P-value says no: 0.083. But this is unexpected because we are used to the photo making such a huge difference even with a fairly small sample. Also bizzarre is that face has a slightly lower click rate -- and yet more donations. Even though these difference can be confidently attributed to chance, it goes against what we would expect with such a big difference as the face photo. This is more evidence that the face photos do not mean the same thing in the top banners as they do in the inline banners. So without a lot of confidence in the winner, we took Face Appeal and matched it with Noface Facts. (Remember that Facts Face did significantly worse than Facts Noface.) +--------------------------------------+------+--------------------------------------+-----------+---------+----------+-----------+ | banner | lpi | banner | donations | amount | amount25 | avg25 | +--------------------------------------+------+--------------------------------------+-----------+---------+----------+-----------+ | B12_072412_FaceTopGreyPersonalappeal | 9393 | B12_072412_FaceTopGreyPersonalappeal | 257 | 5502.00 | 3667.00 | 14.268482 | | B12_072412_NofaceTopGreyFactsask | 4899 | B12_072412_NofaceTopGreyFactsask | 367 | 7177.26 | 5137.26 | 13.997984 | +--------------------------------------+------+--------------------------------------+-----------+---------+----------+-----------+ And the Facts have it! And with lots of confidence. P-value = 0.0001. This is the tool I'm using right now -- the visualization of the results is helpful and there's a big chunk of explanation too. Facts beats Appeal by 43%! After watching that for a while, we wondered if maybe it was an unfair test because the font of the appeal line is smaller than we normally run it. So we started a new test and enlarged the test to see what would happen. Here's what happened: +--------------------------------------+------+--------------------------------------+-----------+---------+----------+-----------+ | banner | lpi | banner | donations | amount | amount25 | avg25 | +--------------------------------------+------+--------------------------------------+-----------+---------+----------+-----------+ | B12_072412_NofaceTopGreyFactsask | 3535 | B12_072412_NofaceTopGreyFactsask | 258 | 5371.50 | 3506.50 | 13.591085 | | B12_072612_FaceTopGreyPersonalappeal | 5898 | B12_072612_FaceTopGreyPersonalappeal | 145 | 2713.00 | 1913.00 | 13.193103 | +--------------------------------------+------+--------------------------------------+-----------+---------+----------+-----------+ Now Facts wins by 77%! And with a ton of confidence. The Facts work! This is good news because we feel like there are more directions we can go in with all that text in the banner. More possibilities than when we're boxed into the once-unbeatable phrase, "Please read, a personal appeal by Wikipedia {role} {name}" |
August 2012 edit
| Aug 21: Trying to figure out what's going on in Spain |
|---|
|
We did a test for a full day across all Spanish Wikipedia. It was a multivariate test. One of the variables was Facts banner vs Appeal. None of the results below are confident - the sample's too small. We're just trying to test the infrastructure now. But one troubling thing appeared in this test. Spain's numbers are really low compared the the first day of the fundraiser last year. What could be wrong with our forms or donation process in Spain? +--------------------+-------------+-------------+ | name | donations11 | donations12 | +--------------------+-------------+-------------+ | Spain | 1007 | 437 | (43%) | Mexico | 214 | 175 | (81%) | Chile | 46 | 69 | (150%) | Argentina | 39 | 64 | (164%) | United States | 35 | 47 | | Colombia | 41 | 45 | | Ecuador | 18 | 16 | +--------------------+-------------+-------------+ Could it just be that one of our banners that we were testing did really badly? +---------------+-----------------------------------+-----------+ | name | banner | donations | +---------------+-----------------------------------+-----------+ | Spain | B12_0815_JimmyControl_esES_ES | 237 | | Spain | B12_NoFaceTopGreyFacts_esES_FR_ES | 171 | | Mexico | B12_NoFaceTopGreyFacts_esMX_FR_MX | 87 | | Mexico | B12_0815_JimmyControl_esMX_MX | 81 | | Chile | B12_0815_JimmyControl_esCL_CL | 42 | | Argentina | B12_NoFaceTopGreyFacts_esAR_FR_AR | 34 | | Colombia | B12_0815_JimmyControl_es_CO | 33 | | Argentina | B12_0815_JimmyControl_esAR_AR | 28 | | Chile | B12_NoFaceTopGreyFacts_esCL_FR_CL | 27 | | United States | B12_0815_JimmyControl_es_US | 25 | | Colombia | B12_NoFaceTopGreyFacts_es_FR_CO | 10 | +---------------+-----------------------------------+-----------+ Nope. It did do worse in Spain. But not enough to explain that 40%. What could it be? Peter's going to be looking into it deep this week -- and here's the great part! -- he's in Spain! |
August 24, 2012 "If everyone donated $5.." in Facts banner. Hint: BIG FIND! edit
Pop quiz from Fundraiser 2011: What banner text did we finally find that actually beat the "Jimmy appeal" banner at some point during the fundraiser? Can't remember? I'll tell you!
Answer: "If everyone reading this donated $5, we would only have to fundraise for one day a year. Please donate to keep Wikipedia free. "
We know this line is really important in the appeal too. We've met with donors and listened to them tell us how important that line is in the Jimmy message. So we're way past due to test this line in our new facts banner.
Here's the test setup:
- Control facts banner: http://en.wikipedia.org/wiki/Control?banner=B12_082412_NofaceTopGreyFactsask
- Facts banner with $5 line: http://en.wikipedia.org/wiki/Five?banner=B12_082412_NofaceTopGreyFactsaskFive
AND THE VERY EXCITING RESULTS THAT MAKE MEGAN SMILE:
+--------------------------------------+-----------+----------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +--------------------------------------+-----------+----------+----------+-----------+-----------+ | B12_082412_NofaceTopGreyFactsask | 579 | 11500.00 | 7048.00 | 19.861831 | 12.172712 | | B12_082412_NofaceTopGreyFactsaskFive | 1041 | 14700.00 | 10283.00 | 14.121037 | 9.878002 | +--------------------------------------+-----------+----------+----------+-----------+-----------+
Here's the confidence on the number of donations:
I'm awake! Great test. Need to find a few more improvements like this before we launch in November.
4 September, 2012 Reality Checking Top facts vs Inline Jimmy edit
In June, we found a top facts banner that actually beat our Jimmy banner from last year. Big find! Over the past few months, we've gone through a bunch of variations of that top facts banner so we wanted to run a reality check on how well we're doing with our current best facts banner against where we were last year.
Today we're running:
- Inline Jimmy: http://en.wikipedia.org/wiki/Main_Page?banner=B12_082412_JW
- Top facts $5: http://en.wikipedia.org/wiki/Main_Page?banner=B12_082412_NofaceTopGreyFactsaskFive
Here are the half-time results after 30 minutes of testing:
+--------------------------------------+-----------+---------+----------+--------+ | banner | donations | amount | amount18 | max | +--------------------------------------+-----------+---------+----------+--------+ | B12_082412_JW | 180 | 3516.00 | 2110.00 | 100.00 | | B12_082412_NofaceTopGreyFactsaskFive | 304 | 5037.50 | 3150.50 | 300.00 | +--------------------------------------+-----------+---------+----------+--------+
And here are the results from the full hour of testing:
+--------------------------------------+-----------+---------+----------+--------+ | banner | donations | amount | amount18 | max | +--------------------------------------+-----------+---------+----------+--------+ | B12_082412_JW | 351 | 6993.00 | 4082.00 | 250.00 | | B12_082412_NofaceTopGreyFactsaskFive | 566 | 9322.50 | 5678.50 | 300.00 | +--------------------------------------+-----------+---------+----------+--------+
It's very encouraging to see this new banner we've been working on is coming out confidently ahead of where we were last year. And we still have a few months of testing to go before the campaign launches! Meganhernandez (talk) 15:04, 4 September 2012 (UTC)
6 September, 2012 Quick 10 minute Test without Google/Yahoo line edit
We're trying out some alternatives to the Google/Yahoo line in the facts banner. ("Google might have close to a million servers. Yahoo has something like 12,000 staff. We have 679 servers and 138 employees.") A brief background: We added this comparison with Google/Yahoo in 2011 and found that it was a huge gain for both donations and total amount raised. We're testing a couple new variations of the same concept.
Today we tested (very quickly- just 10 minutes):
- Google/yahoo Facts banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_082412_NofaceTopGreyFactsaskFive
- No google/yahoo line Facts banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_082412_NofaceTopGreyFactsaskFiveNoG
Here are the results (remember, after just 10 minutes - not long enough to be totally confident in these results):
+-----------------------------------------+-----------+---------+----------+ | banner | donations | amount | amount18 | +-----------------------------------------+-----------+---------+----------+ | B12_082412_NofaceTopGreyFactsaskFive | 293 | 4098.00 | 2957.00 | | B12_082412_NofaceTopGreyFactsaskFiveNoG | 218 | 3074.00 | 2073.00 | +-----------------------------------------+-----------+---------+----------+
6 September, 2012 Another quick 15-min Test: Trying to fix the Google/Yahoo line! edit
Potential good news here. We're trying to get away from the Google/Yahoo line that has caused some unhappiness/concern over the past year. We just ran a quick 15-min test (so, this is a test we'll want to run again to confirm the results), but it looks like we found a new variation of that line that might work just as well. Great potential news! (Numbers below.) We've been testing a bunch of different lines over the last few weeks, but this is the closest we've come so far.
We tested:
- Old text with the Google/Yahoo numbers: http://en.wikipedia.org/wiki/International_Olympic_Committee?banner=B12_082412_NofaceTopGreyFactsaskFive
- New text: http://en.wikipedia.org/wiki/International_Olympic_Committee?banner=B12_082412_NofaceTopGreyFactsaskFiveThous
And the numbers:
+-------------------------------------------+-----------+---------+----------+-----------+----------+ | banner | donations | amount | amount18 | avg | avg18 | +-------------------------------------------+-----------+---------+----------+-----------+----------+ | B12_082412_NofaceTopGreyFactsaskFive | 407 | 6103.31 | 4012.81 | 14.995848 | 9.859484 | | B12_082412_NofaceTopGreyFactsaskFiveThous | 409 | 5311.00 | 3816.00 | 12.985330 | 9.330073 | +-------------------------------------------+-----------+---------+----------+-----------+----------+
So really exciting result. It was a short test, so we'll re-run it to confirm. More later, running to meet a donor!
7 September, 2012 Repeating the Google line test from yesterday edit
Here are the results combined from yesterday and today:
+-------------------------------------------+-----------+----------+----------+-----------+----------+ | banner | donations | amount | amount18 | avg | avg18 | +-------------------------------------------+-----------+----------+----------+-----------+----------+ | B12_082412_NofaceTopGreyFactsaskFive | 1114 | 15928.31 | 10981.81 | 14.298303 | 9.857998 | | B12_082412_NofaceTopGreyFactsaskFiveThous | 1022 | 13860.00 | 9684.00 | 13.561644 | 9.475538 | +-------------------------------------------+-----------+----------+----------+-----------+----------+
September 12, 2012 Retesting the Google/Yahoo line edit
We just reran the test from last week:
- Google millions facts banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_082412_NofaceTopGreyFactsaskFive
- Google thousands facts banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_082412_NofaceTopGreyFactsaskFiveThous
Here are the results from the one-hour test this morning:
+-------------------------------------------+-----------+----------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +-------------------------------------------+-----------+----------+----------+-----------+-----------+ | B12_082412_NofaceTopGreyFactsaskFive | 701 | 10754.00 | 7206.00 | 15.340942 | 10.279601 | | B12_082412_NofaceTopGreyFactsaskFiveThous | 636 | 10426.00 | 6239.00 | 16.393082 | 9.809748 | +-------------------------------------------+-----------+----------+----------+-----------+-----------+
Confidence testing on number of donations:
And here are the combined results from last week and today:
+-------------------------------------------+-----------+----------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +-------------------------------------------+-----------+----------+----------+-----------+-----------+ | B12_082412_NofaceTopGreyFactsaskFive | 1857 | 27607.31 | 18626.81 | 14.866618 | 10.030592 | | B12_082412_NofaceTopGreyFactsaskFiveThous | 1695 | 24939.00 | 16302.00 | 14.713274 | 9.617699 | +-------------------------------------------+-----------+----------+----------+-----------+-----------+
Confidence on number of donations from the combined results:
Sept. 14, 2012 Expander Banner edit
Wait! After this test ran (and I posted the update), we saw there was a mistake in the ask amounts on the new expanding/dropdown form. We somehow accidentally swapped out the $20 option and the ask amounts were different on the two forms. Skip down to the Sept. 17 update to see the results from this test once we fixed the ask button. Meganhernandez (talk) 18:17, 18 September 2012 (UTC)
We're testing a new type of banner today. The banner stays in place when you click on it and the donation form drops down from the banner instead of taking you to a new page.
Take a look:
- Control facts banner: http://en.wikipedia.org/wiki/2012?banner=B12_082412_NofaceTopGreyFactsaskFiveThous
- New expanding/dropdown banner: http://en.wikipedia.org/wiki/September_14?banner=B12_Expandable_Facts_PlainButtons_Image
The results:
+-------------------------------------------+-----------+----------+----------+-----------+----------+ | banner | donations | amount | amount18 | avg | avg18 | +-------------------------------------------+-----------+----------+----------+-----------+----------+ | B12_082412_NofaceTopGreyFactsaskFiveThous | 941 | 13245.11 | 8987.11 | 14.075569 | 9.550595 | | B12_Expandable_Facts_PlainButtons_Image | 1054 | 13821.59 | 9751.59 | 13.113463 | 9.251983 | +-------------------------------------------+-----------+----------+----------+-----------+----------+
Confidence:
September 17, 2012 Expanding/dropdown banner test edit
We're re-testing the dropdown banner.
The banners:
- Control facts banner: http://en.wikipedia.org/wiki/Fact?banner=B12_082412_NofaceTopGreyFactsaskFiveThous
- Dropdown facts banner: http://en.wikipedia.org/wiki/Dropkick?banner=B12_Expandable_Facts_PlainButtons_Image_Amts
The results:
+----------------------------------------------+-----------+----------+----------+-----------+----------+ | banner | donations | amount | amount18 | avg | avg18 | +----------------------------------------------+-----------+----------+----------+-----------+----------+ | B12_082412_NofaceTopGreyFactsaskFiveThous | 673 | 10465.25 | 6392.65 | 15.550149 | 9.498737 | | B12_Expandable_Facts_PlainButtons_Image_Amts | 814 | 10438.77 | 7647.77 | 12.824042 | 9.395295 | +----------------------------------------------+-----------+----------+----------+-----------+----------+
September 20, 2012 Less wordy facts & Amazon test edit
We ran a multivariate test in the US today. I'll report on the landing page test first, since the results from that test were more inspirational than the banner test results.
Amazon landing page test
We tested a form offering Amazon Payments as a new payment method.
Our control form allows users to donate by:
- Credit/Debit card
- PayPal
The new form included:
- Credit/Debit card
- PayPal
- Amazon
Here are the pages:
- Control credit/PayPal: https://donate.wikimedia.org/wiki/L12_0628_norec_JW/en
- New form with Amazon: https://donate.wikimedia.org/wiki/L12_0917_amazon_JW/en
Results:
Results by landing page:
Results broken out by payment method:
+--------------------+----------------+-----------+---------+----------+-----------+-----------+ | landing_page | payment_method | donations | amount | amount18 | avg | avg18 | +--------------------+----------------+-----------+---------+----------+-----------+-----------+ | L12_0628_norec_JW | cc | 266 | 4825.03 | 2930.47 | 18.139211 | 11.016805 | | L12_0628_norec_JW | pp | 295 | 3430.99 | 2669.99 | 11.630475 | 9.050814 | | L12_0917_amazon_JW | amazon | 129 | 1552.00 | 1196.00 | 12.031008 | 9.271318 | | L12_0917_amazon_JW | cc | 231 | 4069.10 | 2518.10 | 17.615152 | 10.900866 | | L12_0917_amazon_JW | pp | 261 | 2836.00 | 2256.00 | 10.865900 | 8.643678 | +--------------------+----------------+-----------+---------+----------+-----------+-----------+
So, it looks like offering Amazon may be increasing the number of donations coming in, and could possibly also affect the amount raised (look at amt18). We'll run some confidence tests on those results and post back. We will probably want to re-run this test for longer at some point since the results look pretty close. It's really encouraging if we're making it easier for more readers to donate.
Banner Test
We tested our control facts banner against a new banner with fewer words and with vertical lines separating each fact.
- Control facts banner: http://en.wikipedia.org/wiki/Words?banner=B12_082412_NofaceTopGreyFactsaskFiveThous
- New banner with fewer words & vertical lines: http://en.wikipedia.org/wiki/Vertical_direction?banner=B12_091912_Facts_LessText_LinesBold
And the banner results which really bum Megan out:
+-------------------------------------------+-----------+---------+----------+-----------+----------+ | banner | donations | amount | amount18 | avg | avg18 | +-------------------------------------------+-----------+---------+----------+-----------+----------+ | B12_082412_NofaceTopGreyFactsaskFiveThous | 699 | 9688.24 | 6788.24 | 13.860143 | 9.711359 | | B12_091912_Facts_LessText_LinesBold | 483 | 7024.88 | 4782.32 | 14.544265 | 9.901284 | +-------------------------------------------+-----------+---------+----------+-----------+----------+
I thought we could make the banner easier for people to read if we made it less wordy, made the font size bigger and added some lines separating each fact. Clearly, I was wrong. We still have a ton of variations of that facts banner to test though. Improvements on the way!
September 24, 2012 Retesting a 2011 campaign edit
Results from Sept. 24, 2012:
+------------------+-----------+----------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +------------------+-----------+----------+----------+-----------+-----------+ | B12_0924_btPf_US | 511 | 11548.00 | 6718.00 | 22.598826 | 13.146771 | +------------------+-----------+----------+----------+-----------+-----------+
Results from Nov. 15, 2011:
+------------------+-----------+----------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +------------------+-----------+----------+----------+-----------+-----------+ | B11_1115_btPf_US | 543 | 15105.00 | 8058.00 | 27.817680 | 14.839779 | +------------------+-----------+----------+----------+-----------+-----------+
September 26, 2012 More Dropdown banner testing edit
Last week (Sept. 17), we tested a new dropdown/expander banner and it looked like we saw a big increase from the standard facts banner. We showed this new banner around to some donors and got some great feedback that it wasn't necessarily clear to everyone that they were supposed to click on the banner.
So today we tested a couple variations of dropdown banners (with arrows & butons) and tweaked the messaging a bit.
The banners:
- Arrow & control text : http://en.wikipedia.org/wiki/Arrow?banner=B12_ExfactsTabarrowSeptext
- Button & control text: http://en.wikipedia.org/wiki/Button?banner=B12_ExfactsTabbuttonSeptext
- Arrow & new "long way" text: http://en.wikipedia.org/wiki/L%C3%B3ng?banner=B12_ExfactsTabarrowLongway
- Button & new "long way" text: http://en.wikipedia.org/wiki/The_Way_(film)?banner=B12_ExfactsTabbuttonLongway
The results:
+-----------------------------+-----------+---------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +-----------------------------+-----------+---------+----------+-----------+-----------+ | B12_ExfactsTabarrowLongway | 376 | 5998.00 | 3927.00 | 15.952128 | 10.444149 | | B12_ExfactsTabarrowSeptext | 454 | 6719.00 | 4422.00 | 14.799559 | 9.740088 | | B12_ExfactsTabbuttonLongway | 329 | 4879.37 | 3358.37 | 14.830912 | 10.207812 | | B12_ExfactsTabbuttonSeptext | 418 | 6018.00 | 4075.00 | 14.397129 | 9.748804 | +-----------------------------+-----------+---------+----------+-----------+-----------+
Arrow vs. Button:
It looks like Arrow is a little ahead of Button, but not with a ton of confidence .
Control vs. new text:
We have more confidence that our control text comes brings in more donations than the new "donations go a long way" text.
October 1 "Forget fundraising & get back to Wikipedia" and Amazon edit
Zack came up with a new fact for the banner and it looks like it's helping big time!
- Control facts banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1001_Facts
- New "back to Wikipedia" facts banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1001_FactsBackto
Results
+----------------------+-----------+----------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +----------------------+-----------+----------+----------+-----------+-----------+ | B12_1001_Facts | 848 | 12163.85 | 8069.85 | 14.344163 | 9.516333 | | B12_1001_FactsBackto | 1123 | 15620.59 | 10474.65 | 13.909697 | 9.327382 | | B12_1001_Half1Cols | 853 | 11620.75 | 8701.75 | 13.623388 | 10.201348 | +----------------------+-----------+----------+----------+-----------+-----------+
The last fact on the "back to Wikipedia" banner helps a lot. We'll test more to see if the "one hour" and "back to Wikipedia" texts are both helping.
The "half of 1%" is also doing about as well as our control facts banner. Which is great since it's missing our really strong fact comparing Wikipedia to Google and Yahoo. There must be parts of this banner that are helping and hurting donations. We'll test more to isolate the parts that are helping.
Amazon retest edit
We also re-tested offering Amazon as a payment option (first test was Sept. 20).
- Credit card & Paypal donation page: https://donate.wikimedia.org/wiki/L12_0628_norec_JW/en
- Credit card, Paypal & Amazon LP: https://donate.wikimedia.org/wiki/L12_0917_amazon_JW/en
Results by landing page
Results by payment method
+--------------------+----------------+-----------+----------+----------+-----------+-----------+ | L12_0628_norec_JW | cc | 619 | 9945.17 | 6480.23 | 16.066511 | 10.468869 | | L12_0628_norec_JW | pp | 723 | 10206.15 | 6695.15 | 14.116390 | 9.260235 | | L12_0917_amazon_JW | amazon | 327 | 4327.15 | 2953.15 | 13.232875 | 9.031040 | | L12_0917_amazon_JW | cc | 489 | 7248.78 | 5096.78 | 14.823681 | 10.422863 | | L12_0917_amazon_JW | pp | 666 | 7677.94 | 6020.94 | 11.528438 | 9.040450 | +--------------------+----------------+-----------+----------+----------+-----------+-----------+
Our two Amazon tests (Sept. 20 & Oct. 1) show the Amazon page coming slightly ahead. We'll run the confidence test on the amount raised later, but both tests look like offering Amazon along with credit card and Paypal increases the amount we raise. (Plus hopefully makes some people happier by having their preferred payment method available :)
ITALIAN TEST SEPTEMBER 30 - OCTOBER 7, 2012 edit
Ask amount Test €5 vs €10 edit
+-------------------------------------------+-----------+----------+----------+-----------+-----------+ | landing_page | donations | amount | amount18 | avg | avg18 | +-------------------------------------------+-----------+----------+----------+-----------+-----------+ | default~default~default~default~control | 1269 | 19764.79 | 13191.77 | 15.575091 | 10.395406 | | default~default~default~default~variable1 | 1026 | 17323.41 | 12624.41 | 16.884415 | 12.304493 | +-------------------------------------------+-----------+----------+----------+-----------+-----------+
-s 20120930174200 -e 20121002132400 -c C12_iwpitWW_FR_IT -l -t
Jimmy photo-at-thon edit
Banners:
- http://it.wikipedia.org/wiki/The_Jacket?banner=B12_InPaJwJkRn_IT
- http://it.wikipedia.org/wiki/Face?banner=B12_InPaJwPfRn_IT
- http://it.wikipedia.org/wiki/Space_Jam?banner=B12_InPaJwKsRn_IT
- http://it.wikipedia.org/wiki/Pagina_principale?banner=B12_InPaJwJoRn_IT
+----------------------+-----------+---------+----------+------------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +----------------------+-----------+---------+----------+------------+-----------+ | B12_InPaJwJkRn_IT_IT | 316 | 4808.21 | 3580.76 | 15.215854 | 11.331519 | | B12_InPaJwJoRn_IT_IT | 315 | 4544.40 | 3503.32 | 14.426667 | 11.121651 | | B12_InPaJwKsRn_IT_IT | 294 | 4382.08 | 3344.86 | 14.905034 | 11.377075 | | B12_InPaJwPfRn_IT_IT | 346 | 5508.44 | 3967.21 | 15.920347 | 11.465925 | +----------------------+-----------+---------+----------+------------+-----------+
ecom -s 20120930174200 -e 20121001095700 -c C12_iwpitWW_FR -b -t
Green background Jimmy vs White background Jimmy edit
- Green: http://it.wikipedia.org/wiki/Pagina_principale?banner=B12_InPaJwG2Rn_IT
- White: http://it.wikipedia.org/wiki/Pagina_principale?banner=B12_InPaJwJoRn_IT
+----------------------+-----------+----------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +----------------------+-----------+----------+----------+-----------+-----------+ | B12_InPaJwG2Rn_IT_IT | 1139 | 15783.10 | 11688.99 | 13.856980 | 10.262502 | | B12_InPaJwJoRn_IT_IT | 1013 | 14156.89 | 10358.20 | 13.975212 | 10.225271 | +----------------------+-----------+----------+----------+-----------+-----------+
-s 20121002132400 -e 20121003191100 -c C12_iwpitWW_FR_IT -b -t
Jimmy vs dropdown edit
- Jimmy banner: http://it.wikipedia.org/wiki/Comune_italiano?banner=B12_InPaJwG3Rn_IT
- Dropdown banner: http://it.wikipedia.org/wiki/Palazzi_di_Cesena?banner=B12_ExpandFactsImage_IT
+-------------------------+-----------+---------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +-------------------------+-----------+---------+----------+-----------+-----------+ | B12_ExpandFactsImage_IT | 755 | 7482.60 | 6447.42 | 9.910728 | 8.539629 | | B12_InPaJwG3Rn_IT_IT | 702 | 8955.38 | 6943.37 | 12.756952 | 9.890840 | +-------------------------+-----------+---------+----------+-----------+-----------+
ecom -s 20121003191100 -e 20121004113100 -c C12_iwpitWW_FR -b -t
Italy Grey dropdown facts vs blue modified facts edit
- grey facts:http://it.wikipedia.org/wiki/Pagina_principale?banner=B12_ExpandFactsImage_IT
- blue (fact 3: focus on wikipedia , 20 average amount highlighted, $5/1 day on right side of form): http://it.wikipedia.org/wiki/Pagina_principale?banner=B12_ExpandHm
+-------------------------+-----------+----------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +-------------------------+-----------+----------+----------+-----------+-----------+ | B12_ExpandFactsImage_IT | 954 | 10286.56 | 8348.97 | 10.782558 | 8.751541 | | B12_ExpandHm | 1452 | 16851.80 | 13411.37 | 11.605923 | 9.236481 | +-------------------------+-----------+----------+----------+-----------+-----------+
ecom -s 20121004113100 -e 20121005120000 -c C12_iwpitWW_FR -b -t
Italy blue modified facts vs New fact 3 edit
- Blue modified: http://it.wikipedia.org/wiki/Pagina_principale?banner=B12_ExpandHm2
- New 3rd fact translation: http://it.wikipedia.org/wiki/Pagina_principale?banner=B12_ExpandHmTrans
+-------------------------+-----------+---------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +-------------------------+-----------+---------+----------+-----------+-----------+ | B12_ExpandHm2 | 552 | 6173.19 | 5110.62 | 11.183315 | 9.258370 | | B12_ExpandHmTrans | 491 | 5724.97 | 4704.82 | 11.659817 | 9.582118 | +-------------------------+-----------+---------+----------+-----------+-----------+
ecom -s 20121005134500 -e 2012101500000000 -c C12_iwpitWW_FR -b -t
Reports left to run edit
- Second step address test
US Long & Short appeals from Dropdown - Friday October 5, 2012 edit
Our "facts banner" provides much more info directly from the banner than last year's "Please read: a personal appeal..." banner. Now that our readers are already learning the reasons to donate directly from the banner, we experimented today with showing a much shorter appeal than last year's appeal once you click on the banner. We're trying to limit repetition and make it quick and easy for people to donate.
- Long appeal: http://en.wikipedia.org/wiki/Main_Page?banner=B12_DropdownL
- Short appeal: http://en.wikipedia.org/wiki/Main_Page?banner=B12_DropdownS
+---------------+-----------+----------+----------+-----------+----------+ | banner | donations | amount | amount18 | avg | avg18 | +---------------+-----------+----------+----------+-----------+----------+ | B12_DropdownL | 1374 | 19980.58 | 13213.33 | 14.541907 | 9.616689 | | B12_DropdownS | 1478 | 20837.64 | 14281.64 | 14.098539 | 9.662815 | +---------------+-----------+----------+----------+-----------+----------+
ecom -s 20121005131000 -e 20121005171000 -c C12_1005_LvS -t
Monday October 8, 2012 EN T1 Jimmy vs Blue Facts edit
Hour 1:
- jimmy banner: http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_1008_Jimmyinlineappeal
- Bluetop facts: http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_1008_Backtobluefacts
Hour 2:
+------+-----------------------------+-----------+---------+----------+-----------+-----------+ | lpi | banner | donations | amount | amount18 | avg | avg18 | +------+-----------------------------+-----------+---------+----------+-----------+-----------+ | 4250 | B12_1008_Jimmyinlineappeal2 | 108 | 1997.65 | 1281.56 | 18.496759 | 11.866296 | | 3955 | B12_1008_Jimmyinlineappeal | 106 | 2884.08 | 1321.77 | 27.208302 | 12.469528 | | 3209 | B12_1008_Backtobluefacts | 375 | 5631.31 | 3684.99 | 15.016827 | 9.826640 | | 3091 | B12_1008_Backtobluefacts2 | 320 | 4198.77 | 2939.87 | 13.121156 | 9.187094 | +------+-----------------------------+-----------+---------+----------+-----------+-----------+
EN T2 (including GB) Jimmy vs. Blue Facts Tuesday October 9, 2012 edit
First 38 minutes when GB was included in EN T2:
- Jimmy banner; http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_1008_Jimmyinlineappeal_Gd
- Jimmy banner: http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_1008_Jimmyinlineappeal_Bd
- Blue banner: http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_1008_Backtobluefacts3
+-------------------------------+-----------+---------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +-------------------------------+-----------+---------+----------+-----------+-----------+ | B12_1008_Backtobluefacts3 | 427 | 5526.81 | 4613.39 | 12.943349 | 10.804192 | | B12_1008_Jimmyinlineappeal_Bd | 51 | 923.62 | 681.31 | 18.110196 | 13.359020 | | B12_1008_Jimmyinlineappeal_Gd | 73 | 1454.45 | 928.01 | 19.923973 | 12.712466 | +-------------------------------+-----------+---------+----------+-----------+-----------+
ecom -s 20121009081500 -e 20121009084800 -c C12_iwpenT2EurAf_FR -b --sql -m | db -t
GB-only Jimmy vs Blue facts banner Tuesday October 9, 2012 edit
- Jimmy banner: http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_1008_Jimmyinlineappeal_Bd_GB
+----------------------------------+-----------+---------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +----------------------------------+-----------+---------+----------+-----------+-----------+ | B12_1008_Backtobluefacts3_GB | 474 | 8285.70 | 6350.45 | 17.480380 | 13.397574 | | B12_1008_Jimmyinlineappeal_Bd_GB | 145 | 2719.86 | 2064.25 | 18.757655 | 14.236207 | | B12_1008_Jimmyinlineappeal_Gd_GB | 142 | 3018.45 | 2095.70 | 21.256690 | 14.758451 | +----------------------------------+-----------+---------+----------+-----------+-----------+
ecom -s 20121009090100 -e 20121009110900 -c C12_iwpenGB_FR -b --sql -m | db -t
EN T2 (excluding GB) Jimmy vs Blue facts Tuesday October 9, 2012 edit
note: removed GB from this T2 EN campaign and restarted this EN test from earlier in the morning.
Jimmy banner; http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_1008_Jimmyinlineappeal_Gd
Jimmy banner: http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_1008_Jimmyinlineappeal_Bd
blue banner: http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_1008_Backtobluefacts3
- NOTE: Blue banner was running at 50%, each Jimmy banner was running at 25% **
+-------------------------------+-----------+---------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +-------------------------------+-----------+---------+----------+-----------+-----------+ | B12_1008_Backtobluefacts3 | 682 | 9888.51 | 7075.81 | 14.499282 | 10.375088 | | B12_1008_Jimmyinlineappeal_Bd | 103 | 2175.41 | 1214.84 | 21.120485 | 11.794563 | | B12_1008_Jimmyinlineappeal_Gd | 117 | 1860.64 | 1368.90 | 15.902906 | 11.700000 | +-------------------------------+-----------+---------+----------+-----------+-----------+
ecom -s 20121009091200 -e 20121009102600 -c C12_iwpenT2EurAf_FR -b --sql -m | db -t
Japanese Jimmy Test Tuesday October 9, 2012 edit
Jimmy banner: http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_1009_JA2HR
+----------------+-----------+---------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +----------------+-----------+---------+----------+-----------+-----------+ | B12_1009_JA2HR | 152 | 4373.23 | 2221.48 | 28.771250 | 14.615000 | +----------------+-----------+---------+----------+-----------+-----------+
ecom -s 20121009071500 -e 20121009120000 -c C12_iwpjaWW_FR -b --sql -m | db -t+
2011 / 2012 Comparison
+----------+---------+----------+----------+---------+----------+ | 11_count | 11_amt | 11_amt18 | 12_count | 12_amt | 12_amt18 | +----------+---------+----------+----------+---------+----------+ | 195 | 6386.34 | 2882.21 | 139 | 4036.25 | 2031.64 | +----------+---------+----------+----------+---------+----------+
October 09, 2012 Tuesday US Dropdown tab test ("Please help" vs "Personal appeal") & highlighting the $20 average edit
We ran a multivariate test in the US today. We're testing a new tab for our dropdown banner with the text "Please read a personal appeal from Wikipedia's founder" against our control "Please help" tab.
We also tested two versions of the dropdown landing page: one with the $20 average amount highlighted, and one without $20 highlighted.
The banners:
- Personal appeal tab: http://en.wikipedia.org/wiki/Main_Page?banner=B12_PersonalAppeal_Tab
- Personal appeal tab / average donation: http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_PersonalAppeal_TabAve
- Please help tab: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1009_Plaintab2
- Please help tab // average donation: http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_1009_PlaintabAvg2
Results
+---------------------------+-----------+----------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +---------------------------+-----------+----------+----------+-----------+-----------+ | B12_1009_Plaintab2 | 874 | 10326.16 | 8034.16 | 11.814828 | 9.192403 | | B12_1009_PlaintabAvg2 | 910 | 13626.00 | 9458.00 | 14.973626 | 10.393407 | | B12_PersonalAppeal_Tab | 856 | 10719.80 | 7923.91 | 12.523131 | 9.256904 | | B12_PersonalAppeal_TabAve | 790 | 10334.86 | 7951.43 | 13.082101 | 10.065101 | +---------------------------+-----------+----------+----------+-----------+-----------+
ecom -c C12_US_1009_Avg2 -s 20121009142300 -e 201210092000 --sql | db -t
Combined results
Commentary: SO! What a great day (see also below). Above you see one interesting discovery and one exciting discovery. First, we have two great banners that work well for us: "Please read a personal appeal from Wikipedia founder" and a long "Facts" banner that makes the big points from our traditional appeal right in-your-face in the banner. Both work great. So wouldn't they work even better if we combined them in the same banner? We combined them today. But the combo didn't do well. It didn't do horribly, but significant drop vs the plain facts. (These Facts are out-performing the appeal banner in the US btw).
Second, we have been looking for a way to bring up the average donation. Last year we basically doubled our number of donations while not-quite halving our average. We're talking people to make a lower donation than they might actually favor doing with the $5 line. So we tried indicating that our average donation is about $20 on the landing page. It worked! The normalized average went up about $1.
EN T1 $5 vs "price of a sandwich" Tuesday October 9. 2012 edit
We are really excited about this test for two reasons:
1) The $5 line that we've been using is great for bringing in a ton of donations -- like tons of $5 donations! But that brings the average donation down too far. We're talking a lot of people who would probably like to give $20 into giving $5. By saying, "price of a sandwich" instead of $5, it makes it a sliding scale: how much does your typical sandwich cost? For some people $5, for others $10, and it wasn't that long ago that I had a sandwich for less than a dollar while traveling in a low-income country. And lo and behold, sandwich brought up the average by a whole dollar (looking at avg18) and two looking at plain avg (which can be thrown off by one large donation.)
2) The $5 line has another problem: We have to convert it into different currencies and amounts in different countries. In fact, testing late last night (for me) and early in the morning (for Megan) we forgot about this and subjected millions in Africa and Europe to "$5".
Next to try: "A couple cups of coffee", "A movie ticket", what else?
Banners:
+------------------------------+-----------+---------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +------------------------------+-----------+---------+----------+-----------+-----------+ | B12_1008_Backtobluefacts2 | 509 | 7459.98 | 4961.57 | 14.656149 | 9.747682 | | B12_1008_BacktobluefactsSand | 506 | 8360.60 | 5441.11 | 16.522925 | 10.753182 | +------------------------------+-----------+---------+----------+-----------+-----------+>
ecom -c C12_iwpenT1_FR -s 20121009150000 -e 20121009200000 --sql -m | db -t
October 11, 2012 Thursday US Sandwich & Average Test edit
We're working to increase our average donation in our testing this week. Over the last couple of testing months, we found a line that brings in a lot of donations, but lowers our average donation ("If everyone reading this donated $5, the fundraiser would be done in an hour.") We saw some promising results from the "price of a sandwich" test (update above) that ran outside of the US, so we wanted to confirm those results with a US test.
We also re-tested highlighting the $20 average on our donation page.
The banners:
- $5 / no average: http://en.wikipedia.org/wiki/Sandwich?banner=B12_1010_FiveNoavg
- $5 / average: http://en.wikipedia.org/wiki/Sandwich?banner=B12_1010_FiveAvg
- Sandwich / no average: http://en.wikipedia.org/wiki/Sandwich?banner=B12_1010_SandNoavg
- Sandwhich / average: http://en.wikipedia.org/wiki/Sandwich?banner=B12_1010_SandNoavg
The Results
+--------------------+-----------+---------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +--------------------+-----------+---------+----------+-----------+-----------+ | B12_1010_FiveAvg | 630 | 9558.00 | 6648.00 | 15.171429 | 10.552381 | | B12_1010_FiveNoavg | 624 | 8010.00 | 5783.00 | 12.836538 | 9.267628 | | B12_1010_SandAvg | 550 | 9841.73 | 6507.73 | 17.894055 | 11.832236 | | B12_1010_SandNoavg | 633 | 9310.00 | 6272.00 | 14.707741 | 9.908373 | +--------------------+-----------+---------+----------+-----------+-----------+
Combined Results
October 12, Friday US No Google line & Average Test edit
Today we tested a banner without the Google/Yahoo comparison. We also tested the new "average donation" page against our control landing page. (When we tested "average donation" page earlier this week, we eliminated the $250 radio button option.)
Banners
- No Google // no average: http://en.wikipedia.org/wiki/Extraterrestrial_life?banner=B12_1011_FactsNoGNAve
- No Google // average: http://en.wikipedia.org/wiki/Extraterrestrial_life?banner=B12_1011_FactsNoGAve
- Google // no average: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1012_FactsGNAve
- Google // average: http://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate/view&template=B12_1012_FactsGAve
Results
+-----------------------+-----------+---------+----------+-----------+-----------+ | banner | donations | amount | amount18 | avg | avg18 | +-----------------------+-----------+---------+----------+-----------+-----------+ | B12_1011_FactsNoGAve | 332 | 5259.50 | 3737.50 | 15.841867 | 11.257530 | | B12_1011_FactsNoGNAve | 397 | 6530.00 | 4051.00 | 16.448363 | 10.204030 | | B12_1012_FactsGAve | 469 | 8258.61 | 5504.61 | 17.608977 | 11.736908 | | B12_1012_FactsGNAve | 468 | 6738.00 | 4731.00 | 14.397436 | 10.108974 | +-----------------------+-----------+---------+----------+-----------+-----------+
Combined Results
October 16, 2012 Tuesday US "No government funds" edit
We found a new line for our facts banner that actually does a bit better than the Google/Yahoo comparison that has performed very well over the past year.
The banners
- Control Google/Yahoo banner: http://en.wikipedia.org/wiki/Google?banner=B12_1016_FactsGoog
- No government funds banner: http://en.wikipedia.org/wiki/Government?banner=B12_1016_FactsGovmore
The results
+-----------------------+-----------+-------+-------+-------+----------+----------+-----------+-----------+ | banner | donations | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +-----------------------+-----------+-------+-------+-------+----------+----------+-----------+-----------+ | B12_1016_FactsGoog | 1520 | 79% | 62% | 79% | 19336.82 | 14235.19 | 12.721592 | 9.365257 | | B12_1016_FactsGovmore | 1680 | 84% | 61% | 77% | 23991.12 | 17488.00 | 14.280429 | 10.409524 | +-----------------------+-----------+-------+-------+-------+----------+----------+-----------+-----------+
ecom -c C12_US_1016_GgvGv -s 20121016133800 -e 20121016173800 http://www.thumbtack.com/labs/abba/#B12_1016_FactsGoog=1520%2C1000000&B12_1016_FactsGovmore=1680%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true
Great news! Not only could this allow us to retire the Google line in the banner, but it also looks like this new text even performs a little better.
We're brainstorming some new versions to try to get our average back up. (Since we've been using the $5 line in the banner, we've increased the number of donations but decreased the average amount.)
More testing to come later this week.
October 19, 2012 US First Editor Test! edit
We've tested our first editor appeal of 2012. Last year, the messages from editors gave the campaign personality and introduced our volunteer community to our readers. It was the fun part of the fundraiser and the comments we got from readers were really inspiring.
Our first editor this year is a retired math teacher from India. Poongothai writes articles on quadratic functions, probability, charts, and graphs.
We tested Poongothai's appeal coming from our facts dropdown banner. Next step is to test a new version of the facts banner that includes her photo.
The banners
- Control facts banner: http://en.wikipedia.org/wiki/Fact?banner=B12_1018_FactsGovmore
- Editor banner: http://en.wikipedia.org/wiki/Editor?banner=B12_1018_FactsEditor
The Results
+-----------------------+-----------+-------+-------+-------+----------+----------+-----------+-----------+ | banner | donations | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +-----------------------+-----------+-------+-------+-------+----------+----------+-----------+-----------+ | B12_1018_FactsEditor | 768 | 42% | 62% | 79% | 12558.00 | 8228.00 | 16.351563 | 10.713542 | | B12_1018_FactsGovmore | 784 | 45% | 66% | 78% | 12164.08 | 8160.08 | 15.515408 | 10.408265 | +-----------------------+-----------+-------+-------+-------+----------+----------+-----------+-----------+
ecom -c C12_US_1015_Editor -s 20121019130500 -e 20121019160500
Those results are really encouraging! In previous years, our editor banners always received great feedback from our readers, but they always brought in fewer donations than our standard founder appeal. In our test this morning, the Jimmy appeal and the editor appeal performed just about the same. Both appeals had the exact same banner, so it looks like that facts banner is really doing the work of compelling readers to donate, and that the message we put on the appeal page has a lesser impact than it did when we used the "personal appeal" banners last year. Great first editor test!
Next week, we'll test a new editor banner that incorporates the facts along with the photo and an invitation to read the editor message. Can't wait.
October 22, 2012 US Comparing where we're at to 2011 edit
We duplicated a campaign setup from 2011 to see how our banner impressions, click rate, and donation rates are comparing to last year when we hold the creative content constant.
Here's the setup from 2011:
3 jimmy banners:
- http://en.wikipedia.org/wiki/Main_Page?banner=B11_1114_JKwfCf_US
- http://en.wikipedia.org/wiki/Main_Page?banner=B11_1114_JKwfLf_US
- http://en.wikipedia.org/wiki/Main_Page?banner=B11_Foundation_Jimmy_KevinSpaceywhite_US
1 landing page: http://wikimediafoundation.org/wiki/L11_1112_JBT/en/US
The setup from 2012:
3 Jimmy banners:
- http://en.wikipedia.org/wiki/Main_Page?banner=B12_1022_JKwfCf_US
- http://en.wikipedia.org/wiki/Main_Page?banner=B12_1022_JKwfLf_US
- http://en.wikipedia.org/wiki/Main_Page?banner=B12_Foundation_Jimmy_KevinSpaceywhite_US
1 landing page: https://donate.wikimedia.org/wiki/L11_1112_JBT/en/US
2011 US Donation Results
+------------------------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +------------------------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+ | B11_1114_JKwfCf_US | 497 | 722 | 41% | 59% | 78% | 11600.00 | 7449.00 | 23.340040 | 14.987928 | | B11_1114_JKwfLf_US | 513 | 772 | 44% | 55% | 79% | 13978.00 | 7655.00 | 27.247563 | 14.922027 | | B11_Foundation_Jimmy_KevinSpaceywhite_US | 536 | 834 | 49% | 58% | 71% | 12618.00 | 7965.00 | 23.541045 | 14.860075 | +------------------------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+
ecom -c C11_1114_CFLFKS_US -s 20111114182500 -e 20111114192700 -g b
2012 US Donation Results
+------------------------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +------------------------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+ | B12_1022_JKwfCf_US | 323 | 533 | 45% | 54% | 68% | 8594.30 | 4922.30 | 26.607740 | 15.239319 | | B12_1022_JKwfLf_US | 344 | 541 | 52% | 59% | 69% | 9002.54 | 5076.54 | 26.170174 | 14.757384 | | B12_Foundation_Jimmy_KevinSpaceywhite_US | 314 | 522 | 51% | 55% | 66% | 10153.35 | 4807.00 | 32.335510 | 15.308917 | +------------------------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+
ecom -c C12_US_1022_RepTst -s 20121022172500 -e 20121022182700 -g b
2011 Banner Impressions
| utm_source | sum(counts) | +------------------------------------------+-------------+ | B11_1114_JKwfCf_US | 1981114 | | B11_1114_JKwfLf_US | 1982674 | | B11_Foundation_Jimmy_KevinSpaceywhite_US | 1979817 |
2012 Banner Impressions
+------------------------------------------+------------+ | banner | sum(count) | +------------------------------------------+------------+ | B12_1022_JKwfCf_US | 1725400 | | B12_1022_JKwfLf_US | 1694200 | | B12_Foundation_Jimmy_KevinSpaceywhite_US | 1717600 | +------------------------------------------+------------+
2011 Landing page impressions
+-------+------------------------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+ | lpi | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +-------+------------------------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+ | 31199 | B11_Foundation_Jimmy_KevinSpaceywhite_US | 536 | 834 | 0% | 0% | 71% | 12618.00 | 7965.00 | 23.541045 | 14.860075 | | 30669 | B11_1114_JKwfLf_US | 513 | 772 | 0% | 0% | 79% | 13978.00 | 7655.00 | 27.247563 | 14.922027 | | 30279 | B11_1114_JKwfCf_US | 497 | 722 | 0% | 0% | 78% | 11600.00 | 7449.00 | 23.340040 | 14.987928 | +-------+------------------------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+
ecom -c C11_1114_CFLFKS_US -s 20111114182500 -e 20111114192700 -g b -i
2012 Landing page impressions
+-------+------------------------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+ | lpi | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +-------+------------------------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+ | 20642 | B12_Foundation_Jimmy_KevinSpaceywhite_US | 314 | 522 | 51% | 55% | 66% | 10153.35 | 4807.00 | 32.335510 | 15.308917 | | 20355 | B12_1022_JKwfLf_US | 344 | 541 | 52% | 59% | 69% | 9002.54 | 5076.54 | 26.170174 | 14.757384 | | 20077 | B12_1022_JKwfCf_US | 323 | 533 | 45% | 54% | 68% | 8594.30 | 4922.30 | 26.607740 | 15.239319 | +-------+------------------------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+
ecom -c C12_US_1022_RepTst -s 20121022172500 -e 20121022182700 -g b -i
October 22, 2012 EN T1 Comparison from Oct. 8, 2012 Test edit
Test setup from Oct. 8, 2012 and Oct. 22, 2012:
- Jimmy banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1008_Jimmyinlineappeal2
- Blue facts banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1008_Backtobluefacts2
Facts landing page (CA running from Jimmy banner): https://donate.wikimedia.org/wiki/Special:FundraiserLandingPage?uselang=en&country=CA&template=Lp-layout-default&appeal-template=Appeal-template-default&appeal=Appeal-facts&infobox=Appeal-default&form-template=Form-template-default&form-countryspecific=Form-countryspecific-control&utm_medium=sitenotice&utm_source=B12_1008_Jimmyinlineappeal2_CA&utm_campaign=C12_iwpenT1_FR_CA
October 8, 2012 Results:
+------+---------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | lpi | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +------+---------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | 4742 | B12_1008_Jimmyinlineappeal2 | 104 | 196 | 44% | 45% | 30% | 1956.78 | 1248.23 | 18.815192 | 12.002212 | | 3537 | B12_1008_Backtobluefacts2 | 309 | 568 | 34% | 38% | 69% | 4046.52 | 2855.49 | 13.095534 | 9.241068 | +------+---------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+
ecom -c C12_iwpenT1_FR -s 20121008180100 -e 20121008190000 -g b -m -i
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1008_Jimmyinlineappeal2=104%2C1000000&B12_1008_Backtobluefacts2=309%2C1000000&B12_1008_Backtobluefacts=9%2C1000000&B12_1008_Jimmyinlineappeal=3%2C1000000&B12_1008_Backtobluefacts_undefi=0%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true d/lpi confidence test: http://www.thumbtack.com/labs/abba/#B12_1008_Jimmyinlineappeal2=104%2C4742&B12_1008_Backtobluefacts2=309%2C3537&B12_1008_Backtobluefacts=9%2C167&B12_1008_Jimmyinlineappeal=3%2C81&B12_1008_Backtobluefacts_undefi=0%2C2&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true lpi/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1008_Jimmyinlineappeal2=4742%2C1000000&B12_1008_Backtobluefacts2=3537%2C1000000&B12_1008_Backtobluefacts=167%2C1000000&B12_1008_Jimmyinlineappeal=81%2C1000000&B12_1008_Backtobluefacts_undefi=2%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true
+------+-----------------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | lpi | landingpage | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +------+-----------------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | 6100 | default~default~default~default~control | 377 | 706 | 35% | 39% | 58% | 5242.07 | 3615.28 | 13.904695 | 9.589602 | | 2429 | default~default~facts~default~control | 48 | 88 | 46% | 45% | 30% | 880.13 | 597.37 | 18.336042 | 12.445208 | +------+-----------------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+
ecom -c C12_iwpenT1_FR -s 20121008180100 -e 20121008190000 -g l -m -i
Oct. 22 Results:
+------+-----------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | lpi | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +------+-----------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | 5722 | B12_1008_Jimmyinlineappeal2 | 103 | 228 | 48% | 38% | 24% | 2086.04 | 1133.73 | 20.252816 | 11.007087 | | 3555 | B12_1008_Backtobluefacts2 | 308 | 539 | 37% | 43% | 70% | 4296.99 | 3008.93 | 13.951266 | 9.769253 | +------+-----------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+
ecom -c C12_iwpenT1_FR -s 20121022180100 -e 20121022190000 -g b -m -i
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1008_Jimmyinlineappeal2=103%2C1000000&B12_1008_Backtobluefacts2=308%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true d/lpi confidence test: http://www.thumbtack.com/labs/abba/#B12_1008_Jimmyinlineappeal2=103%2C5722&B12_1008_Backtobluefacts2=308%2C3555&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true lpi/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1008_Jimmyinlineappeal2=5722%2C1000000&B12_1008_Backtobluefacts2=3555%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true
+------+-----------------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | lpi | landingpage | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +------+-----------------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | 6430 | default~default~default~default~control | 357 | 645 | 39% | 43% | 57% | 5243.58 | 3531.56 | 14.687899 | 9.892325 | | 2847 | default~default~facts~default~control | 54 | 122 | 44% | 36% | 25% | 1139.45 | 611.10 | 21.100926 | 11.316667 | +------+-----------------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+
ecom -c C12_iwpenT1_FR -s 20121022180100 -e 20121022190000 -g l -m -i
Oct. 22 Banner Impressions:
+-----------------------------+------------+ | banner | sum(count) | +-----------------------------+------------+ | B12_1008_Backtobluefacts2 | 459200 | | B12_1008_Jimmyinlineappeal2 | 474600 | +-----------------------------+------------+
(still need Oct. 8 banner impressions)
Oct. 23, 2012 GB Compare test edit
Setup from Oct. 9 and Oct. 23:
- Jimmy banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1008_Jimmyinlineappeal_Bd_GB
- Jimmy banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1008_Jimmyinlineappeal_Gd_GB
- Blue facts banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1008_Backtobluefacts3_GB
October 9 Results
+------+----------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | lpi | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +------+----------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | 8240 | B12_1008_Jimmyinlineappeal_Gd_GB | 137 | 245 | 36% | 49% | 36% | 2812.71 | 2020.81 | 20.530730 | 14.750438 | | 7016 | B12_1008_Jimmyinlineappeal_Bd_GB | 141 | 248 | 39% | 50% | 35% | 2671.65 | 2009.93 | 18.947872 | 14.254823 | | 4749 | B12_1008_Backtobluefacts3_GB | 460 | 688 | 34% | 59% | 72% | 8059.08 | 6160.17 | 17.519739 | 13.391674 | +------+----------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+
ecom -s 20121009090100 -e 20121009103000 -c C12_iwpenGB_FR -g b -m -i
October 23 Results
+------+----------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | lpi | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +------+----------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | 7438 | B12_1008_Jimmyinlineappeal_Gd_GB | 124 | 228 | 35% | 40% | 36% | 2522.54 | 1869.95 | 20.343065 | 15.080242 | | 6853 | B12_1008_Jimmyinlineappeal_Bd_GB | 124 | 229 | 34% | 40% | 36% | 2628.93 | 1781.16 | 21.201048 | 14.364194 | | 4392 | B12_1008_Backtobluefacts3_GB | 391 | 583 | 33% | 56% | 75% | 6327.43 | 4966.44 | 16.182685 | 12.701893 | +------+----------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+
ecom -s 20121023090100 -e 20121023103000 -c C12_iwpenGB_FR -g b -m -i
- Still need banner impression comparison
October 23, 2012 ENT4 Jimmy vs Blue Facts Test edit
Test setup:
- Jimmy banner: http://en.wikipedia.org/wiki/Australia?banner=B12_1024_T4_Proto_Jimmy
- Blue facts: http://en.wikipedia.org/wiki/Australia?banner=B12_1024_T4_Proto_FactsBlueSWDonate
- Landing page (Australia): https://donate.wikimedia.org/wiki/Special:FundraiserLandingPage?uselang=en&country=AU&template=Lp-layout-default&appeal-template=Appeal-template-default&appeal=Appeal-default&infobox=Default&form-template=Form-template-default&form-countryspecific=Form-countryspecific-control&utm_medium=sitenotice&utm_source=B12_1024_T4_Proto_Jimmy_US&utm_campaign=none_US
Full T4 Results
+------+-------------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | lpi | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +------+-------------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | 3970 | B12_1024_T4_Proto_Jimmy | 100 | 199 | 49% | 55% | 70% | 3922.20 | 1291.96 | 39.222000 | 12.919600 | | 2366 | B12_1024_T4_Proto_FactsBlueSWDonate | 259 | 423 | 41% | 57% | 79% | 3545.10 | 2547.98 | 13.687645 | 9.837761 | +------+-------------------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+
ecom -s 20121023220000 -e 20121023230000 -c C12_iwpenT4EastAsia_FR -g b -m -i
AU & NZ BANNER breakdown:
+------+-------------------------------------+----------+-----------+--------+-------+-------+-------+---------+----------+------------+-----------+ | lpi | banner | iso_code | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +------+-------------------------------------+----------+-----------+--------+-------+-------+-------+---------+----------+------------+-----------+ | 2621 | B12_1024_T4_Proto_Jimmy | AU | 77 | 77 | 43% | 100% | 100% | 3451.08 | 965.28 | 44.819221 | 12.536104 | | 1514 | B12_1024_T4_Proto_FactsBlueSWDonate | AU | 186 | 186 | 35% | 100% | 100% | 2708.22 | 2011.17 | 14.560323 | 10.812742 | | 745 | B12_1024_T4_Proto_Jimmy | NZ | 19 | 19 | 74% | 100% | 100% | 417.81 | 275.80 | 21.990000 | 14.515789 | | 1514 | B12_1024_T4_Proto_FactsBlueSWDonate | NZ | 61 | 61 | 59% | 100% | 100% | 500.19 | 411.19 | 8.199836 | 6.740820 | +------+-------------------------------------+----------+-----------+--------+-------+-------+-------+---------+----------+------------+-----------+
ecom -s 20121023220000 -e 20121023230000 -c C12_iwpenT4EastAsia_FR -g bc -m -i
AU & NZ Banner Impression breakdown
+-------------------------------------+---------+----------+ | banner | country | ban_imps | +-------------------------------------+---------+----------+ | B12_1024_T4_Proto_FactsBlueSWDonate | AU | 1377 | | B12_1024_T4_Proto_Jimmy | AU | 1342 | | B12_1024_T4_Proto_FactsBlueSWDonate | NZ | 386 | | B12_1024_T4_Proto_Jimmy | NZ | 390 | +-------------------------------------+---------+----------+
ecom -s 20121023220000 -e 20121023230000 -c C12_iwpenT4EastAsia_FR -g bc -m -i
Time Machine Comparison to 2011 for AU
- PLEASE NOTE: In 2011, Jimmy was running 100%. In 2012, Jimmy was running 50% and blue facts were running 50%.
+----------+---------+----------+-----------+----------+---------+----------+-----------+ | 11_count | 11_amt | 11_amt18 | 11_avg18 | 12_count | 12_amt | 12_amt18 | 12_avg18 | +----------+---------+----------+-----------+----------+---------+----------+-----------+ | 263 | 4952.75 | 3357.46 | 12.766008 | 254 | 5880.73 | 2855.49 | 11.242087 | +----------+---------+----------+-----------+----------+---------+----------+-----------+
timemachine -c C12_iwpenT4EastAsia_FR --sd11 20111116 --sd12 20121023 --st 220000 --et 230000 --country AU --lang en
- PLEASE NOTE: In 2011, Jimmy was running 100%. In 2012, Jimmy was running 50% and blue facts were running 50%.
- If we take just the Jimmy banner results for 2012 and multiply the donations by 2, we can estimate what our AU total would be if we ran Jimmy at 100% in 2012.
- 77 donations X 2 = 154 compared to 263 donations in 2011
- $965 Amt18 X 2 = $1,930 compared to $3,357 in 2011
Time Machine Comparison to 2011 for NZ
- PLEASE NOTE: In 2011, Jimmy was running 100%. In 2012, Jimmy was running 50% and blue facts were running 50%.
Comparison to Nov. 16, 2011
+----------+--------+----------+-----------+----------+--------+----------+----------+ | 11_count | 11_amt | 11_amt18 | 11_avg18 | 12_count | 12_amt | 12_amt18 | 12_avg18 | +----------+--------+----------+-----------+----------+--------+----------+----------+ | 20 | 208.29 | 204.46 | 10.223000 | 78 | 909.80 | 678.79 | 8.702436 | +----------+--------+----------+-----------+----------+--------+----------+----------+
timemachine -c C12_iwpenT4EastAsia_FR --sd11 20111116 --sd12 20121023 --st 220000 --et 230000 --country NZ --lang en
- PLEASE NOTE: In 2011, Jimmy was running 100%. In 2012, Jimmy was running 50% and blue facts were running 50%.
- If we take just the Jimmy banner results for 2012 and multiply the donations by 2, we can estimate what our NZ total would be if we ran Jimmy at 100% in 2012.
- 19 donations X 2 = 38 donations compared to 20 donations in 2011
- $275.80 Amount18 X 2 = $551.6 compared to $204.46 in 2011
Comparison to Nov. 17, 2011:
+----------+--------+----------+----------+----------+--------+----------+----------+ | 11_count | 11_amt | 11_amt18 | 11_avg18 | 12_count | 12_amt | 12_amt18 | 12_avg18 | +----------+--------+----------+----------+----------+--------+----------+----------+ | 17 | 146.59 | 145.40 | 8.552941 | 78 | 909.80 | 678.79 | 8.702436 | +----------+--------+----------+----------+----------+--------+----------+----------+
timemachine -c C12_iwpenT4EastAsia_FR --sd11 20111117 --sd12 20121023 --st 220000 --et 230000 --country NZ --lang en
- PLEASE NOTE: In 2011, Jimmy was running 100%. In 2012, Jimmy was running 50% and blue facts were running 50%.
- If we take just the Jimmy banner results for 2012 and multiply the donations by 2, we can estimate what our NZ total would be if we ran Jimmy at 100% in 2012.
- 19 donations X 2 = 38 donations compared to 20 donations in 2011
- $275.80 Amount18 X 2 = $551.6 compared to $204.46 in 2011
Friday November 9, 2012 US Paypal credit card option edit
We tested different presentations of payment methods in the US.
Page A had three buttons:
- Credit Card (credit card donations were processed through Global Collect)
- PayPal
- Amazon
Page B had two buttons:
- PayPal or Credit Card (credit card donations were processed through PayPal)
- Amazon
Results
We don't have confidence on the results for number of donations, but it looks like processing our credit card donations through PayPal did not help increase donations. We will test this again outside the US, particularly in countries where the proportion of PayPal donations is higher than in the US.
ecom -s 20121109220000 -e 20121109250000 -c C12_iwpenUS_FR -g b
+------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+----------+ | B12_FpcwDropBtnUS_A_US | 1840 | 2683 | 36% | 55% | 84% | 27221.32 | 18315.21 | 14.794196 | 9.953918 | | B12_FpcwDropBtnUS_B_US | 1787 | 2610 | 0% | None | 67% | 24310.83 | 16951.33 | 13.604270 | 9.485915 | +------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_FpcwDropBtnUS_A_US=1840%2C1000000&B12_FpcwDropBtnUS_B_US=1787%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true
US Tuesday Nov. 13, 2012 Trying to get editors in the banner edit
We ran through a few tests this morning of different versions of our facts banner with the image of an editor.
Quick refresher: Last time we tested an editor appeal (Oct. 19), we used two identical facts banners (text-only, no images). One banner led to our standard Jimmy appeal and the other banner led to an editor appeal. The number of donations and amounts were nearly the same. Very encouraging, but the next step was to integrate the editor into the facts banner. Which is what we tried today.
Round 1:
- Control facts banner: http://en.wikipedia.org/wiki/Scientific_control?banner=B12_EditorNoImage
- Left editor image facts banner with "Please read a personal appeal from a Wikipedia Editor" tab: http://en.wikipedia.org/wiki/Horseshoe_Curve_(Pennsylvania)?banner=B12_EditorImage
Results: ecom -s 20121113164200 -e 20121113171300 -c C12_iwpenUS_FR -g b
+-------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +-------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | B12_EditorImage | 105 | 146 | 46% | 63% | 81% | 2151.00 | 1123.00 | 20.485714 | 10.695238 | | B12_EditorNoImage | 201 | 289 | 39% | 54% | 85% | 2837.00 | 1955.00 | 14.114428 | 9.726368 | +-------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_EditorImage=105%2C1000000&B12_EditorNoImage=201%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true
Round 2:
- Control facts banner: http://en.wikipedia.org/wiki/Scientific_control?banner=B12_EditorNoImage
- Right editor image facts banner with "Please read a personal appeal from a Wikipedia Editor" tab: http://en.wikipedia.org/wiki/Rights?banner=B12_EditorImageRight
Results:
ecom -s 20121113171300 -e 20121113201300 -c C12_iwpenUS_FR -g b
+-----------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +-----------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | B12_EditorImageRight | 137 | 209 | 35% | 47% | 83% | 2398.00 | 1476.00 | 17.503650 | 10.773723 | | B12_EditorNoImage | 259 | 388 | 33% | 48% | 82% | 4047.00 | 2662.00 | 15.625483 | 10.277992 | +-----------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_EditorImageRight=137%2C1000000&B12_EditorNoImage=259%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true
Round 3:
- Control facts banner: http://en.wikipedia.org/wiki/Scientific_control?banner=B12_EditorNoImage2
- Right editor image facts banner with "Support Wikipedia Editors" tab: http://en.wikipedia.org/wiki/Image?banner=B12_EditorImageRightSupport
Results:
ecom -s 20121113172800 -e 20121113202800 -c C12_iwpenUS_FR -g b +-----------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +-----------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | B12_EditorImageRightSupport | 129 | 218 | 40% | 46% | 36% | 2010.00 | 1329.00 | 15.581395 | 10.302326 | | B12_EditorNoImage2 | 299 | 419 | 35% | 54% | 86% | 4402.00 | 3017.00 | 14.722408 | 10.090301 | +-----------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+
Round 4:
- Control facts banner: http://en.wikipedia.org/wiki/Scientific_control?banner=B12_EditorNoImage_SameTab
- Right editor image facts banner with "Please help" tab and "Support our editors" under the editor image: http://en.wikipedia.org/wiki/Support?banner=B12_EditorImage_SameTab
Results:
ecom -s 20121113175500 -e 20121113205500 -c C12_iwpenUS_FR -g b +-----------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +-----------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+ | B12_EditorImage_SameTab | 435 | 697 | 42% | 49% | 77% | 6220.95 | 4341.00 | 14.301034 | 9.979310 | | B12_EditorNoImage_SameTab | 690 | 1050 | 41% | 53% | 80% | 9600.72 | 6639.72 | 13.914087 | 9.622783 | +-----------------------------+-----------+--------+-------+-------+-------+---------+----------+-----------+-----------+
What this means: The upside: We were able to improve the performance of the editor banner by running through different versions in testing this morning. The downside: our control facts still beat the editor banner overall. More upside: we learned a lot in just a couple hours testing this morning.
US Tuesday Nov. 13, 2012, Gov/$5 vs Gov/Sandwich edit
A couple months ago when we were using the "google line" in the banner, we that for the third fact, using the "If everyone gave the price of a sandwich" worked better than the "If everyone gave $5.." line.
We later found the "no government funds" banner that beat "Google line" banner. Now we're testing using the sandwich fact with the "no government funds" banner.
Banners:
- "If everyone $5": http://en.wikipedia.org/wiki/Five?banner=B12_FpcwDropIevryUS_A
- "If everyone gave the price of a sandwich" http://en.wikipedia.org/wiki/Sandwich?banner=B12_FpcwDropGovSand_A
Results
ecom -s 20121113224600 -e 20121114020000 -c C12_iwpenUS_FR -g b +---------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | avg | avg18 | +---------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+ | B12_FpcwDropGovSand_A_US | 1346 | 2075 | 29% | 50% | 80% | 19960.67 | 13045.10 | 14.829621 | 9.691753 | | B12_FpcwDropIevryUS_A_US | 1345 | 2096 | 30% | 51% | 78% | 18875.10 | 12901.10 | 14.033532 | 9.591896 | +---------------------------+-----------+--------+-------+-------+-------+----------+----------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_FpcwDropGovSand_A_US=1346%2C1000000&B12_FpcwDropIevryUS_A_US=1345%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true
US Sunday Nov. 18 Ask string vs no string in banner edit
- no ask string in banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_10string1118_string
- ask string in banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_10string1118_nostring
ecom -s 20121118204000 -e 20121118235900 -c C12_iwpenUS_FR -g b -m --having 5
+---------------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | avg | avg18 | +---------------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+-----------+-----------+ | B12_10string1118_nostring | 2794 | 3829 | 33% | 64% | 81% | 47199.29 | 31424.07 | 33056.29 | 41549.29 | 16.893089 | 11.246983 | | B12_10string1118_string | 2878 | 3903 | 34% | 66% | 82% | 47007.22 | 31863.22 | 33531.22 | 41195.22 | 16.333294 | 11.071306 | +---------------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_10string1118_nostring=2794%2C1000000&B12_10string1118_string=2878%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true
US Nov. 19 Super short facts banner edit
- http://en.wikipedia.org/wiki/Main_Page?banner=B12_shortening1119control
- http://en.wikipedia.org/wiki/Main_Page?banner=B12_shortening1119supershort
Nov. 21 $5 & $30 vs no $5 & $30 in the banner US, CA, AU, NZ, GB edit
- Control: http://en.wikipedia.org/wiki/Main_Page?banner=B12_shortening1120cntrl
- No $5 and $30 in the banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_shortening1120no530
US Results ecom -s 20121120225500 -e 20121121060000 -c C12_iwpenUS_FR -g b -m --having 30
+-------------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | avg | avg18 | +-------------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+-----------+-----------+ | B12_shortening1120cntrl | 1239 | 1700 | 34% | 65% | 82% | 18311.82 | 12395.82 | 13119.82 | 16624.82 | 14.779516 | 10.004697 | | B12_shortening1120no530 | 1279 | 1739 | 33% | 67% | 83% | 16975.70 | 11627.70 | 12179.70 | 14847.70 | 13.272635 | 9.091243 | +-------------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_shortening1120cntrl=1239%2C1000000&B12_shortening1120no530=1279%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.000075
Canada Results
ecom -s 20121121000000 -e 20121121060000 -c C12_iwpenCA_FR -g b -m --having 30
+---------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | avg | avg18 | +---------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+----------+ | B12_shortening1120cntrlCA | 487 | 747 | 37% | 51% | 78% | 6575.96 | 4861.51 | 5123.51 | 6170.92 | 13.502998 | 9.982567 | | B12_shortening1120no530CA | 480 | 729 | 37% | 51% | 80% | 6363.67 | 4779.28 | 5051.28 | 6107.96 | 13.257646 | 9.956833 | +---------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_shortening1120cntrlCA=487%2C1000000&B12_shortening1120no530CA=480%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.620773
AU Results ecom -s 20121121013000 -e 20121121053000 -c C12_iwpenAU_FR -g b -m --having 3
+------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | avg | avg18 | +------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+----------+ | B12_shortening1120cntr | 377 | 606 | 31% | 56% | 81% | 5225.45 | 3662.12 | 3836.83 | 4914.38 | 13.860610 | 9.713846 | | B12_shortening1120no53 | 408 | 649 | 35% | 64% | 76% | 5211.51 | 3686.36 | 3848.36 | 4796.50 | 12.773309 | 9.035196 | +------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_shortening1120cntr=377%2C1000000&B12_shortening1120no53=408%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.033164
NZ Results ecom -s 20121121013000 -e 20121121063000 -c C12_iwpenNZ_FR -g b -m
+---------------------------+-----------+--------+-------+-------+-------+--------+----------+----------+----------+-----------+----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | avg | avg18 | +---------------------------+-----------+--------+-------+-------+-------+--------+----------+----------+----------+-----------+----------+ | B12_shortening1120cntrlNZ | 55 | 77 | 73% | 69% | 83% | 727.35 | 494.66 | 514.66 | 663.91 | 13.224545 | 8.993818 | | B12_shortening1120no530NZ | 49 | 66 | 69% | 81% | 65% | 527.16 | 390.00 | 402.00 | 495.44 | 10.758367 | 7.959184 | +---------------------------+-----------+--------+-------+-------+-------+--------+----------+----------+----------+-----------+----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_shortening1120cntrlNZ=55%2C1000000&B12_shortening1120no530NZ=49%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.214758
GB Results
ecom -s 20121121110000 -e 20121121150000 -c C12_iwpenGB_FR -g b -m --having 2
+---------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | avg | avg18 | +---------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+-----------+ | B12_shortening1120cntrlGB | 531 | 718 | 35% | 62% | 83% | 8381.65 | 6213.96 | 6419.15 | 7926.83 | 15.784652 | 11.702373 | | B12_shortening1120no530GB | 560 | 759 | 33% | 62% | 81% | 7852.73 | 6021.75 | 6127.75 | 7111.36 | 14.022732 | 10.753125 | +---------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_shortening1120cntrlGB=531%2C1000000&B12_shortening1120no530GB=560%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.000257
Nov. 21 Canada test "We run on YOUR donations" edit
We tested a new variation of the facts banner addressing the reader as "you" in the banner.
- Control: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1122cntrlCA
- We run on your donations...If you and everyone else gave $5: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1122youCA
ecom -s 20121122013000 -e 20121122653000 -c C12_iwpenCA_FR -g b -m --having 5
+-----------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | avg | avg18 | +-----------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+-----------+ | B12_1122cntrlCA | 310 | 461 | 43% | 58% | 76% | 4836.89 | 3139.22 | 3323.22 | 4357.27 | 15.602871 | 10.126516 | | B12_1122youCA | 298 | 430 | 43% | 60% | 78% | 4549.15 | 3277.48 | 3490.98 | 4296.16 | 15.265604 | 10.998255 | +-----------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1122cntrlCA=310%2C1000000&B12_1122youCA=298%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.120951
Nov. 22 $5 fact in center edit
We tested putting this fact in the center of the banner instead of on the right side of the banner: "If everyone reading this gave $5, the fundraiser would be done within an hour. Please help us forget fundraising and get back to Wikipedia."
Australia:
- Control: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1121cntrlAU
- $5 centered: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1121mid5AU
ecom -s 20121122013000 -e 20121122053000 -c C12_iwpenAU_FR -g b -m
ecom -s 20121122013000 -e 20121122053000 -c C12_iwpenAU_FR -g b -m +--------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | avg | avg18 | +--------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+----------+ | B12_1121cntr | 381 | 613 | 34% | 59% | 78% | 5935.32 | 3798.50 | 4004.50 | 5406.71 | 15.578268 | 9.969816 | | B12_1121mid | 331 | 544 | 31% | 59% | 78% | 4241.59 | 3109.23 | 3255.23 | 4106.84 | 12.814471 | 9.393444 | +--------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1121cntr=381%2C1000000&B12_1121mid=331%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.136039
New Zealand: ecom -s 20121122013000 -e 20121122053000 -c C12_iwpenNZ_FR -g b -m
+-----------------+-----------+--------+-------+-------+-------+--------+----------+----------+----------+-----------+----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | avg | avg18 | +-----------------+-----------+--------+-------+-------+-------+--------+----------+----------+----------+-----------+----------+ | B12_1121cntrlNZ | 57 | 77 | 67% | 69% | 86% | 777.49 | 514.37 | 530.37 | 623.94 | 13.640175 | 9.024035 | | B12_1121mid5NZ | 34 | 52 | 62% | 60% | 81% | 440.44 | 250.17 | 256.17 | 286.89 | 12.954118 | 7.357941 | +-----------------+-----------+--------+-------+-------+-------+--------+----------+----------+----------+-----------+----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1121cntrlNZ=57%2C1000000&B12_1121mid5NZ=34%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.067449
Nov. 22 GB "If Wikipedia is useful to you" edit
When we ask donors why they gave to Wikipedia, they usually say they donate because they use Wikipedia and want to support something they find so useful. We tested a new line incorporating this idea: "If Wikipedia is useful to you, please help keep it online another year."
- Control banner: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1122cntrlGB
- If Wikipedia is useful: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1122usefulGB
ecom -s 20121122110000 -e 20121122160000 -c C12_iwpenGB_FR -g b -m --having 5
+------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | avg | avg18 | +------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+-----------+ | B12_1122cntrlGB | 466 | 660 | 35% | 59% | 79% | 9169.65 | 5752.79 | 5958.79 | 7691.34 | 19.677361 | 12.345043 | | B12_1122usefulGB | 514 | 716 | 33% | 56% | 83% | 9408.20 | 6095.12 | 6309.12 | 8040.05 | 18.303891 | 11.858210 | +------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1122cntrlGB=466%2C1000000&B12_1122usefulGB=514%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.144127
The sample was too small here to see a significant difference, but we'll test this idea again to get more data.
Nov. 24 Top facts vs. dropdown small top bar edit
So this seems to be a HUGE breakthrough. It's a smaller banner -- much smaller. And I think the grey tab finally makes these top banners look slightly normal and not completely jarring. So: smaller, les jarring. And it does way better! But why? It's probably not the blue info icon -- at least not that alone. A short test, that we'll lengthen tomorrow, seemed to rule that out. But maybe all the differences with this banner are just working in the same direction towards more clicks and donations: the info icon, the nicer tab, the wide-column text. And/or maybe the big factor is that the "system alert" look signifies a direct message from the site's (or browser's) owner. Only the people who run the site your on have the power put an alert like this up there -- so maybe that makes it stand out as a non-BS piece of info that is definitely not going to trick you into reading some ad. More tomorrow.
United States:
- Control top tall facts: http://en.wikipedia.org/wiki/November_24?banner=B12_1123_Control2
- Top small facts dropdown bar: http://en.wikipedia.org/wiki/November_24?banner=B12_1123_Twolineinfo
ecom -s 20121123193500 -e 20121123232200 -c C12_iwpenUS_FR -g b -m --having 50
+----------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +----------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | B12_1123_Control2 | 929 | 1221 | 36% | 69% | 83% | 15204.00 | 9732.00 | 10344.00 | 13984.00 | 200.00 | 16.365985 | 11.134553 | | B12_1123_Twolineinfo | 1269 | 1691 | 32% | 66% | 82% | 18910.92 | 12389.92 | 13089.92 | 16885.92 | 200.00 | 14.902222 | 10.315146 | +----------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1123_Control2=929%2C1000000&B12_1123_Twolineinfo=1269%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.002240
United States:
- Top tall facts: http://en.wikipedia.org/wiki/November_24?banner=B12_1123newfactsinfodot_Threecol
- Top small facts dropdown bar: http://en.wikipedia.org/wiki/November_24?banner=B12_1123newfactsinfodot_OnecolSmall
ecom -s 20121123214000 -e 20121123230000 -c C12_iwpenUS_FR -g b -m --having 100
+-------------------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+--------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +-------------------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+--------+-----------+-----------+ | B12_1123newfactsinfodot_OnecolSmall | 602 | 817 | 34% | 68% | 80% | 9173.12 | 6247.12 | 6641.12 | 8523.12 | 200.00 | 15.237741 | 11.031761 | | B12_1123newfactsinfodot_Threecol | 290 | 393 | 30% | 62% | 83% | 5084.20 | 3057.20 | 3249.20 | 4434.20 | 200.00 | 17.531724 | 11.204138 | +-------------------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+--------+-----------+-----------+
Canada:
- Control top tall facts: http://en.wikipedia.org/wiki/November_24?banner=B12_1122cntrlCA
- Top small facts dropdown bar: http://en.wikipedia.org/wiki/November_24?banner=B12_1124_Info_DropCA
ecom -s 20121124010000 -e 20121124040000 -c C12_iwpenCA_FR -g b -m --having 2 +----------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+---------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +----------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+---------+-----------+-----------+ | B12_1122cntrlCA | 328 | 456 | 39% | 59% | 83% | 5177.44 | 3554.99 | 3792.99 | 4771.36 | 250.95 | 15.784878 | 11.563994 | | B12_1124_Info_DropCA | 562 | 845 | 40% | 55% | 77% | 9270.17 | 5744.08 | 6070.64 | 7707.44 | 1003.80 | 16.494964 | 10.801851 | +----------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+---------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1122cntrlCA=328%2C1000000&B12_1124_Info_DropCA=562%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.294505
New Zealand:
- Control top tall facts: http://en.wikipedia.org/wiki/November_24?banner=B12_1121cntrlNZ
- Top small facts dropdown bar: http://en.wikipedia.org/wiki/November_24?banner=B12_1124_Info_DropNZ
ecom -s 20121124010000 -e 20121124040000 -c C12_iwpenNZ_FR -g b -m
+----------------------+-----------+--------+-------+-------+-------+--------+----------+----------+----------+-------+-----------+----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +----------------------+-----------+--------+-------+-------+-------+--------+----------+----------+----------+-------+-----------+----------+ | B12_1121cntrlNZ | 36 | 50 | 75% | 71% | 75% | 388.82 | 340.49 | 346.49 | 388.82 | 40.93 | 10.800556 | 9.624722 | | B12_1124_Info_DropNZ | 64 | 100 | 70% | 64% | 63% | 601.67 | 523.01 | 537.01 | 601.67 | 40.93 | 9.401094 | 8.390781 | +----------------------+-----------+--------+-------+-------+-------+--------+----------+----------+----------+-------+-----------+----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1121cntrlNZ=36%2C1000000&B12_1124_Info_DropNZ=64%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.361465
UK:
- Control: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1122cntrlGB
- Top small facts dropdown bar: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1124_Info_DropGB
ecom -s 20121124110000 -e 20121124150000 -c C12_iwpenGB_FR -g b -m
+----------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +----------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | B12_1122cntrlGB | 446 | 590 | 37% | 63% | 85% | 8732.38 | 5397.94 | 5603.94 | 7303.74 | 399.10 | 19.579327 | 12.564888 | | B12_1124_Info_DropGB | 759 | 1033 | 31% | 61% | 81% | 11936.84 | 8553.42 | 8803.42 | 10881.70 | 159.64 | 15.727062 | 11.598709 | +----------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1122cntrlGB=446%2C1000000&B12_1124_Info_DropGB=759%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.001675
Nov. 24 Blue info dot edit
- Blue dot: http://en.wikipedia.org/wiki/Blue?banner=B12_112418_Control
- No blue dot: http://en.wikipedia.org/wiki/Optical_spectrum?banner=B12_112418_Nodot
ecom -s 20121124180500 -e 20121124200500 -c C12_iwpenUS_FR -g b -m --having 20
+--------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +--------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | B12_112418_Control | 1808 | 2415 | 33% | 65% | 83% | 29264.55 | 18624.54 | 19794.54 | 26404.55 | 250.00 | 16.186145 | 10.948308 | | B12_112418_Nodot | 1605 | 2250 | 33% | 62% | 80% | 25925.40 | 16773.61 | 17823.61 | 23505.40 | 200.00 | 16.152897 | 11.105053 | +--------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_112418_Control=1808%2C1000000&B12_112418_Nodot=1605%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.424329
Nov. 24 shorter text bar CA edit
- Control: http://en.wikipedia.org/wiki/Optical_spectrum?banner=B12_112420_Info_DropCA
- Shorter text: http://en.wikipedia.org/wiki/Optical_spectrum?banner=B12_112420_Info_DropShortCA
ecom -s 20121124200000 -e 20121124220000 -c C12_iwpenCA_FR -g b -m --having 2 http://www.jobquotes.com
+-----------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+--------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +-----------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+--------+-----------+-----------+ | B12_112420_Info_DropCA | 548 | 789 | 39% | 57% | 80% | 8056.07 | 5381.39 | 5689.39 | 7246.76 | 501.90 | 14.700858 | 10.382099 | | B12_112420_Info_DropShortCA | 478 | 703 | 38% | 58% | 76% | 7577.93 | 5078.32 | 5402.32 | 7168.43 | 100.38 | 15.853410 | 11.301925 | +-----------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+--------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_112420_Info_DropCA=548%2C1000000&B12_112420_Info_DropShortCA=478%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.022577
Nov. 25 AU 2 lines vs 3 lines top bar edit
- Control: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1225_Info_DropAU
- Shorter 2 lines: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1225_Info_DropShortAU
ecom -s 20121125000000 -e 20121125030000 -c C12_iwpenAU_FR -g b -m --having 2
+---------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+--------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +---------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+--------+-----------+-----------+ | B12_1225_Info_DropAU | 375 | 563 | 25% | 58% | 82% | 5821.80 | 3993.42 | 4231.42 | 5500.86 | 104.57 | 15.524800 | 11.283787 | | B12_1225_Info_DropShortAU | 336 | 486 | 29% | 72% | 83% | 6357.62 | 3745.91 | 4001.91 | 5543.24 | 209.14 | 18.921488 | 11.910446 | +---------------------------+-----------+--------+-------+-------+-------+---------+----------+----------+----------+--------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1225_Info_DropAU=375%2C1000000&B12_1225_Info_DropShortAU=336%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.267284
Nov. 25 GB "If Wikipedia is useful to you.." edit
- Control: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1125_Info_DropGB
- Useful: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1125_Info_DropUsefulGB
ecom -s 20121125110000 -e 20121125145000 -c C12_iwpenGB_FR -g b -m --having 10
+----------------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +----------------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | B12_1125_Info_DropGB | 711 | 961 | 30% | 61% | 81% | 11421.98 | 8143.38 | 8377.38 | 10298.86 | 320.52 | 16.064669 | 11.782532 | | B12_1125_Info_DropUsefulGB | 826 | 1136 | 31% | 60% | 80% | 13826.33 | 9466.84 | 9782.84 | 12358.33 | 320.52 | 16.738898 | 11.843632 | +----------------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_1125_Info_DropGB=711%2C1000000&B12_1125_Info_DropUsefulGB=826%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.857254
Nov. 25 US "Dear Wikipedia reader" edit
- Control: http://en.wikipedia.org/wiki/Main_Page?banner=B12_112516_Yellowtab_control
- Dear reader: http://en.wikipedia.org/wiki/Main_Page?banner=B12_112516_Yellowtab_dearr
ecom -s 20121125160000 -e 201211252200000 -c C12_iwpenUS_FR -g b -m --having 20
+------------------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +------------------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | B12_112516_Yellowtab_control | 2103 | 2779 | 34% | 70% | 82% | 35499.75 | 21938.91 | 23290.91 | 31099.75 | 500.00 | 16.880528 | 11.075088 | | B12_112516_Yellowtab_dearr | 2422 | 3147 | 34% | 69% | 84% | 40194.49 | 25614.00 | 27248.00 | 36393.00 | 250.00 | 16.595578 | 11.250206 | +------------------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_112516_Yellowtab_control=2103%2C1000000&B12_112516_Yellowtab_dearr=2422%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.610919
Nov. 25 US Blue vs Yellow tab edit
- Yellow tab: http://en.wikipedia.org/wiki/Main_Page?banner=B12_112519_Yellowtab
- Blue tab: http://en.wikipedia.org/wiki/Main_Page?banner=B12_1125BlueTab
ecom -s 20121125193300 -e 201211252200000 -c C12_iwpenUS_FR -g b -m --having 300
+----------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+---------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +----------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+---------+-----------+-----------+ | B12_112519_Yellowtab | 1264 | 1666 | 33% | 71% | 81% | 22203.67 | 13303.67 | 14153.67 | 18953.67 | 1000.00 | 17.566195 | 11.197524 | | B12_1125BlueTab | 1228 | 1616 | 34% | 69% | 84% | 19622.55 | 12854.55 | 13652.55 | 18022.55 | 200.00 | 15.979275 | 11.117712 | +----------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+---------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_112519_Yellowtab=1264%2C1000000&B12_1125BlueTab=1228%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.944970
Nov. 25 US Yellow tab vs Grey button edit
- Yellow tab: http://en.wikipedia.org/wiki/Main_Page?banner=B12_112520_Yellowtab2
- Grey button: http://en.wikipedia.org/wiki/Main_Page?banner=B12_112520_NoTab2
ecom -s 20121125204000 -e 201211252200000 -c C12_iwpenUS_FR -g b -m --having 100
+-----------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +-----------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | B12_112520_NoTab2 | 1121 | 1486 | 33% | 67% | 82% | 19748.67 | 11973.67 | 12763.67 | 17288.67 | 250.00 | 17.617012 | 11.385968 | | B12_112520_Yellowtab2 | 1273 | 1674 | 31% | 68% | 83% | 19771.95 | 13273.50 | 14129.50 | 18181.95 | 250.00 | 15.531775 | 11.099372 | +-----------------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_112520_NoTab2=1121%2C1000000&B12_112520_Yellowtab2=1273%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true Wilcox test of means p= 0.121782
Nov. 26 "If Wikipedia is useful to you" vs "If Wikipedia is useful to you... Please help us forget fundraising and get back to Wikipedia" edit
US, CA, GB, AU, NZ
- Useful: http://en.wikipedia.org/wiki/Main_Page?banner=B12_112605_Control_US
- Useful + Forget fundraising: http://en.wikipedia.org/wiki/Main_Page?banner=B12_112605_Min_US
ecom -s 20121126073000 -e 20131111000000 -g b --sub B12_112605_ --left 14
+----------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | banner | donations | clicks | ccpct | cccnv | ppcnv | amount | amount18 | amount20 | amount50 | max | avg | avg20 | +----------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+ | B12_112605_Con | 4121 | 6120 | 28% | 54% | 79% | 61233.86 | 42116.06 | 44099.06 | 55607.44 | 320.52 | 14.858981 | 10.701058 | | B12_112605_Min | 4879 | 7145 | 29% | 55% | 80% | 73098.62 | 50374.75 | 52748.75 | 66466.80 | 209.14 | 14.982296 | 10.811386 | +----------------+-----------+--------+-------+-------+-------+----------+----------+----------+----------+--------+-----------+-----------+
d/bi confidence test: http://www.thumbtack.com/labs/abba/#B12_112605_Con=4121%2C1000000&B12_112605_Min=4879%2C1000000&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true