Fundraising 2010/Report/Draft Test 2

Banner Test - "Static VS Fader"

For additional documentation on the testing methodology please see the following pages:

• Methodology
• Verbose Test

Banners

"Static Banner". This is a static banner and features the text "If everyone reading this donated $5 our fundraiser would be over today. Please donate to keep Wikipedia free".

"Fader banner". This is a fader banner and features two text strings which toggle by fading into each other for equal intervals. These strings are: "If everyone reading this donated $5 our fundraiser would be over today. Please donate to keep Wikipedia free" and "Only 3 days left to make a tax-deductible contribution to keep Wikipedia free. Please help Wikipedia pay its bills in 2011."

Result

Test Time: 2010-12-29 14:10:00 UTC - 2010-12-29 15:50:00 UTC
Sampling Interval = 2 minutes
Testing Interval = 20 minutes
Total Number of Samples per Class = 50

TEST RESULT INCONCLUSIVE

(*) The rate of donations per banner impression over a fixed time interval.

(**) The rate of amount50 per banner impression over a fixed time interval. Amount50 is the dollar amount raised from donations initiated under a given banner where all donations of more than $50 are recorded as $50 donations. This counters the skewing effect of outlier donations.

Data Analysis

This section analyzes and interprets the results of the tests. It should be noted that the data is analyzed over a period at least as large as the full testing period and that the testing period was chosen based on the period of time where significant hits and donations were observed.

Data Consistency and Cleaning

Impressions broken out in two minute intervals over the test period.

Donations broken out in two minute intervals over the test period.

Donations/Impression broken out in two minute intervals over the test period.

Amount50/Impression broken out in two minute intervals over the test period.

Analyzing the above plots the donation and impression data appear to be quite regular over the interval 2010-12-29 14:10:00 UTC - 2010-12-29 15:50:00 UTC. Therefore, two minute intervals will be used for sampling over this period as a source for the paired t-test to assess confidence in the winner.

Modelling and Hypothesis Testing

"Static Banner" won in each case for donations/impression and amount50/impression with increases of 5.11% and 1.79% respectively. The student's t-test was used to assess confidence over each metric and the confidence in the winner for donations/impression and amount50/impression is at least 75.0% and 90.0% respectively. This is a very slight and non-significant win for the "Static Banner". Both banners effectively performed the same.

Mean and standard deviation over test intervals over Donations / Impression.

Mean and standard deviation over test intervals over Amount50 / Impression

TOTAL DONATIONS "Static banner": 365
TOTAL DONATIONS "Fader banner": 369

TOTAL AMOUNT50* RAISED "Static banner": $5293.97
TOTAL AMOUNT50* RAISED "Fader banner": $5501.81

* AMOUNT50 indicates the total amount raised where all donations greater than $50 are taken to be a donation of $50.

DONATIONS PER IMPRESSION:

Between 75.0% and 90.0% confident about the winner.

Fader VS Static -- 2010-12-29 14:10:00 - 2010-12-29 15:50:00

item 1  = Fading banner
item 2  = Static banner

The winner Static banner had a 5.11% increase.

interval	mean1		mean2		stddev1		stddev2

0		0.00036		0.00032		0.00022		0.00005
1		0.00033		0.00034		0.00017		0.00003
2		0.00030		0.00035		0.00020		0.00016
3		0.00029		0.00030		0.00013		0.00013
4		0.00028		0.00031		0.00017		0.00008


Overall Parameters:

mean1		mean2		stddev1		stddev2
0.00031		0.00033		0.00018		0.00010

AMOUNT50 PER IMPRESSION:

Between 90.0% and 95.0% confident about the winner.

Fader VS Static -- 2010-12-29 14:10:00 - 2010-12-29 15:50:00

item 1  = Fading banner
item 2  = Static banner

The winner Static banner had a 1.79% increase.

interval	mean1		mean2		stddev1		stddev2

0		0.00566		0.00411		0.00243		0.00186
1		0.00501		0.00526		0.00440		0.00029
2		0.00455		0.00533		0.00341		0.00294
3		0.00409		0.00455		0.00227		0.00086
4		0.00395		0.00441		0.00128		0.00154


Overall Parameters:

mean1		mean2		stddev1		stddev2
0.00465		0.00473		0.00295		0.00175

Endnotes

1. Campaign = "20101228JA075"
2. "Static Banner" utm_source = "20101227_JA061_US"
3. "Fader Banner" utm_source = "20101228_JAFader_US"