Banner Test - Jimmy VS SueEdit
For additional documentation on the testing methodology please see the following pages:
Test Time: 2010-12-13 15:05:00 UTC - 2010-12-13 16:53:00 UTC
Sampling Interval = 6 minutes
Testing Interval = 108 minutes
Total Number of Samples per Class = 18
Donations / Impression*: "Jimmy" banner won by 65.69%.
Amount50 / Impression**: "Jimmy" banner won by 58.77%.
(*) The rate of donations per banner impression over a fixed time interval.
(**) The rate of amount50 per banner impression over a fixed time interval. Amount50 is the dollar amount raised from donations initiated under a given banner where all donations of more than $50 are recorded as $50 donations. This counters the skewing effect of outlier donations.
This section analyzes and interprets the results of the tests.
Data Consistency and CleaningEdit
The plots below display the counts of the data sources over the testing period as verification of the consistency of the donation pipeline data used in testing. It should be noted that a certain amount of natural variance is expected in donation counts and is no cause for alarm. The bottom two figures depict the total views and donations with those coming from each banner. This illustrates when the campaign becomes active and is also a useful tool for determining where anomalous behaviour may be visible in the data. It is of note that there are a couple of intervals where the impressions dipped down to zero, however these were sparse enough not to effect the quality of the data significantly. It should be noted that the data is analyzed over a period at least as large as the full testing period and that the testing period was chosen based on the period of time where significant hits and donations were observed.
It should be noted that for the two line plots showing view and donation totals an additional banner was running along with the two being compared and so the totals in those plots may show an overestimate of the impressions generated from the banners being tested.
Analyzing the above plots the donation and impression data appear to be quite regular over the interval 2010-12-13 15:05:00 UTC - 2010-12-13 16:53:00 UTC. Therefore, two minute intervals will be used for sampling over this period as a source for the paired t-test to assess confidence in the winner.
Modelling and Hypothesis TestingEdit
"Jimmy" won in each case for donations/impression and amount50/impression with increases of 65.69% and 58.77% respectively. The student's t-test was used to assess confidence over each metric and the confidence in the winner for donations/impression and amount50/impression is at least 99.0% and 97.5% respectively. This is a significant win for the "Jimmy" banner.
NOTE: These banners were not served to readers in equal amounts. The values in brackets show estimates of the counts if the banners were served at equal amounts. TOTAL SHARE OF IMPRESSIONS "Jimmy": 67.9% TOTAL SHARE OF IMPRESSIONS "Sue": 32.1% TOTAL DONATIONS "Jimmy": 261 (192.19) TOTAL DONATIONS "Sue": 72 (112.15) TOTAL AMOUNT50* RAISED "Jimmy": $6863.33 ($5053.99) TOTAL AMOUNT50* RAISED "Sue": $1945.00 ($3029.56) * AMOUNT50 indicates the total amount raised where all donations greater than $50 are taken to be a donation of $50.
DONATIONS PER IMPRESSION: Between 99.0% and 99.5% confident about the winner. Sue VS Jimmy -- 2010-12-13 15:05:00 - 2010-12-13 16:53:00 item 1 = Sue item 2 = Jimmy The winner Jimmy had a 65.69% increase. interval mean1 mean2 stddev1 stddev2 0 0.00008 0.00009 0.00003 0.00000 1 0.00004 0.00010 0.00002 0.00002 2 0.00006 0.00010 0.00010 0.00002 Overall Parameters: mean1 mean2 stddev1 stddev2 0.00006 0.00010 0.00006 0.00002
AMOUNT50 PER IMPRESSION: Between 97.5% and 99.0% confident about the winner. Sue VS Jimmy -- 2010-12-13 15:05:00 - 2010-12-13 16:53:00 item 1 = Sue item 2 = Jimmy The winner Jimmy had a 58.77% increase. interval mean1 mean2 stddev1 stddev2 0 0.00216 0.00275 0.00007 0.00052 1 0.00128 0.00250 0.00045 0.00036 2 0.00145 0.00251 0.00286 0.00025 Overall Parameters: mean1 mean2 stddev1 stddev2 0.00163 0.00259 0.00167 0.00039
- Campaign = "20101213JA009"
- "Jimmy" utm_source = "20101213_JA012A_US"
- "Sue" utm_source = "20101210_SA004A_US"