Fundraising 2010/Report/Draft Test 3
Landing Page Test: "Where Your Donation Goes" Pie Chart edit
For additional documentation on the testing methodology please see the following pages:
Landing Pages edit
Result edit
Test Time: 2010-12-30 01:00:00 UTC - 2010-12-30 01:50:00 UTC
Sampling Interval = 2 minutes
Testing Interval = 10 minutes
Total Number of Samples per Class = 25
Donations / View*: "No Pie Chart" landing page won by 2.37%.
Amount50 / View**: "No Pie Chart" landing page won by 9.54%.
WINNER: "No Pie Chart" Landing Page
(*) The rate of donations per banner view over a fixed time interval.
(**) The rate of amount50 per banner view over a fixed time interval. Amount50 is the dollar amount raised from donations initiated under a given banner where all donations of more than $50 are recorded as $50 donations. This counters the skewing effect of outlier donations.
Data Analysis edit
This section analyzes and interprets the results of the tests. It should be noted that the data is analyzed over a period at least as large as the full testing period and that the testing period was chosen based on the period of time where significant hits and donations were observed.
Data Consistency and Cleaning edit
Analyzing the above plots the donation and impression data appear to be quite regular over the interval 2010-12-30 01:00:00 UTC - 2010-12-30 01:50:00 UTC. Therefore, two minute intervals will be used for sampling over this period as a source for the paired t-test to assess confidence in the winner.
Modelling and Hypothesis Testing edit
The landing page "No pie chart" won in each case for donations/view and amount50/view with increases of 2.37% and 9.54% respectively. The student's t-test was used to assess confidence over each metric and the confidence in the winner for donations/view and amount50/view were at least 95.0% and 99.0% respectively. Although the the increases are not large they are significant and it is notable that the increase in amount50/view is more than donations/view. There are two potential causes here, (1) the variability of amounts tends to be much greater than donations (since donations are binary and amounts are values subject to a relatively wide range), (2) the winner actually had an affect on engaging higher amount donors. However, since both results are significant (ie. the cumulative variance was small enough) this may point to the latter explanation.
TOTAL DONATIONS "No pie chart": 348 TOTAL DONATIONS "Pie chart": 309 TOTAL AMOUNT50* RAISED "No pie chart": $3873.73 TOTAL AMOUNT50* RAISED "Pie chart": $3362.00 * AMOUNT50 indicates the total amount raised where all donations greater than $50 are taken to be a donation of $50.
DONATIONS PER VIEW: Between 95.0% and 97.5% confident about the winner. Pie chart -- 2010-12-30 01:00:00 - 2010-12-30 01:50:00 item 1 = Pie chart item 2 = No pie chart The winner No pie chart had a 2.37% increase. interval mean1 mean2 stddev1 stddev2 0 0.03043 0.03778 0.00988 0.02216 1 0.03239 0.04426 0.00886 0.00472 2 0.05725 0.04604 0.02748 0.01531 3 0.05320 0.05948 0.02602 0.03636 4 0.05524 0.04636 0.00410 0.00290 Overall Parameters: mean1 mean2 stddev1 stddev2 0.04570 0.04678 0.01803 0.02039
AMOUNT50 PER VIEW: Between 99.0% and 99.5% confident about the winner. Pie chart -- 2010-12-30 01:00:00 - 2010-12-30 01:50:00 item 1 = Pie chart item 2 = No pie chart The winner No pie chart had a 9.54% increase. interval mean1 mean2 stddev1 stddev2 0 0.52351 0.51283 0.24953 0.30449 1 0.47199 0.86516 0.29552 0.18720 2 0.83610 0.66408 0.06229 0.01991 3 0.76068 1.09058 0.16285 0.82047 4 0.89079 0.68263 0.09534 0.08091 Overall Parameters: mean1 mean2 stddev1 stddev2 0.69662 0.76306 0.19447 0.40196
Endnotes edit
- Campaign = "20101229JA083_US"
- "No pie chart" landing page = "WMFJA064"
- "Pie chart" landing page = "WMFJA068"