User:CKoerner (WMF)/Work/A-B Test Results 1

Improving the Wikipedia portal: one search at a time

The Wikipedia portal introduces people to the work of our volunteers and various projects within the Wikimedia movement. It is an entrance to the 280 language Wikipedias and sister projects and the portal accounts for 1.5–2% of our total page views. One of the goals of the Discovery team is to update and modernize the portal experience. After positive results from a recent round of A/B testing, we will soon be making big improvements to the portal site for our visitors.

More prominent search box with new language selection dropdown

A/B testing the portal

The team recently completed a round of A/B testing and discovered some subtle, but effective improvements.

Between January 13th – 20th, 2016, a small percentage of visitors to Wikipedia.org received a variation of the portal page. This variation included a larger, more prominent search box with a new language selection dropdown. Another small percentage of visitors received a slightly different variation - they saw the same variations but had an enhanced experience of type-ahead search results populated with images and metadata (as available). Our testing goal was to measure the clickthrough rate on the two variant pages as compared to the design of the existing portal page.

More prominent search box with type-ahead metadata and image results.

With the first test group, we did not find any tangible improvement, as the clickthrough rate was between −0.9% and 2.8%. However, with the second group, we found that 1.7% to 5.5% more sessions ended in a clickthrough as compared to the control group. That's roughly an increase of 300,000 to 1.3 million additional page views a day from the portal into our projects!

Next Steps

Before this testing cycle we noticed that some of our user visits to the portal page had resulted in some form of inaction and we want to change this behavior. With a few small improvements (such as ensuring non-JavaScript support), the Discovery team recommends that the more prominent search box (with metadata and images in the search results) be deployed to the production Wikipedia.org portal. The current target is by the end of March.

To learn more about this A/B testing, have a look at the report and its findings on Commons. To keep up-to-date with Discovery work, visit Wikimedia Discovery on MediaWiki.org.