Research:Account creation UX/CAPTCHA

A short experiment associated with account creation improvements.

Background

The purpose of the CAPTCHAs in the MediaWiki registration process is to hamper the creation of accounts by illegitimate bots, typically spammers.

During our user testing of the account creation experience, we observed that users of both the control and test interfaces faced difficulty with the CAPTCHA. Several users failed to pass the CAPTCHA on the first try, and expressed their frustration. This is no surprise.

We also observed that even slight changes in the font rendering of CAPTCHAs can have large impacts on the ability of humans to complete them. As part of a recent work on SwiftMedia, the images were regenerated across the cluster and inadvertently caused a temporary spike in requests for account creation (i.e. more users could not read the CAPTCHA, and asked that an account be created by a trusted volunteer for them).

While the purpose of CAPTCHAs is to prevent spam, dedicated spammers easily use a variety of methods to index our CAPTCHA images, either automatically or with human assistance, making them only a first step in our spam defense ecosystem. In short: CAPTCHAs during account creation act as one of the primary barriers to users during registration, but largely fail to effectively prevent spam on Wikimedia projects. This short experiment aims to explore how effective our current CAPTCHAs really are, and what impact they have on users.

Further reading

• Requests for comment/CAPTCHA
• Wiki spam
• Manual:Combating spam
• Anti-spam features

Research questions

• What is the impact on total conversion rates during account creation if CAPTCHAs are not present? Of the accounts created that are believed to be humans, how does that compare with past creation rates for apparent humans in an equivalent time period?
• What is the impact on the rate of illegitimate bots, especially spammers, that have to be mass reverted and blocked?

Metrics

Account creation click-through rates

Number of users clicking on the submit button per landing page impression

Account creation conversions

Number of accounts successfully created for a given treatment per landing page impression

User blocks

Proportion of new accounts that go on to be blocked

Experiment one

For our first test, we will be removing the CAPTCHA from account creation on English Wikipedia for a maximum of two hours (1-3pm Pacific Time). This is largely driven by A). the volume of account creations and the largely known rate per hour B). performing this first experiment within the bounds of the team's deployment window gives us complete control over the configuration changes necessary to start or stop the test quickly.

Rates of account creation in early January have risen the farther from New Year holiday we get. Starting at little over 90 per hour for the 1/1/2013, and having grown to nearly 300 per hour by 1/8/2013. Our hypothesis for this short test is that a significant proportion of bots are unlikely to discover the lack of a CAPTCHA that quickly, and as such this first experiment will be a better measure of the first research question above, concerning the human impact of the CAPTCHA.