Wikidata/Development/Testing

There are three different frameworks

Selenium tests (mw:Selenium Framework, old page)
QUnit tests (mw:Manual:JavaScript unit testing)
PHPunit tests (mw:Manual:PHP unit testing)

Also note the page at mw:Requests for comment/Unit testing.

Wikidata Test Server

The Wikidata test server runs all the mentioned tests and takes care of other QA related jobs. Its jobs are documented here.

Browser Testing for Wikidata

Prerequisites

Install Ruby on your local machine
- For Linux, the easiest way is to use RVM, as described on https://rvm.io
```
\curl -L https://get.rvm.io | bash -s stable --ruby
```
  Alternatively you can install Ruby by running
```
sudo apt-get install ruby2.1
```
- For Windows, use the installer from http://rubyinstaller.org/downloads. IMPORTANT: Use the latest 1.9 version, NOT the 2.0 version! 2.0 is not compatible with Cucumber under Windows yet!
- On MacOS Ruby usually is included. You can also use homebrew to easily get Ruby by running
```
brew install ruby
```
- Further helpful information can be found on https://www.ruby-lang.org/en/downloads/

Update RubyGems
- ```
gem update --system
```

Install Bundler
- ```
gem install bundler
```

Install required packages

Checkout the Wikibase repository
```
cd tests/browser
```
```
bundle install
```
Make sure you have the latest version of Firefox installed

Setup test configuration

You can specify your configuration in two ways. You can either set the environment variables yourself or you use a configuration file. If you use a file you need to copy config/config.yml.sample to config/config.yml and adjust config.yml to your needs.

Mandatory settings are:

WIKIDATA_REPO_URL
WIKIDATA_REPO_API
WB_REPO_USERNAME
WB_REPO_PASSWORD
ITEM_NAMESPACE
PROPERTY_NAMESPACE
ITEM_ID_PREFIX
PROPERTY_ID_PREFIX
LANGUAGE
LANGUAGE_CODE
LANGUAGE_NAME
BROWSER

If you want to test using SauceLabs, you have to add the following settings:

SAUCE_ONDEMAND_USERNAME
SAUCE_ONDEMAND_ACCESS_KEY
BROWSER
VERSION
PLATFORM

Running tests

Run all tests:

bundle exec cucumber

You can also just run a specific feature by specifying the feature-file.
e.g. run the label tests:

bundle exec cucumber features/label.feature

You can also just run a specific scenario of a feature by specifying the line-number of that scenario inside the feature-file.
e.g. run test which checks the label UI elements:

bundle exec cucumber features/label.feature:14

You can also run scenarios with a specific tag.
e.g. run tests which are tagged with @ui_only:

bundle exec cucumber features/label.feature --tag @ui_only

Some scenarios need a valid login given in the environment variables WB_REPO_USERNAME and WB_REPO_PASSWORD:

WB_REPO_USERNAME=user
WB_REPO_PASSWORD=password

Using a specific browser

By default your browser tests will use Firefox. If you want them to run in a different browser you can specify that by setting an environment variable.

Linux/Unix:

export BROWSER=chrome

Windows:

set BROWSER=chrome

For browsers other than Firefox you would need to install an appropriate driver, e.g. chromedriver for Chrome. See the following links for details:

Run the tests in the cloud

For testing different OS/Browser combinations it makes sense to run the tests on an external service. Saucelabs offers this possibility and you'll just need a few commands to run your tests in their cloud.

You'll have to set your Saucelabs username and accesskey:

Linux/Unix:

export SAUCE_ONDEMAND_USERNAME=myusername
export SAUCE_ONDEMAND_ACCESS_KEY=myaccesskey

Windows:

set SAUCE_ONDEMAND_USERNAME=myusername
set SAUCE_ONDEMAND_ACCESS_KEY=myaccesskey

Make sure you point to a public accessable Wikibase instance in your config/config.yml! Do NOT use the live site for testing!
Now you can start your tests as described in the examples above and watch them running in your Saucelabs account.

Run tests in parallel

You can run features in parallel with the parallel_test gem. If you run bundle update the gem will be already installed.

You can then run your features in parallel by doing:

bundle exec parallel_cucumber features/

By default it will run as many features in parallel as your machine has CPUs. You can specify the number with -n. This will run e.g. 3 features in parallel:

bundle exec parallel_cucumber features/ -n 3

If you want to pass options to underlying cucumber, you can pass them as string with -o. This will run the tests that have the @smoke tag and will skip tests that have the @skip tag:

bundle exec parallel_cucumber features/ -o '--tags @smoke --tags ~@skip'

Run tests headless

If you're on a Linux machine you can run your tests on a headless browser. That makes them slightly faster and you wont have annoying browser windows popping up.

The only prerequisite is that you have Xvfb installed:

apt-get install xvfb

If you want to run your tests headless, all you have to do, is setting the appropriate environment variable:

export HEADLESS=true

And then run your tests as usual.

QUnit tests

Before being able to run QUnit tests, they have to be enabled in the LocalSettings.php by adding:

$wgEnableJavaScriptTest = true;

As of now, Wikibase specific QUnit tests only exist for the repo extension. To run all QUnit tests simply point your browser to the following URL:

http://localhost/<path to MediaWiki installation>/index.php/Special:JavaScriptTest/qunit

You might want to run the Wikibase tests exclusively which can be achieved by applying a filter:

http://localhost/<path to MediaWiki installation>/index.php/Special:JavaScriptTest/qunit?filter=wikibase

PHPunit tests

Installing on Ubuntu

Install PEAR if its not already installed, that makes everything a lot easier. On an Ubuntu box this is done with a

sudo apt-get install php-pear

or you could do a

wget http://pear.php.net/go-pear.phar
php go-pear.phar

After its installed do a

sudo pear upgrade pear

Then your at mw:Manual:PHP unit testing/Installing PHPUnit#Using PEAR. Only difference is that you probably should use --alldeps to save some griefs

sudo pear install --alldeps phpunit/PHPUnit

To be able to run tests I had to set absolute paths in LocalSettings.php, this could perhaps be better solved in suite.xml

Running tests

MediaWiki provides a custom wrapper for the standard phpunit command, which is located in tests/phpunit/phpunit.php. It supports all command line options and parameters the original phpunit command does (plus a few arcane ones we don't need).

To run all tests, including tests for any extension you have configured in LocalSettings.php:

 cd tests/phpunit
 php phpunit.php

To run only tests for the Wikibase group (provided you have the Wikibase extensions configured in your LocalSettings.php):

 php phpunit.php --group Wikibase

To run one specific test class:

 php phpunit.php ../../extensions/Wikibase/repo/tests/phpunit/includes/api/ApiGetItemId.php

To run one specific test function:

 php phpunit.php --filter testMyThing

(note that ---filter actually takes a regular expression).

Advanced Options and Configuration

In order to catch more problems when running tests, enable strict error reporting in your LocalSettings.php:

 error_reporting( E_ALL | E_STRICT );
 ini_set( 'display_errors', 1 );
 
 $wgShowExceptionDetails = true;
 $wgShowSQLErrors = true;
 $wgDebugDumpSql  = true;
 $wgShowDBErrorBacktrace = true;

There are several things you may want to change in test/phpunit/suite.xml:

Turning off verbose output, removing all the annoying details about skipped and incomplete tests:

 verbose="false"

Disabling test timeouts (and other strict checks):

 strict="false"

Or increasing the timeout:

 timeoutForSmallTests="8"

Note: make sure never to check in your modified suite.xml! you could also place your modified suit.xml in a different location and tell phpunit where to find it:

 php phpunit.php --configuration /path/to/phpunit/suite.xml

For testing different setups (e.g. different database engines), you can tell phpunit to load a different LocalSettings.php file:

 php phpunit.php --conf /path/to/server/LocalSettings.php

Writing PHPUnit Tests

Test case classes should use the following code skeleton:

/**
 * ...
 *
 * @ingroup Wikibase
 * @ingroup Test
 *
 * @group Wikibase
 * @group Stuff
 *
 * ...
 */
class MyStuffTest extends MediaWikiTestCase { 
    function testFoo() {
        ...
    }
}

The important bits are:

extend MediaWikiTestCase
use a class name (and matching file name) that ends in "Test".
use method names that start with "test" for actual test functions. You can have other (helper) functions that don't use that prefix.
use @ingroup in the class comment to indicate to doxygen into which group documentation about this class should go. "Wikibase" and "Test" should be there.
use @group in the class comment to indicate to phpunit to which test group this test belongs. "Wikibase" should be there.

There are some special groups that trigger special behavior if you apply them:

@group Database causes phpunit to set up temporary database tables for use by the test, so that modifications performed by the test are not visible in the actual wiki database. This must be done for all tests that need the database, because Jenkins will run tests without @group Database without a valid database connection.
@group medium causes phpunit to consider the test to be "medium heavy" instead of the default "small". This will apply a greater timeout to the test when phpunit runs in strict mode.

Registering Extension Test Cases

Update the registerUnitTests method in the Wikibase.hooks.php file (resp. WikibaseLib.hooks.php or WikibaseClient.hooks.php) with file paths to the individual test files. This is all that is necessary to get it up and running. To test this from the command line use

php phpunit.php --group Wikibase

When everything is in place on a central test server it should be possible to run tests for the Wikidata extension just like any other tests.

File structure

Test cases should be placed in the tests/phpunit directory under the extension's directory. Below that, follow the directory structure used by the php files under test: if the file to test is in includes/api, put the test case in tests/phpunit/includes/api, and so on.

Resources

Michael Hunter's You Are Not Done Yet (pdf), a comprehensive checklist of what can and should be checked. It seems to be aimed at Windows based desktop applications, but many points still apply to web applications.