Multimedia usability project report

Executive summary

The Multimedia Usability Project was a special project of the Wikimedia Foundation realized thanks to a $300,000 grant from the Ford Foundation. It ran from October 2009 to November 2010; its core team was composed of a Product Manager and a Software Developer.

The project's goal was to increase multimedia participation on Wikimedia websites. The main means chosen to reach this goal was to facilitate the upload process to Wikimedia Commons, the central media repository for Wikimedia sites.

Two main products were delivered as part of the project, both based on extensive user research: a new multi-file upload system for Wikimedia Commons, featuring a wizard-style interface and a temporary holding area; and an illustrated licensing tutorial, explaining the basics of copyright and free licenses in plain language.

A usability study was conducted by an independent firm, who compared the existing and new upload systems. Their results showed an indisputable improvement of the users' experience. Long-term impact of the project is still to be assessed.

The Wikimedia Foundation indicated its intention to continue to improve the upload system beyond this project, and to support volunteers worldwide who share multimedia files on Wikimedia Commons.

2016 update

As of March 2016, over 8.6 million files have been uploaded with UploadWizard; this represents over a quarter of all files ever uploaded to Wikimedia Commons. This notably includes many pictures uploaded as part of massive international contests like Wiki Loves Monuments.

Introduction

Free works & Wikimedia websites

Wikipedia, the "free encyclopedia that anyone can edit", is supported by the Wikimedia Foundation, a San Francisco-based nonprofit dedicated to empowering "every single human being" to "freely share in the sum of all knowledge". Although the English Wikipedia is the most famous website hosted by the Wikimedia Foundation, it is only one of more than 700 wikis.

They all share some common characteristics: first, almost all of their content is in the public domain or under a "free license", which means the author explicitly decided to share their work and allow anyone to use it for any purpose, even for commercial use. On a case-by-case basis, some wikis have also chosen to host limited unfree content that falls under "Exemption Doctrine Policies" (EDP) such as "fair use", in accord with the official Licensing Policy of the Wikimedia Foundation.

All their content is entirely created, improved and curated by volunteer participants. All these wikis are also based on the same wiki platform called "MediaWiki", a free and open source software written in PHP and mainly maintained by volunteer developers.

Wikimedia Commons

Wikimedia Commons is the central media repository for the more than 700 wikis supported by the Wikimedia Foundation, including Wikipedia. It was created in 2004 and now contains over 7 million media files, i.e. about twice as much as the total number of articles on the English Wikipedia, created in 2001. All content on Wikimedia Commons has either been explicitly released under a free license by the copyright holder, or fallen into the public domain; there is no "fair use" on Commons. Any media file uploaded to Wikimedia Commons can then be embedded into any Wikipedia article in any language, or any other Wikimedia website. It is even possible, for recent websites using MediaWiki outside Wikimedia, to embed media files from Commons transparently.

As a Wikimedia website, Commons is a collaborative platform based on MediaWiki, which provides an interface localized in more than 300 languages. Then, it serves an educational purpose; it is not a social networking website where users share their personal pictures with their friends. Also, Commons contains only freely reusable works; copyright violations and content with an unknown copyright status are speedily and proactively deleted. In this regard, Wikimedia Commons is different from popular media sharing platforms such as Flickr or Facebook.

However, even amongst Wikimedia websites, Commons has some characteristics of its own that make it unique. First, it is a multimedia collection, whereas all other Wikimedia websites are primarily based on textual content. Then, it is a multilingual central media repository for more than 700 other wikis. This has implications in terms of communications between participants, who have different cultural backgrounds and come to discuss in languages they are not necessarily proficient in. The other consequence is that Commons is often seen by participants only as a way to add a media file to another Wikimedia website, such as Wikipedia.

Over the years, uploading a media file to Wikipedia or Wikimedia Commons has grown increasingly difficult. Users are presented with a verbose and complicated upload form, as a consequence of both implementation-driven design and limited capabilities of the software, which was developed for textual content. Several attempts were made over the years to make the upload process easier without changing the software. However, these changes could not replace dedicated improvements of the software to address the specific challenges of Wikimedia Commons.

In July 2009, the Ford Foundation awarded a $300,000 grant to the Wikimedia Foundation to facilitate multimedia participation on Wikimedia websites, especially Wikimedia Commons, for people around the world. The subsequent "Multimedia Usability Project" heavily relied on User research to better understand the goals and needs of participants, and to build on MediaWiki to improve the User experience when contributing multimedia content.

Project summary

Project scope and main accomplishments

The grant proposal referenced two main deliverables:

• "Integrated upload tool: Through a pop­-up tool, users will be enabled to upload media directly from the editing interface. This upload tool will intelligently distinguish between Wikimedia Commons and a local project, depending on the licensing status of the file being uploaded. The user interface will be minimal, and more advanced options will be accessible for use cases that deviate from default assumptions."
• "Help page / tutorial system. A key problem with the current uploading process is that instructions, warnings and explanations for many fringe cases have been mixed with the actual uploading interface. We will develop a context­-aware help system that can optionally be accessed to explain difficult concepts such as fair use, free content licensing, and so forth. This help system may even include an interactive quiz that could potentially be used as a precondition for participation."

Two main products were developed as part of this project, that reflect these two main deliverables: the Upload Wizard and the Licensing tutorial.

Upload wizard

The upload wizard is a step-by-step wizard allowing users to upload multiple media files to Wikimedia Commons. Its design was based on heavy user research, in order to improve the uploading experience for new, occasional and experienced users. It focuses on the most common cases, while still allowing experienced users to customize their interface with shortcuts. Similarly, the process was streamlined to focus on the most important pieces of information, while allowing experienced or curious users to provide additional information if they wish so. The upload wizard also contains a new built-in, context-aware help system to provide access to relevant information without overwhelming the user.

The upload wizard is not yet accessible from the edit window, but it was built in a way that would facilitate its future integration in the editing workflow on projects like Wikipedia, as opposed to requiring users to visit a separate website. The current separation actually makes sense while the projects maintain separate infrastructure for purposes such as user-to-user messaging and change notification. Until we have improved integration of this infrastructure, it would be too risky to simply transparently send uploads to Wikimedia Commons, as users would not see these notifications, and the normal interaction with uploaders on Wikimedia Commons could break down. For the same reasons, it is not yet possible to implement an intelligent system that automatically directs media files to different wikis depending on the files' copyright status.

Licensing tutorial

The second main product created as part of this project is the Licensing tutorial. It complements the built-in help system of the upload wizard. The licensing tutorial is an educational one-page comic strip presented to users who upload a media file for the first time. It explains the basics of copyright and free licenses, to avoid mistakes from new users who are not familiar with free licenses. The format of a comic strip was chosen in order to be more enticing than a "wall of text" that has been proven to put users off.

In the beta version of the upload wizard, the tutorial is always shown. It will ultimately be less obtrusive and its integration will be refined to be both more subtle and more accessible.

Project resources

Core team

Chronologically:

• Naoko Komura as Program Manager. Originally hired as Project Manager for the Wikipedia Usability Initiative, the scope of Naoko Komura's responsibilities was expanded to include the Multimedia usability project. In March 2010, she was promoted to the position of Head of User Experience programs but decided to leave the Wikimedia Foundation in June 2010.
• Guillaume Paumier was hired in October 2009 as Product Manager and Interaction Designer. Guillaume was responsible for user research, product requirements, design and communications.
• Neil Kandalgaonkar was hired in December 2009 as Software Developer. Neil was responsible for all software development, integration and deployment activities.
• Alolita Sharma was hired in July 2010 as Features Engineering Programs Manager. Alolita took over the Project oversight in July 2010 after Naoko Komura left.

Extended team

Chronologically:

• Christopher Nash and Kelly Goto, from gotomedia, conducted the June 2010 User experience study.
• Artist Michael Bartalos created the Licensing tutorial. Jay Walsh, Head of Communications for the Wikimedia Foundation, and David Peters, Design director for exbrook, were also involved in the Licensing tutorial project in an advisory capacity.

Project spending

See page 20 of the grant proposal for the planned budget.

User research

Research plan

A common approach would have been to conduct a User experience (UX) study from the start and work from there. However, a preliminary analysis showed that much material was already available to identify critical issues with the current upload process and workflow: the large community of users is usually happy to provide information and feedback on a continuous basis. Besides, our resources limited the number of UX studies we were able to conduct during the project. As a consequence, we chose to focus first on an extensive preliminary research, combining both qualitative and quantitative data. We chose to wait until we had a first prototype before conducting a formal UX study. Our research plan therefore consisted of three phases:

• First, a preliminary user research phase in the early stages of the project (October-December 2009), consisting of an inventory of the existing documentation, an online user survey, an open forum, ethnographic interviews (as described by Cooper et al.) and discussions with users, stakeholders & subject matter experts. The major goals of this phase were to observe and understand why and how users used the product, to identify or confirm critical issues that they encountered during their use and to collect input from various knowledgeable references.
• Then, a formal UX & usability study six months after the beginning of the project (March 2010), consisting of a "classic" study conducted with UX specialists. Participants would be presented with two interfaces: the initial interface as it was before any improvement, and a prototype developed using the preliminary research phase. The main goals of this phase were to formally record the original user experience, to validate the issues identified by the preliminary study, to uncover new issues and to validate the first design decisions.
• Last, a final validation study at the end of the project (October 2010), to formally assess the progress made and to identify remaining or new issues.

The final validation study was scheduled for November 2010, i.e. at the end of the project. However, it appeared that the visible features of he upload wizard wouldn't be very different from the version tested during the previous study in June. Testing a really similar interface again would have been a waste of resources, since it wouldn't have provided us with much additional information compared to the previous study.As a consequence, we decided to postpone the validation study until the end of the next phase of development.

Online user survey

This section is a summary of the survey results; only the results with an impact on the design decisions are presented. For more details, see the full results.

The results of the online user survey showed a lack of awareness about Commons, even among Wikimedia participants. We also identified several types of users according to behavioral patterns, goals and habits, such as "Wikipedians", "Commoners" and "Gleaners".

Method

The main goal of the survey was to get a better understanding of why the respondents used Wikimedia Commons (or why they did not), what activities they engaged in and why. An underlying goal was to see how users reflected upon their own use of Wikimedia Commons, and to compare this subjective self-reflection to objective data. Last, we wanted to identify behavioral patterns based on correlations between goals, activities and levels of activity.

The survey ran during three days and was linked from all Wikimedia websites for all logged-in users. It was available in 20 languages in order to address the multilingual nature of Wikimedia websites and particularly Commons. During this time, 25,150 complete responses to the questionnaire were recorded. We used Pearson's χ² tests to evaluate the independence or correlation of some factors, using a threshold of 0.01%.

Users

Reasons for looking up files in Commons (or not).

First, we asked the users if they used Wikimedia Commons, and for what purpose; if they used it infrequently or not at all, we asked why not. Almost 2/3 (62%) of the respondents declared they used Commons (users), while the others declared they did not (non-users). More than half the users identified their main goal when searching files on Commons as to illustrate an article on Wikipedia or another Wikimedia project; the second goal was to use media files offline. We also investigated the reasons why some respondents did not use Commons; given that the target audience of the survey was logged-in Wikimedia users, we expected them to be aware of Commons, even if they did not participate in it. On the contrary, it appeared that 70% of the respondents who did not use Commons simply did not know about it until the survey. Other reasons included issues due to the search feature or the predominance of English as the lingua franca.

Motivations to participate

Main goals of participants.

The users were then asked if they also participated in the activities of Wikimedia Commons. More than half the users (60% of users, 37% of respondents) declared they did. Baytiyeh & Pfaffman recently studied the motivations of "administrators" volunteering their time on Wikipedia. They found that the main reason was a desire to learn and "an altruistic desire to create a resource for others to use". Our survey was broader and was not intended to be as thorough as Baytiyeh's. However, the fundamental difference between Wikipedia and Wikimedia Commons raised this question: were the motivations shared universally amongst Wikimedia projects? We asked the participants to identify the main reason why they participated in Wikimedia Commons. 65% of respondents declared their main motivation was to illustrate Wikipedia or another Wikimedia projects. The altruistic desire to share with the world was consciously identified as main reason to contribute by only 11% of respondents; this proportion reaches 28% when consolidated with similar, but less explicit answers. Overall, contributing to Wikimedia Commons is widely considered by participants as a means to illustrate Wikipedia (or other Wikimedia projects), rather than a goal in itself.

Activities & levels of participation

Distribution of respondents by amount of files uploaded to Commons.

We asked the participants how many files they had uploaded to Wikimedia Commons, how many edits they had made there and the frequency at which they performed certain tasks. We were surprised to find that half the participants declared they had uploaded less than 10 files to Commons: this means that many Wikimedia participants upload very few media files to Commons, or even none at all. Another third occasional participants uploaded between 10 and 100 files.

Own works

Proportion of own works among the respondents' uploads.

From a workflow point of view, it makes a significant difference whether users are uploading works they created themselves ("own works") or works created by other people (either relatives or strangers). If they created the work themselves and they want to share it on a Wikimedia website, they only have to choose a free license compatible with the Wikimedia licensing policy. Uploading someone else's work requires a completely different workflow, involving the possibility to check with third-party copyright holders that they actually gave permission to use their work under the specified free license.

In order to gain a better understanding of these two cases, we asked the users to evaluate the proportion of own works amongst the total number of files they had uploaded. Based on preliminary analysis, we originally defined four slices: less than 10%, from 10% to 50%, from 50% to 90% and more than 90% own works*. It appeared the answers were generally very similar for users in the two central groups (between 10% and 90%). The two extreme groups, however, are most significant; 39% of the participants declared they had uploaded more than 90% of own works, thus constituting a group of "Creators" (photographers, illustrators, mapmakers, etc.). 32% of the participants uploaded less than 10% of own works.

A correlation can be established between the ratio of own works and the frequency of uploads. Participants who uploaded less than 10% of own works are also the ones who rarely upload media files at all. Similarly, there is a correlation between participants who uploaded between 10 and 90% of own works and those who upload new files on an occasional or a regular basis. Last, participants who uploaded more than 90% of own works are correlated to the participants who upload new files very often.

What this means in terms of design is that the interface should be optimized for the most common case, i.e. for people who share their own works, while still allowing some flexibility for the other users.

Multimedia developers and users workshop

In November 2009, Wikimedia France and the Wikimedia Foundation organized a multimedia-focused workshop in Paris, France. One of the key objectives of the meeting was to help inform the Multimedia Usability Project team to improve the usability of Wikimedia Commons.

During the meeting, expert users and developers worked together to share understanding, produce documentation and develop or deploy new multimedia tools. The main accomplishments of the workshop included:

• an increased awareness of our shared activities through demonstrations and discussions
• the experimental roll-out of a feature to track usage of media from Wikimedia Commons across other Wikimedia projects; a first implementation of wiki-editable subtitles for videos, and smaller hacks and improvements.
• draft ideas and concepts for improving the user experience on Wikimedia Commons as a whole
• a clearer articulation of the needs that are specific to working with cultural institutions (galleries, libraries, archives, and museums – "GLAM")

Open forum

At the end of the survey, respondents were offered the possibility to leave free comments on a dedicated page on Wikimedia Commons. The page, called Commons:Usability issues and ideas, was also advertised in other places. The goal was to provide a central and persistent page where users could report issues in a familiar setting, rather than on an unintuitive bug tracker. The page was also used to collect feedback on the Upload wizard once the prototype was released. The issues were then summarized and consolidated on another page.

Analysis of bugs and feature requests

We analyzed several hundreds bugs and feature requests related to Multimedia already registered in MediaWiki's and Wikimedia's bug tracker. A much needed cleanup was done; the remaining items were listed on the usability wiki and organized in nine categories, depending on what they affected: Uploading, Maintenance, Editing, Using and reusing content, Crosswiki integration, Multilingual support, Navigation, Metadata and Back-end. They helped us identify the areas that needed improvement, and guided design decisions.

Ethnographic interviews

"Ethnographic interviews" (as described by Cooper et al.) consisted of field observations and contextual inquiries performed in the user's usual environment, in order to cause little disruption and to observe the user as they usually interact with Commons and the upload process. The original goal was to perform a number of interviews, model user personae and use them to design the products. However, due to time and financial constraints, it was not possible to fully follow this method. Instead, we had to create provisional personae, with the hope to revisit them later.

Four users were interviewed, with different background activities. Notes of the sessions were posted publicly on the usability wiki.

Upload wizard

Design

Set of wireframes describing the upload wizard

Workflow view of the upload wizard

The upload wizard was designed based on this extensive preliminary research phase. It aims to replace the default upload form present in MediaWiki.

Our research highlighted the following issues:

• In an attempt to prevent all mistakes, long walls of text were presented to the user, mixing:
  • rules about what was allowed or not;
  • detailed instructions on how to fill the form;
  • education on & advocacy for free licenses;
  • various pieces of advice and recommendations.
• Everything was on one long page.
• The layout didn't emphasize important fields (like "License") over seldom-used or optional ones (like edit tools, or "additional information")
• It displayed wikitext.

In contrast, the design of the upload wizard was guided by the following principles and decisions:

• Using a wizard format, with several steps, to emphasize specific moments and let the user focus on them;
• Decoupling the educational part about free licenses from the actual upload form, by presenting a separate licensing tutorial;
• Encouraging the user to naturally do the right thing, instead of providing pages of instructions of what they must and must not do;
• Facilitating the most common workflows, while still allowing uncommon cases for experienced users (this includes providing sensible defaults and hiding uncommon fields)
• Using plain English language instead of jargon or legal terms, while still being specific;
• Asking as little information as possible (and required), and suggesting as much as possible (this includes extracting relevant information from the file's metadata).

Because of our limited grant period, we had to prioritize features included in the upload wizard. We optimized for the most common cases, in order to benefit as many users as possible, based on our research.

Software architecture

UploadWizard's multiple-file upload timeline diagram

For more technical details, see the UploadWizard's technical documentation on mediawiki.org

The Upload wizard relies on the same architecture already in use in MediaWiki and on Wikimedia Commons. However, it adds several other features and layers:

• It handles the simultaneous upload of multiple files.
• It provides a wizard-style interface
• It uses a "stash" system, a new way of storing data and media files that stops just short of publishing them to the Wiki.

The last item is particularly important mostly due to a quirk of how web browsers have traditionally worked. Web browsers can't analyze the files they are uploading or provide any information about them, not even a thumbnail -- they need help. As a consequence, the Upload wizard first uploads the files to the server, and then gets:

• Thumbnails for each image (helpful for identifying multiple files)
• An analysis of the metadata in each file. For instance, many photos have information hidden within them that tells us when they were taken. We can use that information to prefill many form fields. (The user can still change them).

The user can complete filling out all this information in relative peace, focusing on one thing at a time, not worrying if they've accidentally released an unlicensed file into the public sphere. And then when they're ready, they can publish it to the wiki.

Prototype

A prototype wiki was set up to provide a testing environment that would reflect the latest state of the upload wizard. The prototype's configuration was similar to that of Wikimedia Commons, but only contained a small subset of its content. Due to various compatibility issues, the prototype wasn't updated as often as desired. Nonetheless, it allowed volunteer testers to provide constructive feedback and to see the prototype evolve and improve over time.

UploadWizard 0.1

Screenshot of the simultaneous upload of files with the UploadWizard

A beta version of the UploadWizard was launched on Wikimedia Commons on November 30, 2010 A discrete link to it was added to the "own work" upload workflow on Commons. Files uploaded with the upload wizard are placed in a dedicated tracking category.

A detailed analysis of the tool's success was not possible in the time allocated, as expected in the grant proposal. Further research will need to be conducted to formally assess using qualitative and quantitative metrics.

Localization and dissemination

The upload wizard was developed as an extension for MediaWiki and released under the GPL. While it was primarily targeted at Wikimedia Commons, and though it is still beta software, it can be reused by any (1.16 and above) MediaWiki-based website that wishes to provide its users with a better uploading experience. It was built in such a way that it can be customized depending on each wiki's rules, especially regarding copyright.

As a MediaWiki extension, the upload wizard was made available for translation by volunteer translators on http://translatewiki.net. As of October 15, 2010, volunteers from 63 languages had participated in the translation process. The software interface was fully, or almost fully (> 90%), translated into 25 languages, including French, Indonesian, Chinese, Polish, Russian and Japanese.

Licensing tutorial

The licensing tutorial was the second main deliverable of the Multimedia usability project. Its primary audience was new users who were uploading their first media file to Wikipedia or Wikimedia Commons.

Most of them are not familiar with free licenses and have never been educated about copyright. The tutorial would be shown by default to any new user when they first upload a file, regardless of their language, culture or country of origin.

Objectives

Learning goals

The tutorial would be considered a success if the user completed it and:

• had a basic understanding of what copyright is
• had a basic understanding of what it means to release a file under a free license
• understood that any material is considered to be unfree unless explicitly said otherwise
• was able to determine, for simple cases, to which of the two big families (unfree or free) a media file belongs. Examples of "simple cases" include: a CD cover, a picture taken by the user, a picture from a movie and a random picture found on the Internet.

Requirements

We defined the following requirements for the tutorial:

• Short: the basic tutorial is not a goal per se, but a necessary step for new users who want to upload a picture. Going through the tutorial shouldn't last more than a minute, max. 2.
• Easily translatable: Wikimedia websites target an international audience, and Wikimedia Commons in particular is multilingual, as the central media repository for all other Wikimedia websites.
• From an international, cross-cultural standpoint: our audience is potentially anyone on the planet who has access to Internet, so we couldn't use cultural references that could be offensive or misunderstood by a specific subset of our audience.
• Fun, enticing, not too boring: we wanted users to actually read and complete the tutorial.
• Lightweight: our audience is very broad and we wanted everyone to be able to watch the tutorial, regardless of their bandwidth.

Format

Example of a comic strip created by a volunteer to advocate for dual licensing.

There had been several attempts in the past to create comic-like tutorials to explain basic topics about copyright and free licenses (see Supporting information), but their format made it particularly difficult to watch online and translate. An animated or still comic-like format (that would be displayed in an overlay or inline) was the best way to meet our requirements. Wikimedians usually prefer vector graphics to allow for resizing, localization and general editing; the preferred format for MediaWiki is SVG. Since animated SVGs were still in a prototype state, and not yet supported by MediaWiki, we decided to commission a set of still SVGs.

We also decided to avoid speech bubbles since they would make localization more difficult (especially for "verbose" languages).

The final deliverables were to be released under the Creative Commons - Attribution - Share Alike license.

Chronology

Supporting slides for the discussion with Creative Commons

Sharing creative works: a comic strip by and about Creative Commons

In February 2010, we invited Mike Linksvayer, Lila Bailey and Alex Roberts, from Creative Commons, to discuss the licensing tutorial. Educating the public about free licenses is a goal Creative Commons and the Wikimedia Foundation share. Creative Commons shared their experience in creating visual documents to explain free licenses, and offered to provide feedback on our tutorial.

Over the next few months, we discussed the project and the kind of person we needed for it with Jay Walsh, Head of Communications for the Wikimedia Foundation, and we created a creative brief.

Given our timeline, we decided to look for a local artist, rather than to send a global call for proposals. Jay Walsh reached out to his network, keeping in mind two major criteria: our limited budget, and our peculiar conditions (such as releasing the work under a free license).

We reviewed the work of several possible artists and narrowed down the list, with the help of David Peters, a local design director with whom the Wikimedia Foundation has been working on several design projects.

In August 2010, as a result of this preliminary work, we reached out to Michael Bartalos, an illustrator from San Francisco. Michael had previously done some interesting work for the California Academy of Sciences; his style was particularly close to what we expected for the tutorial. Moreover, he was one of the only artists already familiar (and comfortable with) free licenses.

After a few email discussions, we met in September 2010 to discuss the main topics that needed to be addressed by the tutorial; Michael Bartalos presented draft sketches. Following this discussion, we decided to move forward with this partnership.

Over October and November 2010, we went through a few cycles of creation, feedback, and corrections. A panel of experienced Wikimedians reviewed the successive iterations of the artwork and provided constructive and high-quality feedback, first on the general approach and content, then on specific details of wording and graphics.

Consolidated summaries then helped the illustrator work on the next version without having to deal with long discussions and contradictory statements.

Final product

Final tutorial in English

The "final" tutorial in English was officially published on November 5, 2010 It was only "final" from a project point of view. Indeed, the Wikimedia community quickly appropriated the tutorial and improved it further, particularly to optimize its size and rendering online.

Translation, localization and dissemination

Once the English version was published, the translation and localization process began. A call was made to volunteer Wikimedia translators to help translate the tutorial in as many languages as possible. A particular interest was given to languages whose Wikipedia has disabled file uploading locally, since they were more likely to upload files to Wikimedia Commons.

A translation and localization hub was created on meta-wiki for the licensing tutorial, containing recommendations and instructions to help volunteers. The tutorial quickly garnered interest from translators, who also appreciated the exhaustive information on the translation hub.

The translation of the content and localization of the graphics were handled separately, in order to more easily distribute the work depending on each volunteer's strengths.

As of January 2011, the text of the tutorial had been completely translated into 24 languages, and integrated into dedicated versions of the graphics (see below). Languages include Arabic, Bengali, German, Greek, Macedonian, Malayalam and Vietnamese.

Individual graphical pieces of the tutorial were uploaded to Wikimedia Commons in order to facilitate reuse. Some of them have already been remixed. Puzzly, the character from the tutorial, was also given a social presence on twitter, identi.ca and facebook.

• 
  Arabic
• 
  Bulgarian
• 
  Bengali
• 
  Catalan
• 
  Czech
• 
  Danish
• 
  German
• 
  Greek
• 
  Spanish
• 
  Finnish
• 
  French
• 
  Hebrew
• 
  Italian
• 
  Japanese
• 
  Korean
• 
  Macedonian
• 
  Malayalam
• 
  Norwegian
• 
  Polish
• 
  Portuguese
• 
  Romanian
• 
  Russian
• 
  Vietnamese
• 
  Swedish

Measures of success

Final study report

The grant proposal defined "the key measure of success" as "the ability of users in tests before and after the completion of the project to perform the given tasks": "If the failure rate of users in performing the given tasks is significantly reduced, the project can be considered a success." In order to measure this ability, we partnered with an independent UX and Usability research firm.

We reached out to some UX firms and published a Call for proposals in February. Several firms submitted proposals; after serious consideration, we chose to work with gotomedia, a San Francisco-based firm that seemed to align best with our goals and values.

The study was planned to take place in March, but was postponed because the prototype was not ready. In the meantime, we asked some of our co-workers to test it in order to uncover the most obvious flaws & bugs.

The actual testing eventually took place in June 2010. Eleven users participated in the test: five locally in San Francisco, and six remotely within the US. We considered conducting similar testing abroad, in order to identify language-specific issues; but in the end, it turned out that we wouldn't learn a lot by simply replicating the same test script. Multilingualism on Commons (and Wikimedia websites generally) is a huge piece of work that deserves dedicated efforts, and dedicated UX studies.

Results

Three highlight videos were produced out of this study: one about the testing of the current interface on Commons, one about the testing of the prototype, and the last one about how the prototype could still be improved. Highlight videos are edited summaries of the main findings of the study.

One of the main remaining issues was the fact that users don't really understand copyright and free licenses. Our Licensing tutorial was not yet available at the time of testing, but we believe it will help address this issue.

The content of these sections comes from the study report provided by independent usability firm gotomedia. The full study report is also available in PDF format.

Findings: Current Wikimedia Commons Upload Interface

Current interface testing video

In order to establish a baseline for usability, and to document the current interface, five participants were asked to use the current Wikimedia Commons Upload Interface to upload an image. For in-person participants, an image file was provided for the participant to upload. Remote participants were asked to upload any image file from their own computer.

• Participants rated the interface a 3.8 for “ease of use” on a scale of 1 to 10. All participants gave the interface relatively poor marks for “ease of use.” On a scale of 1 to 10, ratings ranged from 2.5 to 5, with an average score of 3.8.

“I’d give it a 5. It wasn’t necessarily difficult, but there were parts of it that left me a little lost.” -Carmen, 25, Student.

• Too much text. All participants who saw the current interface used phrases such as “too much text,” “text-heavy,” and “pages of text” to describe the interface. The current interface for uploading original work presents detailed instructions and guidelines for uploading files at the top of the page, and the full uploading form at the bottom of the page, below the fold. At first glance, none of the form fields are visible, and the page truly is all text. This proved to be intimidating to users, many of whom said that they would have been unlikely to continue with the upload were they not participating in the study.

“It was a real turnoff to engage in uploading a file and the first thing they sent me to was a page that was really text heavy with warnings. The message really is ’You might not know what you’re doing so you better not upload it.’” –Elina, 28, Graphic Designer

• The presence of wikitext, without clear instructions, intimidated users. All of the participants who tested the current interface had limited or no experience with editing Wikipedia. These users were understandably intimidated by the wikitext that they encountered in the upload form, and the help content that was provided effectively confused users even more. [...]

“Authors, tags and syntax, some people would be turned off by all this. At my level of knowledge I wouldn’t really know how to edit this part. There’s a real learning curve to upload it properly.” – Alexandra, 54, Treasure Hunt Designer

Findings: Wikimedia Commons Prototype Upload Interface

Prototype highlight video

All participants were asked to use the prototype interface to upload an image as if it were their own work. For in-person interviews, users were provided with a selection of images of the Golden Gate Bridge on the testing machine. For remote interviews, participants were asked to use any image file from their own computer.

All participants were then asked to upload a ‘found’ image from the web. For this task, participants were referred to a photo blog page that was created for this test. The blog page included a photo of the Golden Gate Bridge, with the author and licensing information clearly displayed on the page.

• All participants who saw the current interface vastly preferred the prototype. From the moment that the upload wizard loaded, users were immediately more comfortable with the prototype than they had been with the current Commons upload interface. Some participants who did not go through the current interface during testing had some previous experience with the current upload process. These users also preferred the prototype to their memory of the interface they had used in the past.

“I can already see that there’s a lot put in place to be a lot more encouraging and guiding for the user.” – Elina,28, Graphic Designer, on page load of the upload wizard.

• Participants rated the interface an 8.9 for “ease of use” on a scale of 1 to 10. All participants gave the interface high marks for “ease of use.” On a scale of 1 to 10, ratings ranged from 7 to 10, with an average score of 8.9.

“Oh, this was great. This was a 10, easily. I’m actually very happy through this process. It brings me a little bit of joy, and makes me want to upload more files.” – Jichen, 27, Videographer

“A 10! It was easier than I think any sort of file upload I’ve ever used. Very straightforward.” - Cassie, 29, Student

• The first step of the upload wizard was well received by all participants. Participants really liked the visual representation of the upcoming 4-step process. None experienced any difficulty at this step.

“It’s nice to have all the steps laid out for you, like explanations of how to do things that are visually clean.” – Erin, 27, Environmental Planner.

“First thing I’m noticing, I’m seeing this bar. I like that because it tells me there are 4 things, there’s a process and it tells me where I am in the process.” – Candy, 49, Business Consultant

• The pre-selected default license was well received. All participants easily navigated the first portion of the Release Rights step, selecting “This file is my own work.” Users appreciated that a license was recommended for them. Most indicated that they would be inclined to use the recommended license. Some were more concerned with the specifics of the licenses than others, but all appreciated the guidance provided by the recommendation.

“I’ve got a suggested license option, nice. That’s good.” – Ian, 40, University Professor

Summary

Usability and Comprehension

All participants who saw both the current Wikimedia Commons upload interface and the Upload Wizard prototype vastly preferred the prototype, and with good reason. Users found the current interface unnecessarily intimidating and cumbersome. One user went so far as to say the interface left him feeling “hurt and suspicious,” and another noted that the copious warnings in the interface left her feeling like she’d better not upload anything because she may not know what she is doing.

In contrast, the Upload Wizard prototype was very warmly received. Among those participants who had just used the current interface, an audible sigh of relief was often the first response upon seeing the first step of the prototype. All users understood what was expected of them during the upload process. All appreciated the clean uncluttered feeling of the layout, along with the top progress bar that laid out the upcoming steps and marked their location within the larger process. The prototype effectively removes wikitext from the upload process completely, which should be a goal for all Wikimedia Usability Initiative projects.

While participants used the prototype with relative ease, their comprehension of the licensing decisions they made during the upload was sub-optimal. It should be noted that there was no significant difference in licensing comprehension between users who saw only the prototype and those who saw both the prototype and the current interface. In the current interface, users were inundated by too much text, and so they did not read very much of it. In the prototype, users simply were not provided with the information that they needed to understand the licensing options. It would appear that comprehension for users who saw the current interface was no better than for users who did not.

Room for improvement

Room for improvement highlight video

The Upload Wizard prototype represents an excellent start on a new interface for uploading images to Commons. Going forward, efforts should be concentrated in two areas: making the interface more robust and reliable, and providing better access to licensing information and decision-making tools.

There were a few features missing from the interface at the time of testing, and performance issues hindered users’ experience as well. The progress bar did not display for most users during upload for example. Category selection was missing from the interface altogether. Now that a good solid infrastructure has been built for the interface, it is time to fill it out with a complete feature set and shore up the code to eliminate bugs and performance issues.

On the messaging front, the interface needs to provide access to more complete information about licenses without falling into the same traps that hampered the UX in the current interface. Wikimedia needs to provide just enough information within the wizard, along with access to information outside the wizard, to empower users to make informed licensing decisions, and without spoiling the clean uncluttered interface that our participants so clearly enjoyed. A change in the way uploads are described (‘donations’ instead of ‘contributions’) may also help user comprehension.

The Add Media Wizard prototype also represents a great improvement for Wikipedia and Wikimedia Commons. The wizard removes a lot of the direct manipulation of wikitext that is currently required in order to add an image to an article. The wizard will also raise awareness of Commons and the vast resources available there. Many Wikipedia users are completely unaware of Wikimedia Commons. When implemented, the Add Media Wizard will make contextually relevant content from Commons available to Wikipedia editors from within the editing interface. This should significantly improve the synergy between the two projects. Going forward, the relatively minor changes outlined above will improve usability.

Files

The highlight videos are available on Wikimedia Commons; per our agreement with gotomedia, all the videos were released under the Creative Commons Attribution - Share alike 3.0 license.

• Current interface testing: File page on Commons – Download OGV file (4m11s, 29.89 MB) – Watch on Vimeo.
• Prototype testing: File page on Commons – Download OGV file (5m32s, 35.43 MB) – Watch on Vimeo.
• Room for improvement: File page on Commons – Download OGV file (3m51s, 23.02 MB) – Watch on Vimeo.
• Final study report: File:Multimedia UX study June 2010 final report.pdf – Download PDF file (35 p., 1.6 MB)

Long-term analysis

In the long term, we expect to see a relative increase of participation in Wikimedia Commons. We will be able to measure this increase by comparing the rate of uploads per day before and after the deployment of the upload wizard. However, providing such measurements was not possible within the grant period. Still, files uploaded with the Upload wizard on Commons are tracked in a dedicated category.

Communications

During the entire project, an effort was made to be as transparent and inclusive as possible. This meant reaching out and listening to users and the Wikimedia community, but also proactively and regularly communicating our progress to a variety of audiences.

Usability wiki

Almost all of the documentation was hosted on the usability wiki and linked from the Multimedia hub. The Multimedia usability project was the first grant-funded project whose full grant proposal was published and publicly available.

The usability wiki was retired in January 2011, as both grant-funded projects that used it were over.

List of online communications

• Wikimedia Foundation receives Ford Foundation grant to grow Wikimedia Commons, a free educational media repository. Press release, Wikimedia Foundation, July 2009.
• Ford Foundation Awards $300K Grant for Wikimedia Commons. Erik Möller, Wikimedia Blog, July 2, 2009.
• Multimedia Usability Project Underway. Naoko Komura, Wikimedia Blog, January 26th, 2010.
• Améliorer l’ergonomie de Wikimedia Commons pour encourager la participation. Delphine Ménard, Wikimedia France's blog, November 19, 2009.
• Beyond Text: Report from the Multimedia Usability Meeting in Paris. Erik Möller and Delphine Ménard, Wikimedia Blog, December 1st, 2009.
• Help us collect good ideas to improve Wikimedia Commons. Guillaume Paumier's weblog, December 1, 2009
• IRC office hours, February 4, 2010
• Wikimedia Multimedia UX testing videos. Guillaume Paumier's weblog, July 23, 2010.
• Multimedia upload wizard: preliminary test results promising. Jarry1250, Wikipedia Signpost, July 26, 2010.
• Upload wizard: Questions & Answers, Multimedia usability team, August 7, 2010.
• Prototype upload wizard unveiled for Wikimedia Commons. Guillaume Paumier, Wikimedia blog, August 7, 2010.
• Multimedia Upload Wizard ready for testing. Jarry1250, Wikipedia Signpost, August 9, 2010.
• One-click reuse buttons on Wikimedia Commons. Guillaume Paumier's weblog, October 4, 2010.
• Illustrated licensing tutorial for Wikimedia Commons. Guillaume Paumier, Wikimedia blog, November 5, 2010.
• Wikimedia Commons licensing tutorial: the making-of. Guillaume Paumier's weblog, November 19, 2010.
• Wikimedia Commons Licensing Tutorial. Michael Bartalos, Bartalos Illustration, November 26, 2010.
• New Upload Wizard launches in beta on Wikimedia Commons. Guillaume Paumier, Wikimedia blog, November 30th, 2010.
• Commons Upload Wizard. TheDJ, Wikipedia Signpost, December 6, 2010.

Conferences

KDE Akademy 2010

Video of the KDE Akademy 2010 talk

Supporting slides for the KDE Akademy 2010 talk

In July 2010, Guillaume Paumier attended the KDE Akademy 2010 conference in Tampere, Finland. Along with Parul Vora, he gave a talk entitled Wikimedia User Experience programs: lowering the barriers of entry and presented the work done as part of the Wikipedia usability initiative, and the Multimedia usability project.

It might seem odd for Wikimedia to be presenting at KDE Akademy: Wikimedia is mostly about online content, while KDE is mostly about desktop software. Yet, they share common goals & values. On the one hand, a common criticism made against KDE is its complexity, a consequence of its high customizability. On the other hand, MediaWiki, the software on which rely Wikipedia and the other Wikimedia websites, suffers from the same flaws: it has always been "designed" by developers. As a consequence, the interface reflects the implementation model, and often doesn't match, or even conflicts with, the user's mental model. The Wikimedia Foundation recently started to include user research and design as part of their development cycle, where user experience is taking a increasingly critical role. This presentation at Akademy was an opportunity to share experience. Both KDE and Wikimedia communities struggle to improve complex interfaces, and both communities have a lot to learn from each other.

Thanks to KDE e.V. and their volunteers, the full video of the talk (and the follow-up discussion) was made available from the Akademy schedule page under the Creative Commons Attribution Share Alike license. A slightly edited version is also available from Wikimedia Commons, along with the supporting slides.

Besides this presentation, Akademy was also an opportunity to meet with the KDE community and discuss collaboration opportunities, especially to work towards a better integration of Wikimedia websites with the desktop. Ideas related to multimedia content include a mass upload tool, a "Picture of the day" desktop widget, and an application plugin to find media files from Commons from within an office application.

• Video: File page on Commons – Download OGV file (28m4s, 162 MB)
• Supporting slides: File page on Commons – Download PDF file (2.12 MB)

WikiSym 2010

Supporting slides for the WikiSym 2010 talk

On July 7-8, Guillaume Paumier attended the "International Symposium on Wikis and Open Collaboration" (WikiSym 2010). For the first time, WikiSym was collocated with Wikimania in Gdańsk, Poland (see below). It was a opportunity for researchers & Wikimedians (or "practitioners", as researchers call them) to get together and meet. The Wikimedia Foundation was then engaged in an effort to include research into their decision-making process in order to make it more data-driven. It was thus the perfect time to try and build awareness, understanding and relationships between the two communities.

Guillaume Paumier submitted a research paper presenting the results of the user research made for the Multimedia usability project, in particular the survey. The paper was accepted by the Program Committee, but could eventually not be published due to copyright restrictions imposed by the proceedings publisher. Nonetheless, he was able to give a 20-minute presentation of his work, entitled Understanding the users of Wikimedia Commons. The talk was well received and the audience particularly liked the videos from the UX study.

• Supporting slides: File page on Commons – Download PDF file (504 KB)

Wikimania 2010

Supporting slides for the Wikimania 2010 talk

On July 9-11, following the WikiSym conference, Guillaume Paumier attended the Wikimania 2010 conference, the annual community conference of the Wikimedia movement.

Guillaume Paumier gave a 40-minute presentation of the project, entitle "Increasing Multimedia participation on Wikimedia websites". The talk included an overview of the user research, a demo of the current interface, an assessment of the challenges Commons faces, and a demo of the prototype. Videos from the UX study were also presented. The talk was well received and the audience expressed interest in the upload wizard's deployment to production.

The presentation was filmed, but the recordings haven't been made available yet.

• Video: not available yet.
• Supporting slides: File page on Commons – Download PDF file (2.36 MB)

References