Research:Section Translation Post-Improvements Testing
Section Translation brings translation support to mobile device editors. This project provided usability testing after a number of tool improvements and at a time when it was becoming available in a greater number of wikis, including Thai Wikipedia. We are grateful for the support of Teak Research Consultancy to help carry out this project.
Background
editWe observe high mobile usage across many language versions of Wikipedia, including relatively ‘small’ wikis such as Bengali and Malayalam, as well as those with relatively larger article collections, such as Thai. Section Translation became available in the first wiki, Bengali, in early 2021. Since that time, a number of improvements have been implemented based on previous research projects (e.g., Bengali Wikipedia Section Translation Usability Testing, Section Translation Entry Points Research). This project provided post-improvements usability testing of Section Translation in Thai Wikipedia shortly after it became available in additional wikis with a greater diversity of wiki, cultural, and linguistic qualities.
Research Goals
editThe primary goal of this project was to evaluate the general usability of the Section Translation experience and workflow, including a number of recent improvements. In particular, what are the main points of friction that may in part or fully prevent editors from making successful contributions. Relatedly, where is better support needed throughout this process? Research sessions and data collection will result in either validation of the various tool elements and workflow and/or findings to support recommendations for modifications.
The Section Translation workflow includes a number of steps that must be completed successfully in order for editors to publish a new article section. These steps include:
- Confirming and/or adjusting source/target language settings
- Selecting an article and article section to translate
- Reviewing section contents in the source/target language and initiating the translation
- Reviewing, editing, and adding to the machine translation output generated by the tool
- Completing the translation and publishing
In addition to the main Section Translation workflow, the project evaluated a recently implemented entry point, and one other entry point concept in the form of a clickable prototype. A short warm-up interview portion at the beginning of each research session collected information about more general aspects of Thai Wikipedia’s editors’ experiences, particularly related to translation.
Approach
editModerated research sessions with 17 participants using a smartphone allowed an opportunity for existing Thai Wikipedia editors to use Section Translation to translate and publish real wiki content. It also included targeted questions throughout the task observation, as well as pre- and post-task questions. Finally, the sessions ended with participants being asked to rate and comment on their ratings for various aspects of the Section Translation experience.
Highlights of Results
editFor the full report of results and recommendations, please refer to the complete Section Translation Thai Wikipedia Testing Final Report. The section below is not intended to be inclusive of all project results; instead, it contains some highlights of project results and recommendations.
Overall, Section Translation was perceived by the participants who are current Thai Wikipedia Editors as a simple tool for newcomers and people with average editing markup skills. They perceived the overall process as straightforward and easy to follow, but faced difficulty with discovery, relevant suggestions, publishing with confidence, and lack of a Source Editor option. Some important challenges and points of friction include the following:
- Current entry points lack discoverability
- Current suggestions (suggested articles and sections) do not complement the interests of editors
- Publishing options aren't easy to locate, thus making editors want to leave the process
- Experienced editors prefer the Source Editor interface
Just over half of these participants found the process of reviewing the source and target language articles helpful, and it was easy to switch between the two languages. However, it is important to have the source preview match the complete existing article in Thai so that content can be easily compared.
As for potential newcomers (multilingual web users with experience creating content and using translation, but not currently registered Wikipedia editors and with no extensive coding experience) Section Translation was perceived as a complicated tool The overall experience is logical, but they could not associate many of the interface options with their intended functions. They think that the UI should provide a clear direction so that they better understand the task. Many of the specific points of friction and challenges noted were similar to those faced by experienced Thai Wikipedia editors, including:
- Current entry points lack discoverability
- Current suggestions (suggested articles and sections) do not complement the interests of editors
Notable differences were that the process of reviewing the source and target article was more challenging for newcomers. Specifically, they didn't understand how to accomplish the task in this process, in part because they struggled to compare the different language versions of the article. Secondly, newcomers had difficulty understanding UI. And, third, the tutorial did not provide sufficient instruction regarding the usage and process of the tool.
There are a number of recommendations offered based on findings of this evaluative research project. The following list highlights a few of these:
- Improve entry points, including an easy way of accessing the Section Translation dashboard.
- Add greater personalization to the suggested articles/sections, so that users are presented with relevant and interesting options.
- In the section selection process, make the search option more apparent as this is currently the most common way that users identified articles/sections to translate.
- Emphasize the benefit of reviewing an article by displaying the location of the missing section in the existing article. This helps editors to better understand the mismatch between the source section and the current target language article.
- Make the tutorial more interactive and inclusive of more key aspects of different tool features, and only proactively display intructions when an editor initially accesses Section Translation.
- To support experienced editors, provide a Source Editor option. If this is not possible, minimally address the limitations and challenges they face when using Visual Editor.
- Provide options for translating by paragraph as well as sentence-by-sentence.
- Provide more tools related to translation, such as dictionaries and spell check.
- At the publishing stage, ensure that editors have choices and feel like they're in control. Support their desire to easily 'save as draft', and consider how their informal peer review processes support both quality and confidence in publishing.
A number of other recommendations, including interface suggestions may be found in the full report, along with full reporting of results.