Grants:Project/Frimelle and Hadyelsahar/Scribe: Supporting Under-resourced Wikipedia Editors in Creating New Articles/Midpoint

This project is funded by a Project Grant

proposal

people

timeline & progress

finances

midpoint report

final report

Report accepted

This midpoint report for a Project Grant approved in FY 2018-19 has been reviewed and accepted by the Wikimedia Foundation.

• To read the approved grant submission describing the plan for this project, please visit Grants:Project/Frimelle and Hadyelsahar/Scribe: Supporting Under-resourced Wikipedia Editors in Creating New Articles.
• You may still review or add to the discussion about this report on its talk page.
• You are welcome to email projectgrantswikimedia.org at any time if you have questions or concerns about this report.

Summary

• First version of the gadget deployed to test Wikipedia
• Community communication
• Hiring software developer
• First prototypes of Scribe with volunteers at hackathons
• First steps of automating Scribe’s essential parts

Methods and activities

(1) Technical Development

Scribe interface development

We have followed an iterative process for the technical development of scribe. After each iteration we make sure to take the feedback of the community members.

The first development activities of scribe was at May 2010 at the Wikimedia Hackathon in prague [Scribe prototype 1 demo] this version was kept as a userscript.

The second prototype we implemented was at Wikimania hackathon in August 2019 [Scribe prototype 2 demo], during our interactions with the community members in Wikimania we got many eye opening comments and suggestions. The most helpful of them was to make scribe support mobile editing, being very popular among new editors in under-resourced languages. What we learned during this development is the technical challenges and points of contact from the community for questions, especially those related to Visual Editor interactions.

The Third iteration has started since Nov. 2019 when our team became complete with Eugene, we started the development of Scribe as a Media wiki gadget while the back-end databases and server code is hosted on WikiMedia cloud services. This will enable extensibility and maintainability by the community after the period of the project.

The current version of scribe has already automated several parts of the pipeline such as section retrieval and recommendation, while other parts of the pipeline such as references suggestion is yet to be automated in the 2nd half of the project. As of January 2020 Scribe has been deployed on test wikipedia as a gadget with a set of supported example articles to be able to test the UI. On the technical side we have all learned a lot from this experience, such as designing extensible software, the differences between a gadget, a userscript and an extension and each of their advantages/disadvantages. Engaging the technical community and ask for feedback early. Our iterative approach of developing Scribe UI, has enabled us to have a base for discussion.

Research on Scribe

We follow the usual Make It Work Make It Right Make It Fast path when doing research in scribe and pushing it into production. After surveying a large amount of literature that is related to scribe tasks (summarized in our Wikimania presentation). For the current iteration of Scribe we decided to start with a few selected examples from GLAM for Catalan and Arabic languages. The first objective of this experiment was to understand the challenges with respect to scarcity of resources when dealing with topics in underserved languages. The second objective is during this first path we will automate as much as possible of the full scribe pipeline yet have something that is final and deployed to acquire the community feedback.

Section extraction and recommendation: has been fully automated and extensible to examples beyond those few selected examples. We noticed that editors follow a common Section structure between relevant articles of the same type. The titles of those sections might slightly differ making their alignment a challenging task than slight text matching for example in articles about Museums we can find 3 versions of sections describing the history of the museum “History”, “History of the museum”, “It’s History”. Nevertheless, we devised a method that relies on selecting the most redundant sections based on type of the target topic. This method proved to be quite effective and yielded high quality section suggestions,  it is yet to be tested to other domains beyond GLAM in the second part of the project.

References retrieval: to have a better understanding of this task and how to automate it we manually went through the process of finding references online & automatically summarizing them. We learned from this that finding references in Catalan is quite difficult, because Spanish is so dominant, so this might yield us to investigate more high coverage approaches such as relying on search engine that indexes large selection of articles.

References ranking and recommendation: we created a white-list of references by language and domain to analyse references credibility on Wikipedia the summary of this investigation has been presented by Lucie in her NewsQ workshop presentation in New York, USA.  We learned from this that for both the domains of Museums and Women in science there is a high overlap between references used in  Catalan and Arabic Wikipedias, these are mostly in English references. While Catalan Wikipedia cited more references Catalan and Spanish languages. In Arabic the top 10 cited references are social media sites  such as youtube, twitter, instagram. Finally, we provision that to ensure we suggest of references with high credibility we will have to exclude some of those references, even from the white-list this can be done through a manual intervention process by the community, i.e. using the Wikipedia community to create a domain specific lists of high quality sources. A similar approach has been used for the Strephit project for references retrieval for Wikidata statements.

(2) Community and Outreach

Our outreach activities focused on building a community around Scribe, inside and outside the Wikimedia world, to collect feedback and to gain an understanding for the needs of the community, ultimately shaping the project so that it can suit each community in the best way.

Understanding editor’s behaviour

One of the main focuses was user research, to understand how editors currently work. Therefore, we focused on speaking with editors on events and through online interviews. Semi-structured interviews were conducted remotely with 7 editors of different language communities, focusing on how they currently edit and in which ways they discover references. The interviews are not yet published, as they are still ongoing and we plan to use to contacts we gained from conference attendees to gather more perspectives. We already got an insight on how editors work on different topics, depending on whether they have already knowledge about it and therefore offline resources or it is a topic they perceive as needed but don’t have resources, and therefore research online. This helped us understand what type of sources are typically used.

One of the most important changes due to the conversations with editors, online and offline, was that we were pointed to the fact that a high number of editors uses mobile devices to edit, which lead us to reshape the development towards a mobile-first approach. This has the advantage that we can provide for the communities we are aiming at as well as forces us to think of a modular step-by-step process as there is a limited amount of things we can display on a mobile page. Following these conversations, we created a new design for the Scribe gadget as well as focused on the Wikimania hackathon to implement with a focus on mobile devices.

Wikimedia conferences

To build a strong connection with the Wikipedia communities, we attended a row of conferences, and worked with the communities there, the technical communities as well as the editors of different language Wikipedia communities. We attended the Wikimedia Hackathon, where we worked with newcomers on a first prototype for Scribe. We attended Wikimania and the Wikimania Hackathon, where we built a new prototype as well as presented Scribe in the context of research [link to slides] and at the Communities Growth space [link to slides]. Further, we participated in WikidataCon, where we gave a presentation on the usage of Wikidata for low-resource languages and Scribe in particular [link to slides]. At WikiArabia, we got a chance to connect with the Arabic speaking community and presented Scribe [link to slides] as well as research around Arabic Wikipedia [link to slides].

We collaborated on Scribe with the new editors team and the editor retention team at WMF, having meetings in and outside conferences with both teams.

External activities

Besides promoting and working with volunteers inside the Wikimedia community, we made an effort to also present in context outside the traditional Wikimedia world. We attended the Civil Servant workshop in Stockholm, August 2019, where the conversation was around research projects in Wikimedia. We were invited to give a presentation about Scribe at the NewsQ workshop in New York, January 2019, focusing on referencing in Scribe [link to slides]. We were invited to present our work on Scribe at the Citizen Beta event in London, January 2019 [link to slides]. Finally, we worked on a tool using Scribe’s reference finding for a news credibility in the Hack the Press Hackathon in London, January 2019. The project is called WikiRank and ranks references used in Wikipedia based on a topic domain. The code was developed in collaboration with 6 hackathon participants and can be found here

To emphasize the importance of the research parts of Scribe, and potentially getting researchers of related fields involved, we published a blogpost

Midpoint outcomes

We could have the very first version of Scribe deployed to test.wikipedia, which enables us to test on the first live setup. This was a great result of the work of Eugene, who worked on the implementation of the previous prototypes into a real extension and thanks to the volunteers who enabled us to deploy the extension to test.wikipedia. The code for the gadget can be found here.

We were able to build strong ties to the technical community, leading us to the first two prototypes, the first one from the Wikipedia hackathon in Prague can be found here built in collaboration with Ondřej Merkun and Joe Reeve. Joe Reeve also supported the second implementation of a prototype at the Wikimania hackathon in Stockholm that focuses on mobile-first development. This switch to mobile-first development came from one of the many interactions we had with the community.

We did the first advances on collecting references and section titles, based on efforts to understand how referencing works first manually and then automated. We created whitelists for museums and female scientists for Arabic and Catalan Wikipedia. The code and datasets can be found here

Working on references of Wikipedia is an important topic also outside the Wikimedia communities. The previous work was put up in a hackathon, building a tool for credibility of news sources. The code can be found here.

We also created a phabricator board with the help of the community, that can be found here.

Our outreach activities focused on connecting with a variety of editors and technical community members at conferences and through e-mails. The focus was to get feedback and people interested to suggest installing the gadget in their home Wikipedias. We also reached out to researchers through a blogpost, and by attending different outside Wikipedia events.

Finances

With regards to finances, we have spent them conservatively in order to be able to afford the events and meetings with community members, once we can test Scribe in-person with them.

Project Management, Research: We have spent the money on the project management accordingly to the time we (Hady and Lucie) spent working on the project.

Part-time developer: We have hired Eugene, who is now working with us in a software developer role since 1st November 2019.

Visit of Wikimedia Events: We have visited Wikimania, WikiArabia and WikidataCon, which was calculated as part of our travel budget. We are planning to visit the Wikimedia Hackathon, and we still have left a fair amount to enable us to visit this or another Wikimedia conference that we see fit to meet the community and work with them on Scribe.

Professional Translation fees: So far we have not needed a professional translator. We will spend part of this budget on professional transcription of the interviews, which we have not yet done.

Event Organization Giveaways & Buffer: For the buffer and give-away we still have all the money to our disposal, but this will also mostly spend on community activities once we start testing with the community.

Handling tax: bookkeeping and tax calculation was quite challenging to do on our own. Tax consultancy helped us a lot with this task, which is part of the expenses we did not account for previously but falls within the limits of our buffer. While this depends on the geographical and the experience of the grant holders, this is is something we recommend future grant holders that have similar situation such as ourselves.

Learning

The best thing about trying something new is that you learn from it. We want to follow in your footsteps and learn along with you, and we want to know that you are taking enough risks to learn something really interesting! Please use the below sections to describe what is working and what you plan to change for the second half of your project.

What are the challenges

• Difficulty of time management
  • Given other commitments, such as full-time work on the side, we struggled in the beginning with management of the time.
  • Our solution was to make regular meetings (once per week for development and once per week for research) to keep everything going, and use holidays to meet for focused work sessions on specific problems
• Prioritization of tasks
  • We had problems to decide, which task is most important, how to implement the minimal functional version first, and what are the most important features.
  • We tackled this by discussion and task lists (e.g. on phabricator) to have clear goals and an understanding of dependencies between tasks as well as deadlines.
• Scope of the project
  • The overall scope of the project, closly tying in to the previous point, made it hard to focus and trying to develop a solution for all problems at once made us lose overview at times.
  • We addressed that by doing one step at a time, ensuring high quality of the project at each step. Instead of doing everything at once, we focus on one domain at a time, which makes our work more precise and testable and we can then scale it step by step to more domains.
• Hiring: finding candidates with the relevant set of skills
  • Most candidates had relevant technical skills but lacked experience in the Wikimedia world. For example the technical skills needed to e.g. set up the backend on cloud services, visual editor interaction, or working with templates.
  • We disseminated through Wikimedia channels and ended using the help of CodeLn, which were connected to us through the community to find a developer that worked in the Wikimedia technical community before and has connections to the community (Eugene).

What is working well

Technical development of Scribe is going well, although it includes many technical challenges, especially when it comes to the integration into Wikipedia and the MediaWiki environment, such as VE, cloud services and similar. Eugene was able to solve the problems to date, using previous experience, extensive research and his connections to the technical community.

Working with the community and exchanging with community members has been already very beneficial. When talking about the project we got a lot of new ideas, some of which changed the project quite fundamentally such as going mobile-first.

Disseminating  has given us very positive feedback about our project and helps us to prepare the next step for community outreach through the connections we have built with different low-resource language Wikipedia community members.

Next steps and opportunities

• The focus now is to deploy Scribe to several low-resource language Wikipedias
• For this, we will do user-testing with community members and get consent from Wikipedia communities to install the extension
• We will publish the previously done interviews, possibly collect more interviews with the Arabic community using connections made through WikiArabia
• Developing the reference retrieval methodology with a community contribution to whitelist of references

Grantee reflection

We have been working on Scribe over the past 6 months, with a focus on building a community around it, that is interested to be involved in our project. This was a particularly motivating experience, collecting feedback from the community members involved and changing priorities, such as working on a mobile-first interface. The community generally has been very supportive of our efforts. Getting the technical Wikimedia community involved was extremely helpful, as we could learn from their experiences and understand which challenges are ahead of us, such as the integration into the visual editor. With regards to that, we would like to thank the community members, who have been supporting us along the way, both from the technical community (such as Joe and Ondre) as well as the Arabic Wikipedia community and the Wikidata community and the Wikimedians we got to interview for the study on how editors currently work. Working with Eugene particularly was a joy - we got to work with a talented engineer, who has worked in Wikimedia projects previously, and supported the development of the extension.

Integrating research ideas, such as information retrieval and document summarization for the community has lead us to think more about the connection of the research and the Wikipedia community, leading us to organize an event for the communities to interact on the 08.02.2020 in London.