Wikimedia Meta-Wiki

Grants:Project/Susannaanas/GLAMpipe

< Grants:Project
statusselected
GLAMpipe
summaryGLAMpipe is a Swiss knife from Finland for content and data manipulation and upload to Wikimedia projects.
targetWikimedia Commons, Wikidata
type of granttools and software
amount5 087 €
contact• susannaanas(_AT_)gmail.com
this project needs...
volunteer
affiliate
advisor
join
endorse
created on13:30, 6 October 2016 (UTC)

Project idea edit

What is the problem you're trying to solve? edit

Content imports to Wikimedia Commons and data imports to Wikidata are always complex. They require sorting out messy (meta)data, using a combination of tools and technologies, working your way through practices and policies, and finally creating a software tool for the import. And the work is always done on a case-by-case basis.

What is your solution? edit

To facilitate these workflows, Wikimedia Finland has developed a tool that can read data from different sources, manipulate it in a number of manners, and export it to a variety of repositories.

GLAMpipe is an extensible, open source web-application intended for cultural metadata manipulation. The workflow is based on documents and nodes instead of plain tables. Nodes are like building blocks of data flow. They can act as a data source, they can split, combine, create wikitext or process data other ways, and nodes can export data to files or web services like Wikimedia Commons or Wikidata.

GLAMpipe is a tool for power users. Compared to Pattypan, it serves more complex use cases. Compared to GLAM-Wiki Toolset, it offers tools to work with the data, and makes import and export very easy. It gives the user the power of bots without the need to code.

Project goals edit

When successfully launched and deployed, the tool can unlock vast amounts of data and content. It gives data-savvy users, such as GLAM professionals, the ability to upload content to Wikimedia Commons, or data to Wikidata.

It will give the users an opportunity of reusing the earlier data manipulation solutions created by the community. The programmable nodes are like customizable legos for each step of the import process for data or files.

  1. With this project we want to focus on bringing data to Wikidata: creating a node for preparing data, and another one for saving prepared data to Wikidata.
  2. Secondly, we focus on the distribution: delivering the tool in different delivery packages - as an online tool and a more powerful desktop application.
  3. As a third point, we work with a group of test users to make the interface easy to use for GLAM professionals and other data partners. We work with them by tracking issues in the code repository, collect evaluations with a survey and meet online to discuss the experiences and to develop solutions.
  4. The process will produce tutorial videos and documentation for using the application and developing additions to it. The aim is to have the tool adopted by the GLAM-Wiki community.

GLAMpipe has been developed in 2016 with the support of the Finnish Ministry of Education and Culture. This project will help it be adopted by the key GLAM-Wiki contributors, and extend it’s usage to Wikidata. The plan to develop the tool extends beyond the grant period. More capabilities can be developed in further project extensions based on needs.

Project plan edit

Activities edit

General framework development edit

An online version of GLAMpipe will be created that is able to use OAuth login. Currently GLAMpipe is intended for desktop installation. However, installation is quite technical and it is not suited for non-technical users. Initial online version exists, but it requires more work in order to be reliable. A regular web location will be established in Wikimedia Labs, for example.

Wikidata node edit

New nodes for transforming data for Wikidata will be created. The transformation node needs to take into account the triplet data structure, be able to map data to Wikidata and format appropriately. The Wikidata export node takes transformed data hash as parameter, makes sanity checks for it and saves it to Wikidata. The export node checks that there are no duplicate values and merges statement to existing ones if possible. The export node is made using a Widar-like OAuth interface and plans to share its code.

Co-development and community building edit

To measure the expectations of our test users and their satisfaction with the tool we will carry out a survey. The survey can be complemented with additional data further along the lifecycle of the project.

We will also arrange 2 online meetings with the test users to go through their feedback and solve design issues together.

The GitHub repository will serve as the platform for user support and communicating about the issues that arise during testing.

A community site will be established in the context of GLAM outreach and documentation.

Outreach and learning edit

A set of instructional videos will be produced to illustrate the use of GLAMpipe for different use case scenarios.

There will be 3 blog posts: in Wikimedia Finland blog, in Wikidata blog (by permission) and one other. GLAMpipe’s process will be publicized in the Wikidata + GLAM Facebook group and the Wikidata mailing list.

GLAMpipe will be presented in several Finnish and 1 international events during the grant period.

Reporting and documentation edit

The grant report will be created bearing in mind any possible learning outcomes, and published possibly in other formats and environments.

Schedule edit

The project is scheduled to run 6 months from January 2017 to June 2017. The due date of the project will be postponed to 30 September 2017. – Susanna Ånäs (Susannaanas) (talk) 16:19, 13 June 2017 (UTC)

Budget edit

Issue Ari Kimmo Susanna Amount Sum In-kind hours In-kind sum
General development
Online version development 16 h 16 h 368 € 16 h 368 €
Installation in the server environment 8 h 8 h 16 h 368 € 6 h 138 €
Wikidata import node
Data transformation design 32 h 32 h 736 €
User interaction design and testing 16 h 16 h 368 €
Node development 32 h 32 h 736 € 12 h 276 €
Wikidata export module 32 h 32 h 736 €
Co-development
Test user survey 8 h 8 h 184 €
Online discussions 6 h 6 h 6 h 18 h 414 €
Community discussion and support 8 h 8 h 16 h 368 €
Outreach and learning
Instructional videos 8 h 8 h 184 €
Publicity (blog posts, social media) 4 h 4 h 4 h 12 h 276 € 2 h 46 €
Travel to Wikimedia hackathon x 250 €
Accommodation at Wikimedia hackathon x 3 x 79 € 237 €
Reporting, documentation
Reporting 4 h 4 h 10 h 18 h 414 €
Documentation of core and node creation 12 h 6 h 18 h 414 € 6 h 138 €
Total 90 h 100 h 52 h 6 053 €
In-kind 42 h 966 €
Applied sum 5 087 €
hourly salary 23 € / h

The budget table has been updated 11.10.2016. Part of Ari's contribution was made into an in-kind contribution, since he can work on those tasks as part of his day job.

Additional support edit

We request to be eligible for travel funding, if suitable events emerge for presenting the tool.

We will ask for permission to host the tool in Wikimedia Labs. While we do not expect to require specific help, we hope to make note of the possibility.

Community engagement edit

A small group of early adopters is our initial community. We will develop and enhance the functionality of the tool in collaboration with them. They do test uploads and we review their experiences together, both through issue reporting in GitHub and in joint video sessions.

Interested volunteers may take one of the following roles:

  • Developers can review and create custom nodes.
  • GLAM partners can learn to use the tool for actual uploads. If they do not feel capable, they can team up with a Wikimedian, or a technical member of their staff.
  • Wikimedians working with GLAM imports can join the test users.

Sustainability edit

The application is an extensible, open source web-application. The source code is maintained in an open repository. Clear documentation of the code aids contributors to step up.

The tool is distributed on several platforms: as an online tool and a more powerful desktop version. This increases the ease of use and widens the user base.

The modifiable nodes can be altered or created by the users. With the help of the documentation contributions from the users will increase.

The tool takes advantage and reuses existing third party components.

Measures of success edit

Successful contributions made with GLAMpipe will form the basis of metrics for the project. Another indicator is the growth in the number of the users of the tool.

To measure the success in responding to the community needs, we will create an online survey. The key indicators are the ease of use, efficiency and comparison to alternatives.

Number of edits made with GLAMpipe A growing trend, numbers to be negotiated
Change in the number of users 200%
Ease of use Over 60% positive
Efficiency Over 60% positive
Comparison to alternatives Over 60% positive

Get involved edit

Participants edit

Kimmo Virtanen edit

Kimmo has administered the Finnish Wikipedia for several years. He develops solutions to integrate Wikidata in Wikipedia while taking care of the everyday technological framework of Wikipedia in Finland. While Kimmo is a Wikimedia Finland board member currently, the term is changing at the end of the year.

Ari Häyrinen edit

Ari worked for Wikimedia Finland as a technical specialist in 2014–15, and he has continued to conclude the work in the GLAMpipe project. His dissertation from the University of Jyväskylä summed up the use of open source tools and open data in the cultural sector. He works with GLAM metadata in his day job.

Susanna Ånäs edit

Susanna has been an active member of the GLAM-Wiki community for several years, pushing forward tools development for GLAM uploads and advocating the use of Wikidata. She promotes Wikimedia for historical mapping and as a component of Digital Humanities research. She administered Wikimedia Finland in 2014–15.

Community notification edit

Endorsements edit

Do you think this project should be selected for a Project Grant? Please add your name and rationale for endorsing this project below! (Other constructive feedback is welcome on the discussion page).

  1. This is a cool idea! Indeed we have also been discussing the need for such tool. -Vojtěch Dostál (talk) 20:27, 12 October 2016 (UTC)
  2. Great idea! It would be very useful for us at WMAR :) Mauricio V. Genta (talk) 20:32, 12 October 2016 (UTC)
  3. This sounds like a fantastic development, and something i would definitely use to share National Library of Wales data. Jason.nlw (talk) 20:53, 12 October 2016 (UTC)
  4. Sounds like a useful tool. - PKM (talk) 21:19, 12 October 2016 (UTC)
  5. Really interesting and useful tool. --Marcok (talk) 21:44, 12 October 2016 (UTC)
  6. The tool sounds great. Having a tool that easy to use then writing your own bot is great. ChristianKl (talk) 22:28, 12 October 2016 (UTC)
  7. Well-designed tools to support content contribution are sorely needed. This proposal looks detailed and actionable. Jtmorgan (talk) 22:29, 12 October 2016 (UTC)
  8. Cool! It sounds to be very useful. Uomovariabile (talk) 08:12, 13 October 2016 (UTC)
  9. Looks like it could be really useful for the GLAM community! --Magnus Manske (talk) 09:14, 13 October 2016 (UTC)
  10. This sounds like a really great and relevant tool 85jesse (talk) 09:35, 13 October 2016 (UTC)
  11. Might become a very useful tool. Wesalius (talk) 11:27, 13 October 2016 (UTC)
  12. Mass upload of content from external organisations is either very time consuming, requires a huge amount of technical knowledge or both. This project has the potential for many more people to be able import content from external organisations and websites avaiable under a suitable license.
    Additionally if this tool is able to make importing data from external sources into Wikidata easier this would be an enormous help. Currently the process for importing data using automated tools is incredibly time consuming, requires a lot of knowledge and trial and error.
    In short the potential is huge John Cummings (talk) 19:29, 16 October 2016 (UTC)
  13. A very reasonable amount of funding for a project that has also received financial support from outside our movement, which is great. See my input on the talk page. Endorse per John Cummings. Spinster (talk) 09:40, 18 October 2016 (UTC)
  14. Slowking4 (talk) 02:48, 20 October 2016 (UTC)
Retrieved from "https://meta.wikimedia.org/w/index.php?title=Grants:Project/Susannaanas/GLAMpipe&oldid=16877360"