Research:Newsletter/2018/January

Vol: 8 • Issue: 01 • January 2018 [contribute] [archives]

Automated Q&A from Wikipedia articles; Who succeeds in talk page discussions?

With contributions by: Eddie891, Thomas Niebler, Barbara (WVS) and Tilman Bayer

"Reading Wikipedia to Answer Open-Domain Questions"

Reviewed by Thomas Niebler

This paper by Chen et al. proposes to use the Wikipedia article corpus as a source of world knowledge in order to answer open domain questions. They point out that Wikipedia articles contain a lot more information than current knowledge bases, such as DBPedia or Freebase. While knowledge in KBs is encoded in a more machine-friendly way, the vast majority of Wikipedia's knowledge is not covered in KBs, but contained in unstructured text and is thus difficult to access in an algorithmic way. The proposed approach, called "DrQA", aims to overcome that limitation by leveraging the article content. It first retrieves Wikipedia articles relevant to a question, and then uses a recurrent neural network (RNN) to detect relevant parts in the article's paragraphs that could be used as answers. This RNN is based on a set of pretrained word embeddings as well as a set of other features.

Their results indicate that DrQA seems better suited to answer open domain questions than other competitors, based on a set of four question benchmarks. While the evaluation score improvement seems rather small (77.3 vs 78.8 F1 score), the whole task of machine reading at scale using Wikipedia gives directions for interesting future research and applications. For example, depending on the speed of the framework (which unfortunately was not discussed), a new Wikipedia service for answering such open domain questions could be established. Furthermore, this process of answering common knowledge questions could help in improving chatbots.

Are you a policy wonk? Who succeeds in talk page discussions

Reviewed by Barbara (WVS)

This Carnegie Mellon University study quantified the success of those editors who engage in talk page discussions and their roles in these discussions. The roles assigned to each editor was:

• Moderator - decides when a decision is final to support their views
• Architect - designs the article and its sections to support their views
• Policy Wonk - quotes acronyms that represent policy/rules/guidelines to support their view
• Wordsmith - determines the best article titles and section titles based upon their point of view
• Expert - interjects facts into the discussion to support their point of view

Unlike earlier studies exploring editor interactions, editors in this study could be assigned simultaneous roles on an article talk page. Success of each editor was determined by analyzing subsequent edits to the article under discussion which were promoted by a particular editor and longevity of these edits. Those editors that are more detail-oriented tend to have more success than those more interested in organization. Multiple editors assuming the role of organization lessens the success of individual editors. The study assessed 7,211 articles, 21,108 discussion threads, 21,108 editor discussion pairs, and the average number of editors per discussion. The number of total edits by an editor is not associated with success.

The researchers also published a dataset consisting of "53,175 instances in which an editor interacts with one or more other editors in a talk page discussion and achieves a measured influence on the associated article page".

"Determining Quality of Articles in Polish Wikipedia Based on Linguistic Features"

Summarized by Eddie891

This article focuses on the 1.2 million unassessed articles in the Polish Wikipedia, and considers "over 100 linguistic features to determine the quality of Wikipedia articles in Polish language." From the conclusion: "Use of linguistic features is valuable for automatic determination of quality of Wikipedia article in Polish language. Better results in terms of precision can be achieved when the whole text of [an] article is taken into the account. Then our model shows over 93% classification precision using such features as relative number of unique nouns and verbs (unique, 3rd person, impersonal). However, if we take into account only [the] leading section of an article, relative quantity of common words, locatives, vocatives and third person words are the most significant for determination of quality. Using the obtained quality models we [assess] 500 000 randomly chosen unevaluated articles from Polish Wikipedia. According to result, about 4–5% of assessed articles can be considered by Wikipedia community as high quality articles."

Conferences and events

See the research events page on Meta-wiki for upcoming conferences and events, including submission deadlines.

Other recent publications

Other recent publications that could not be covered in time for this issue include the items listed below. contributions are always welcome for reviewing or summarizing newly published research.

Compiled by Tilman Bayer

• "Enrichment of Information in Multilingual Wikipedia Based on Quality Analysis" From the abstract: "Wikipedia articles may include infobox, which used to collect and present a subset of important information about its subject. [sic] This study presents method for quality assessment of Wikipedia articles and information contained in their infoboxes. Choosing the best language versions of a particular article will allow for enrichment of information in less developed version editions of particular articles." See also coverage of related papers involving the same author above, in our last issue: "Assessing article quality and popularity across 44 Wikipedia language versions", and below:
• "Analysis of References Across Wikipedia Languages" From the abstract: "This paper presents an analysis of using common references in over 10 million articles in several Wikipedia language editions: English, German, French, Russian, Polish, Ukrainian, Belarussian. Also, the study shows the use of similar sources and their number in language sensitive topics."
• "Wikipedia as a space for discursive constructions of globalization" From the abstract: "This article [...] compares, through computer-assisted text analysis and qualitative reading, entries for the word ‘globalization’ in six major Western languages: English, German, French, Spanish, Portuguese, and Italian. Given Wikipedia’s model of open editing and open contribution, it would be logical to expect that definitions of globalization across different languages reflect variations related to diverse cultural contexts and collective writing. Results show, however, more similarities than differences across languages, demonstrated by an overall pattern of economic framing of the term, and an overreliance on English language sources."
• "FRISK: A Multilingual Approach to Find twitteR InterestS via wiKipedia" From the abstract: "In this paper we describe Frisk a multilingual unsupervised approach for the categorization of the interests of Twitter users. Frisk models the tweets of a user and the interests (e.g., politics, sports) as bags of articles and categories of Wikipedia respectively [...]"
• "Introduction to anatomy on Wikipedia" From the abstract: "No work parallels the amount of attention, scope or interdisciplinary layout of Wikipedia, and it offers a unique opportunity to improve the anatomical literacy of the masses. Anatomy on Wikipedia is introduced from an editor's perspective. Article contributors, content, layout and accuracy are discussed, with a view to demystifying editing for anatomy professionals."
• "The institutionalization of free culture movement based on the study of Wikimedia projects in the East-Central Europe" From the English abstract: "The author of the publication presents the processes of institutionalization occurring in the projects of the Wikimedia Foundation, co-organized in the framework of the free culture movement. These processes on the one hand lead to the relative closing up of the members of groups belonging to regional cultures, especially those who speak the same language, on the other hand to encouraging interregional cooperation. Common enterprises undertaken by partners from East-Central Europe are not only contribution to the free culture movement, but may also point to emphasizing the common identity of prosumers of post-socialist societies."
• "The Russian-language Wikipedia as a Measure of Society Political Mythologization" From the abstract [sic]: "The analyzed in this article myth about inheritance rights of Russia to the Kyivan Rus’1 arose in the 15th century. Recently this myth is being actively spread by the Russian propaganda in the mass media – in particular this is performed through Wikipedia being one of the most attended Internet resources. [...] the purpose of this myth consists in activation of separatist sentiments of Russian-speaking Ukrainian citizens. Purpose – to explore vulnerability of Wikipedia policy of openness on the basis of a specific example as well as to explore its efficiency for formation of political myths; to analyze the technology used for creation of Wikipedia articles in the process of formation of myths.Methods. Comparison method is applied – texts of Wikipedia articles on various time stages of their creation were compared; results of analyzing Wikipedia pages were correlated to political events of Russian-Ukrainian relations.[...] Results. Mythology not obliged to prove anything and Wikipedia aimed at forming the concept and creating only an impression of scientificness and not knowledge as such are perfectly agreed. That is why Wikipedia is one of the most efficient spreaders of myths (first of all political myths) supporting a definite ideology."
• "Analysing Timelines of National Histories across Wikipedia Editions: A Comparative Computational Approach" From the abstract: "... we aim to automatically identify such differences by computing timelines and detecting temporal focal points of written history across languages on Wikipedia. In particular, we study articles related to the history of all UN member states and compare them in 30 language editions. We develop a computational approach that allows to identify focal points quantitatively, and find that Wikipedia narratives about national histories (i) are skewed towards more recent events (recency bias) and (ii) are distributed unevenly across the continents with significant focus on the history of European countries (Eurocentric bias). We also establish that national historical timelines vary across language editions, although average interlingual consensus is rather high ..."
• "Using WikiProjects to Measure the Health of Wikipedia" From the abstract: "We analysed 3.2 million Wikipedia articles associated with 618 active Wikipedia projects. The dataset contained the logs of over 115 million article revisions and 15 million talk entries both representing the activity of 15 million unique Wikipedians altogether. Our analysis revealed that per WikiProject, the number of article and talk contributions are increasing, as are the number of new Wikipedians contributing to individual WikiProjects." From the results section: "In comparison to Suh et al. and Halfaker et al., our findings suggest that based on the WikiProject activity, Wikipedia is not in decline, but still enjoying growth with new users, edits, and discussion activity. Akin to other complex online communities, using traditional methods to measure community and system health may not reflect their true state ..."

References

Wikimedia Research Newsletter
Vol: 8 • Issue: 01 • January 2018
About • Subscribe: Email • [archives] • [Signpost edition] • [contribute] • [research index]