Grants:Project/Accessibility of equation rendering: a comparative evaluation

statusnot selected

Project Grants

Accessibility of equation rendering: a comparative evaluation

summaryWe propose a study to analyze the accessibility of equation rendering techniques on the web, closing an important gap as no similar research is publicly available. This evaluation will focus in particular on the techniques currently used on Wikimedia sites as well as alternatives that are feasible to adopt by Wikimedia in alignment with its strategic direction and platform evolution.

targetAll Wikimedia sites containing mathematical formulas as well as MediaWiki in general.

type of grantresearch

amount55,000.00 EUR

type of applicantgroup

grantee• zorkow• pkra

advisor• Mobrovac-WMF• physikerwelt

contact• v.sorge@progressiveaccess.com

created on20:15, 14 November 2018 (UTC)

Friendly space expectations

Project idea

What is the problem you're trying to solve?

What problem are you trying to solve by doing this project? This problem should be small enough that you expect it to be completely or mostly resolved by the end of this project. Remember to review the tutorial for tips on how to answer this question.

There is a recognized need to modernize equation rendering in MediaWiki, thereby eliminating the current bad practice of serving different content to visual and non-visual users. However, there exists no comprehensive, in-depth comparative study on the quality of accessibility of equation rendering on the web today, which could help Wikimedia make an informed decision.

Background

Wikipedia and other Wikimedia sites are a critical resources of STEM [1] content worldwide. Ensuring the accessibility of mathematical formulas is critical for enabling knowledge equity in the STEM fields. Mathematical formulas on the web can be represented in a variety of formats, from bitmap images to pre-rendered SVG to MathML expressions. All render with varying quality leading to a rather non-uniform reading experience for users. This is in particular the case for users depending on screen reading technology to gain access to the STEM content. For example on commonly used websites like stackexchange.com mathematics is rendered directly from LaTeX input using the MathJax library, which allows the use of MathJax's built-in accessibility extension that provides assistive technology features like speech output, stepwise exploration of content, synchronized highlighting of expressions across all platforms. In contrast pages like Wikipedia, serve mathematical formulas in different formats to visual and non-visual users: visually in SVG format and non-visually in MathML format which can only be made accessible with specialist software on a limited number of platforms.

As differences between rendering techniques are generally not obvious to the less tech-savvy reader, it often leads to confusion on what technology can ensure a seamless accessible reading experience. This confusion has been expressed to the authors in a number of conversations by accessibility practitioners at specialist conferences such as CSUN [2], [3] and Accessing Higher Ground [4], [5]. However, there is currently not enough solid research to allow for a well-founded recommendation on the ideal document format that ensures both visual and non-visual high-quality rendering across all platforms.

There has been prior work, mainly on identifying the technology gap (see [6]) or aiming to provide support matrices to help identify suitable assistive technology for different types for mathematical and scientific documents on various platforms. To our knowledge two of these tools exist: The STEM Enable Wiki [7] and the Math Support Finder [8]. While the former aims to cover accessibility issues concerning a broad range of representation formats for mathematics, the latter focuses on accessibility of formulas in (Presentation) MathML only. Currently the STEM Enable Wiki appears to be defunct; Math Support Finder is available in a beta release since February 2017 with the most recent system test results from August 2016. In particular some of the latter’s functionality is currently not available or not yet implemented --- e.g., automatic detection of system settings --- and its database is very sparsely populated -- a full database readout yields 24 total results, with no entries available for the majority of assistive technology (AT) systems, file formats, or platforms indicated in the checkbox menus. Nevertheless, it aims to give accessibility practitioners some indication whether or not a particular assistive technology can produce one of three output formats (voice/Braille/visual) for MathML input. While this approach can be useful to initially triage the small number of systems and document types for which tests have been conducted, as it only provides binary pass/fail information for application/browser with limited distinction of release versions (i.e., with exception of Internet Explorer, “All” versions of an application are either accessible or not) based on a test set of 100 MathML expressions, it is too broad to begin to capture any nuances to make a qualitative assessment of rendering in various output formats. In particular, unaddressed issues are that no current system implements the full MathML specification, leaving the range of formulas that can be expressed unclear. Non-visual presentation of formulas often depends on heuristic interpretations of the markup, e.g., to correctly voice formulas or distinguish ambiguous representations. In addition interaction features such as stepwise exploration of content or synchronized speech and highlighting of expressions are an important factor, which are not addressed in Math Support Finder.

In summary, these classification attempts approach the problem either from a user perspective (STEM Enable Wiki) -- what is the right system or system combination for a particular need? --or from the system side perspective (Math Support Finder) -- which system can make MathML accessible on what platform? -- but they are not asking the question: What is the ideal format for a document to allow for a high-quality, cross-platform output in terms of visual and non-visual rendering?

What is your solution?

For the problem you identified in the previous section, briefly describe your how you would like to address this problem. We recognize that there are many ways to solve a problem. We’d like to understand why you chose this particular solution, and why you think it is worth pursuing. Remember to review the tutorial for tips on how to answer this question.

We will design a study for the qualitative comparison of rendering techniques for mathematical formulas in web pages. We thereby consider both visual and non-visual rendering, with a focus on the latter and their combination. We will primarily consider features such as speech output and interaction that are crucial to provide accessibility, in particular for screen reader users.

The results of the study will help MediaWiki and Wikimedia identify the best possible solution for accessible equation rendering on its platform while providing rich data to allow the wider community to benefit from our results. We will study solutions from a pragmatic point of view while incorporating long term trajectories of the involved technology stacks. The goal is to identify which solutions provide the widest possible positive impact on users as well as the clearest path towards further improvements at Wikimedia.

The results of this research aim to support Wikimedia's strategic direction [9] towards both main goals: knowledge as a service and knowledge equity. For knowledge as a service, our study can serve as a precondition to increase the quality of the content delivered to users (enriched formulas) and enable new interfaces (such as speech); cf. also the Product & Technology Working Group's report [10]. For knowledge equity, non-visual components of formulas that can be studied and explored will enable more people to comprehend Wikimedia content. Finally, the study will be beneficial both to Wikimedia as well as the communities, as it will provide a measurable benchmark for the quality of formula-related content on Wikimedia projects.

The open-source testing infrastructure we develop for the study will be based on the same technology stack as the MediaWiki Math Extension [11] and we will develop it so as to enable a possible adoption by Wikimedia (which would be independent of our proposed work).

Project goals

What are your goals for this project? Your goals should describe the top two or three benefits that will come out of your project. These should be benefits to the Wikimedia projects or Wikimedia communities. They should not be benefits to you individually. Remember to review the tutorial for tips on how to answer this question.

Our results will help MediaWiki and Wikimedia identify the best possible solution for accessible formula rendering on its platform while providing rich data to allow the wider community to benefit from our results. We will study solutions from a pragmatic point of view while incorporating long term trajectories of the involved technology stacks. The goal is to identify solutions that provide the widest possible positive impact on users and the clearest path towards further improvements.

From our combined experience in this space, we know that there is no silver bullet. The technological requirements of a platform, especially one as large as MediaWiki, creates a strong set of requirements -- from authoring guidelines via authoring tooling to rendering technologies. Nevertheless, we believe that there is a (local) maximum that can be identified, especially in the short to mid term, without limiting future development directions.

Project impact

How will you know if you have met your goals?

For each of your goals, we’d like you to answer the following questions:

1. During your project, what will you do to achieve this goal? (These are your outputs.)
2. Once your project is over, how will it continue to positively impact the Wikimedia community or projects? (These are your outcomes.)

For each of your answers, think about how you will capture this information. Will you capture it with a survey? With a story? Will you measure it with a number? Remember, if you plan to measure a number, you will need to set a numeric target in your proposal (e.g. 45 people, 10 articles, 100 scanned documents). Remember to review the tutorial for tips on how to answer this question.

Summary

Before detailing the structure of our methodology, we briefly summarize expected outputs and outcomes of our project.

Outputs

To achieve our goal, we plan six sprints (cf. work plan below) which will result in the following outputs:

• Assembling a set of formula samples providing a suitable representation of Wikipedia content

• Developing a test framework for rendering samples using different rendering technologies (several markup alternatives combined with all major browsers, platforms, and screenreaders)

• A documented experimental procedure for testing samples with the collated test platforms

• An analysis of advantages and restrictions of each rendering solution leading to a set of recommendations for future technical developments of accessible formula rendering in Wikimedia

Outcomes

The main outcome of the project will be an analysis of the pros and cons of each rendering solution to help Wikimedia identify a long term strategy for improving consistent and accessible formula rendering across all Wikimedia projects.

In addition there will be a number of outcomes that will be of interest for the scientific community and the web community at large and that can form the bases of further projects. Results of lasting scientific value will be

1. an experimental methodology to evaluate formula rendering on the web
2. a number of metrices to qualitatively compare formula rendering with respect to visual, aural and interaction properties.
3. a sample set of mathematical formulas that encompasses all aspects of formula rendering and accessibility.

Outcomes of a more transient nature will be

1. a comparison matrix for formula rendering quality with current web technologies.
2. a best practice recommendation for representation of formulas in MediaWiki/Wikipedia.
3. extensibility for future consideration with respect to areas such as Braille output, copy & paste, etc.

Technology review

In this study, we will evaluate commonly used methods for equation rendering. We outline the relevant rendering techniques for both visual and non-visual representation of equational content. In particular we will focus on the different type of source formats for rendering equations and how this can be translated into both visual and aural output with the variety of platforms and assistive technology tools available.

Visual Rendering

Visual rendering quality is perhaps the most important consideration for accessibility as the vast majority of users are at least partially sighted, including a majority of AT users.

To a considerable degree, the quality of visual rendering is a matter of opinion or taste. Nevertheless there are many technical aspects that can be evaluated objectively. Given Wikimedia’s history of using TeX-generated PNGs and now MathJax-generated SVGs (cf. below), its users are accustomed to visual equation rendering in the style defined by of Donald Knuth in Appendix G of the TeX Book [12]. This approach is generally considered the Gold Standard of equation layout and followed by many tools that are not using the TeX engine itself, notably Microsoft Word and MathJax. However, today’s level of browser layout engines means that such layout quality cannot be achieved without ensuring the availability of specific (web)fonts. While TeX-style layout may be considered dominant, there are several alternative approaches focused on font-independent layout using CSS. Notable examples include, Wikipedia’s math templates [13], MathJax’s PreviewHTML output [14], jqmath [15], and MathQuill [16].

Non-visual rendering techniques

The main part of this work will lie in evaluating the non-visual rendering, with a particular focus on the integration/combination of visual and non-visual quality.

Speech output

There are number of rule sets for translating formulas into speech as well defined grammars and/or practically implemented in working systems. The two most commonly known ones are: MathSpeak, a standard rule set that tries voice mathematics with as little ambiguity as possible, and Clearspeak, which aims at a more natural speaking style closer to regular reading of mathematical expressions, while still trying to be as little ambiguous as possible. However, Clearspeak is more designed for mathematics at the level of secondary education and can not cope as well as mathspeak with very complex expressions.

Other rule sets that have been implemented in working systems but never been formally defined are for example Simple Speak in MathPlayer [17], aimed at dyslexic students that discards verbal disambiguation for much shorter voicing and improve synchronization with expression highlighting, or some of the (undocumented) speech rule sets in Jaws [18] or VoiceOver [19].

Rule sets that go beyond simple voicing of expressions are implemented in Aster [20] and ChromeVox [21]. Their aim is to shorten speech strings for expressions by disambiguating via prosody elements, e.g., different speech rate or pitch sub- and superscripts or nested roots as well as variable length pausing.

Most of these standard speech rule sets attempt to form a general basis for generating unambiguous speech for equational content, these do not yield acceptable results for every, in particular more complex, formula, and can lead in some STEM subjects to downright incorrect voicing. Speech can be greatly improved when subject specific information can be taken into account. For instance MathPlayer used to contain some customization rules for specialist high school subjects like geometry or statistics, but nothing that would work on more advanced subjects, such as logic or theoretical physics. More recent attempts in MathJax work on incorporating context information from documents for improving interpretation and generation of speech output.

Heuristics for Semantic Interpretation

One important aspect of voicing mathematical formulas is their initial interpretation that allows the correct application of speech rules from rule sets mentioned above. This is in particular necessary, as representation languages like Presentation MathML have often little relation to the actual semantic meaning of an expression that needs to be voiced. On the other hand purely semantic languages --- like Content MathML for example --- often represent formulas in a format that has little correspondence to the visually rendered output, e.g., omitting visually important elements such as parentheses, meaning that their direct voicing can yield results that give in particular blind readers a disadvantage to their sighted peers. Consequently systems for voicing mathematics have to work with heuristics to interpret certain elements or even rewrite the presentation elements entirely into a more suitable semantic format.

Different systems have different quality of semantic interpretation, which can alter meaning of formula drastically. A classic example is the use of fractions and, in particular, the representation of binomial coefficients. The official Presentation MathML specification [22] recommends its representation as a fraction with invisible fraction line. Consequently, for voicing such an expression correctly depends on whether the thickness of division line is considered. If the thickness is not taken into account, the expression is incorrectly voiced as a fraction. If a heuristic rule checks for fraction lines with zero line-thickness it can be correctly spoken as binomial coefficient. However, a naive implementation of such a heuristic --- as currently in VoiceOver at time of writing --- leads to elements that are stacked using invisible fractions to be voiced as binomial coefficients, even for elements that are not enclosed in parentheses. But even if a visible fraction line is present, expressions do not always represent fractions. E.g., the Legendre symbol uses a fraction like notation. Similarly, inference rules [23] in logic or semantics of programming languages are usually rendered by the use of fraction constructs. Pronouncing any of these as fractions is, however, incorrect.

Other important expressions that can only be solved with adequate and often elaborate heuristics, are the correct interpretations of various set notations, of layout structures such as multiline equations or case statements, as well as of very domain specific vernacular such as the Bra-Ket notation in physics. For a longer list of ambiguous notation, cf. [24].

Interaction/exploration

Another important aspect of non-visual rendering is the ability for users to interact with the content (sometimes also called exploration). In many ways, this is the most critical part of accessibility as it determines the range of users who effectively benefit from any particular solution and it holds the most promise to leverage the web platforms ability to go beyond print paradigms of accessibility.

While a sighted reader might easily skim read a formula, select the order in which to visually parse it or go back to previous parts of an expression, this is not as easily achieved in a linear medium like speech. Verbal descriptions of formulas are often very long and easily confusing. Hence the ability to step-wise explore a formula, to jump between parts of it or summarizing subexpression are important. This is also an important aspect for readers with learning disabilities like dyslexia, where abstraction, summarization and shifting focus on sub-expressions but also selective and synchronized highlighting of the parts of a formula that is currently read, can greatly help with comprehension [25]. In particular for the latter a good interplay between visual and aural rendering is crucial.

In practise the above ideas are implemented by offering readers interaction with equational content via different exploration paradigms: For example linear exploration can be modelled by stepping through expressions in reading order of the speech, while more elaborate models exploit the logical tree of an expression to dive into subexpressions. In addition cursor virtualisation can allow users to jump between parts of formulas or allow them to explore complex two dimensional layouts in a more customized way. Likewise sub-expressions can be summarized or sometimes even visually and aurally abstracted. The quality of many of these techniques generally depend on the correctness of semantic interpretations of the expressions in question, as discussed in the previous sub-section.

An important issue here is also the smoothness of user interaction between document and equation level. Firstly, when reading documents, the presentation of equations should be embedded reasonably well into the context. E.g., during casually reading of a mathematical text it should not necessarily be always interrupted by very detailed presentation of every single formula. On the other hand, only announcing the presence of a formula without giving any clue as to its content is generally also not satisfiable. However, when a reader indeed wants to dive into the details for a particular formula the question on how well this can be embedded into the interaction flow is important. Technically, solutions range from direct browser interaction (often facilitated by a special key combination, effectively making every formula a rich web application), via changing the browsing mode of the assistive technology used, to the opening of external viewers with their own bespoke functionality and interaction model.

Our focus will lie on user interaction at both the document and equation level and the interfaces between those levels as well the quality of visual and non-visual highlighting and the synchronization of the two. This will include both native AT support as well as solutions that can be controlled by developers.

Markup techniques

There are essentially 4 approaches to realizing equational content on today’s web, with a few notable variations on each.

• <img> tags (*)
  • either using raster graphics (external or data-uri)
  • or using SVG graphics (external or data-uri) (*)
  • usually with inline styles (for alignment of inline content)
• tags with CSS styling
  • either with (web)fonts-dependent CSS
  • or with font-independent CSS
  • usually with various inline styles
• <svg> tags for inline SVG markup
  • either with characters as <path> (usually, optimized via <use> tags, locally or globally)
  • or with characters as Unicode (alongside data-uri embedded webfonts)
  • usually with inline styles (for alignment of inline content)
• <math> tags for inline MathML markup
  • requiring specialized (web)fonts (with OpenType MATH tables) for quality

For non-visual enhancements, the following techniques are most commonly used

• top-level, fixed textual descriptions
  • using alt attributes, aria-label/aria-labelledby attributes
  • provides fixed textual descriptions
  • compatible with: <img>, and <svg>
• JavaScript applications to enable voicing and exploration
  • can provide variable voicing styles, allows navigation through expressions
  • works with and <svg> only
• visually hidden but accessible MathML (*)
  • all methods

The current approach in Wikimedia platforms is marked (*).

It's important to note that the choice of markup impacts the non-visual rendering because HTML, SVG and MathML will be processed differently across the accessibility tool chain, in particular support varies greatly.

Considerations for MediaWiki and Wikimedia

Current technology

Originally, equations in Wikimedia sites used <img> tags with PNG graphics, alongside TeX sources as alt text; this rendering required an installation of texlive or other TeX/LaTeX distribution server-side.

Since 2015, the MediaWiki Math extension is using <img> tags with SVG graphics by default. These images are hidden from AT and combined with MathML markup that is visually hidden in a way that leaves the MathML exposed to AT. This method uses MathJax server-side for SVG and MathML generation; PNG images are still available for legacy platforms but are now generated from the SVGs (however, T194768 indicates the current implementation fails).

Before the switch in 2015, the maintainers of the Math extension had proposed to use MathML also for visual rendering on Firefox; after community feedback this idea was discarded. As a response to perceived problems [26], the use of MediaWiki templates (providing font-independent HTML with CSS for simple expressions) has found renewed interest among editors and additional templates have been created.

Serving different content to visual and non-visual users this way is generally considered bad practice for web content because it negatively impacts visual users who benefit from non-visual rendering (e.g. users with learning disabilities), cf. also [27]. In addition, MathML can only be made accessible with specialist software on a limited number of platforms.

From our previous discussions with Wikimedia staff, we are aware that visual rendering using native MathML or webfonts-dependent CSS is not an option for Wikimedia at this time due to quality issues and bandwidth considerations, respectively. The proposed research will be independent of these considerations.

MediaWiki- and Wikimedia-specific requirements

MediaWiki installations and Wikimedia sites in particular have certain requirements that afford both constraints and opportunities on the visual and non-visual rendering.

Speech. As speech strings would be pre-generated server side, one cannot use rule sets that strongly rely on prosody for disambiguation as the web platform currently lacks support for exposing such information. Currently, identical equations are stored only once in the database making it impossible to have different context specific readings of the same formula in parallel.

Semantics. An enrichment tool chain can make use of context information to enhance non-visual representations, e.g., as is currently being developed at MathJax. In particular, it should be possible to exploit information from Wikipedia pages directly as well as on their categories to improve heuristics for formula interpretations and speech generation. To that extent employing wikidata ids that are given internally in formulas could prove useful to provide improved semantic analysis [28] and thus non-visual rendering such as speech.

Exploration.Exploration of complex content is a key marker of the quality of accessible equation rendering. Given the complexity of the visual layout and the range of semantics, it remains a key struggle on the web today. For basic expressions, exploration is relatively easy to achieve while visually complex content or content that requires complex semantic analysis often requires (lightweight) JavaScript to provide exploration features as progressive enhancement. While JavaScript is usually ruled out by Wikimedia’s requirements such approaches might fit into a larger project on enhancing diagrammatic content across Wikimedia sites.

Formats. Given that Wikimedia currently relies on MathJax for server-side rendering, we will focus our evaluation on MathJax’s output options (MathML, SVG, CommonHTML, PreviewHTML) and we will build our test infrastructure on the same stack as the MediaWiki Math Extension [29]. However, our methodology is ultimately independent of the tools used to create such content and will immediately apply to other solutions based on MathML, SVG or HTML with CSS.

Methodology

We will mainly follow an experimental methodology to analyse the quality of visual and aural rendering of Wikimedia formulas. Initially we will assemble a sample set that will exhibit a diverse range of properties needed for the different aspects of rendered output we wish to examine. We then will define measures for examining formulas with respect to visual rendering, non-visual rendering and interaction. We now sketch the methodology in more detail as well as hint at some further properties that could be examined with the proposed sample set but would go be beyond the scope of the project.

Sample set design

Designing a good sample set faces several challenges. It needs to be large enough to cover most use cases yet small enough to keep testing a feasible option. It should give us general information on the state of technology yet provide insights that apply to the real world content on Wikimedia sites, in particular Wikipedia.

To tackle these challenges, we will design the sample set along the following axes:

1. Layout complexity
2. Subject areas
3. Layout ambiguity
4. Annotations

The first axis revolves around layout complexity. On the one hand, in terms of numbers of equations Wikipedia has more content in advanced areas, often requiring more complicated notation and layout. On the other hand, elementary areas using simpler equation layout make up a much higher usage of Wikipedia’s traffic [30]. Additionally, MW’s templates are limited to elementary layout, thus the number of elementary content is larger than expected. Our sample sets will balance these needs of both simple and complex equation layout by evaluating base layout constructs alongside transition tests that analyze how combining smaller pieces to larger constructs affects accessibility.

The second axis will be semantic analysis and knowledge domains. Given the breadth of knowledge domains available on Wikipedia (alongside metadata), our sample sets will test the ability of AT to leverage such information, using expressions from, e.g., mathematics, physics, chemistry, theoretical computer science. In particular we will examine if expressions are voiced correctly, understandable but not necessarily appropriate for the domain or outright incorrect as well as examine if speech changes if a formula is given outside the context of a particular page.

The third axis will be layout ambiguity. Many constructs of traditional equation layout are highly ambiguous, making it difficult to convey the information accurately across visual and non-visual renderings. For example, differentiating between a fraction or the Legendre symbol, between a 2-dimensional vector and a binomial coefficient, or between a table of equations and a Jacobian matrix can be very difficult. This difficulty increases further across knowledge domains, e.g., differentiating BraKet notation. Identifying such patterns is a significant problem for understanding equational content, in particular for non-visual users. Our sample set will cover ambiguous notations both elementary and advanced, across various domains.

The fourth axis will be annotational elements. Equational layout often includes annotations (textual or otherwise) using under/side braces, columns of plain text (e.g., function cases) and other means of providing additional context. Identifying such elements and meaningfully conveying their content is extremely difficult. Our sample set will cover typical constructs of annotational elements.

While we aim to generate our sample set mostly by leveraging real world Wikipedia expressions we might need to expose certain problems using custom-build expressions as well. For the former, we will build on the list of frequently used formulas (by salixalba), thereby we will also be mindful to ignore expressions built for specific purposes (e.g., user signatures). For the latter, we will consider already known issues such as use of special glyphs in MathML representations, unsupported unicode characters in TeX input, etc. This part will also be useful for any follow up work on localization or bidirectional rendering. In addition, we will seek community input and contributions to our sample set both during our initial design phase and when iterating later on.

While we will focus on English content for practical reasons, we will include samples that test internationalization and localization from Wikipedia's Spanish and Arabic version to ensure that our results apply independently of the host language.

Evaluating Aural quality

For the non-visual rendering we primarily will focus on the speech that can be generated for aural display of equations. In particular we will examine:

• Quality of semantics analysis
  • Are complex nested structures correctly voiced?
  • Is ambiguous notation recognized and correctly disambiguated?
  • Are annotations recognized and distinguished from the formula text?
• Domain adaptations
  • Are formulas correctly voiced with respect to their subject domain?
  • Is context information being taken into account?
• Naturalness of voicing and use of rule sets
  • Does the voicing follow any established rule sets (e.g., MathSpeak, ClearSpeak)?
  • Are formulas “naturally voiced” as suggested by the context?
  • Can there be ambiguous interpretations of a formula?
• Localization
  • Is localization an option?
  • Can formulas be automatically localized?
• Customization via authoring
  • Is author supplied information in the formula markup taken into account?
  • Is author supplied contextual information taken into account?
  • What means of customization are supported (Alt text, Aria labels, custom data elements)?

Evaluating Interaction/Exploration quality

As suitable user interaction is an important aspect of full accessibility of formulas we will also examine the following aspects pertaining to exploring expressions:

• • user interaction during document-level exploration
  • accessing equations (cf landmarks)
  • navigating in and out of equations
  • triggering equation exploration
  • summary equation descriptions vs detailed descriptions
• expression navigation
  • character by character vs tree navigation
  • correctness of exploration order
  • positional announcements such as denominator/enumerator
• highlighting
  • visual options (color choices, hi-contrast modes etc)
  • non-visual highlights (prosody, aural hints / earmarks)
  • subexpression highlighting
  • synchronized highlighting with speech output

Evaluating Visual quality

As beauty is in the eye of the beholder we will focus our evaluation for this minor part of our study on aspects that can be assessed more objectively. The following aspects will be considered:

• impact on non-visual rendering and exploration
• integration into surrounding context, in particular
  • baseline alignment for inline content
  • scope and quality of CSS inheritance (color inheritance, text style, link styles etc)
• robustness, in particular
  • consistency of visual rendering across platforms
  • rendering quality under failure conditions (no CSS, no fonts with relevant glyphs)
  • zoom
• adaptability, in particular
  • quality in various output formats (e.g., regular HTML, mobile app, browser reader modes, epub, PDF)
  • adapting to viewport size (e.g., overflow handling)

Regarding integration and CSS inheritance, we should note that there are incompatibilities between the traditions of print and web-based equation layout. In print, equations can only convey meaning visually, and adapting to the text context is traditionally intentionally limited (e.g., in a print heading, an equation will not match boldness of the heading). On the web, other traditions have evolved. For example, it is considered bad practice not to inherit the color and decorations within an <a> tag, especially if the link’s content consists only of an equation; properly accessible rendering will have no problem conveying the difference.

Evaluating Internationalization and Localization

An accessible equation rendering solution should work for all people and thus it should support content in all languages and layout traditions. This includes proper support for Unicode (in particular for graphemes) as well as bi-directional and vertical content. Such support of course should extend to non-visual representations. In our experiments we will test support for internationalization in order to derive a recommendation for the future development of equation rendering in MediaWiki.

These is of particular importance as in its current state there is no way to incorporate localized non-visual information in the MediaWiki ecosystem. Equation rendering on Wikipedia is stored in a global database, re-using renderings across languages. Consequently, there can at most be one speech string attached to each formula, restricting possible localization to one language, only. In addition MediaWiki's Math Extension (and thus WMF sites, including Wikipedia) lacks support for most Unicode ranges on the input end and they also lack support for bidirectional equation content.

Another complication stems from the Math Extension’s input format, a (limited) TeX syntax. Backwards compatibility with TeX/LaTeX has been a preference in the past yet despite longstanding feature requests (cf. https://phabricator.wikimedia.org/T4458, https://phabricator.wikimedia.org/T50118, https://phabricator.wikimedia.org/T50032) there is no agreed upon methodology for handling localization and complex Unicode constructs in TeX’s math mode.

An overview of relevant standards and the state of technologies in our recommendations will allow Wikimedia to make an informed decision to provide proper internationalization in the future.

Platforms

We will test all valid (cf. below) combinations of the following platforms

• Operating systems: Microsoft Windows, Mac OS, Linux, iOS, Android
• Browsers: Google Chrome, Microsoft Internet Explorer, Microsoft Edge, Firefox, Safari
• Screen readers: NVDA, JAWS, VoiceOver, Talkback, ChromeVox

We will limit our evaluation to latest versions and to combinations that are officially well supported; for example, some screen-readers are known not to work with some browsers. On the other hand some screen readers only have math support via external libraries such as MathPlayer. We will not evaluate screen magnifiers and related tools such as ZoomText, Dolphin, TextHelp EquatIO as they have limited functionality, and often do not leverage the same accessibility APIs as screen readers.

Other Considerations

While this project will concentrate on the issues above, we are mindful of other considerations that might be investigated in the future and will try to take them into account when putting together out sample set.

Braille

Unfortunately, the web platform currently lacks means for exposing both aural and Braille alternatives thus there is effectively no way to provide Braille directly. While there are proposals to fill this gap (including prototypes in AT), there is nothing to base tests on.

Currently, existing ATs with support for creating Braille representations for equation content rely on the liblouis library [31] to generate specialized Braille formats such as Nemeth Braille from formats such as MathML. As the only exception, Speech Rule Engine is currently working on a new approach for generating Nemeth Braille from enriched MathML. Some proprietary tools generate and expose Braille in various forms on the web (e.g. Desmos, Pearson equation editor).

We will therefore not directly test Braille support but include an overview of relevant standards and the state of technologies that would allow MediaWiki to provide Braille in the future.

Copy & Paste

One advantage of Wikipedia pages is the easy reuse of text via copy and paste. For formulas similar considerations could be made. For example, some browsers allow for copy and pasting alt text descriptions and LaTeX alt text can easily be reused in reader documents. Investigating and providing formats that easily support this, ideally with different output and without inclusion of noise could be a future concern.

Computation

Similar to copy and paste supporting the direct use of formulas in computational engines like computer algebra systems or statistic packages is of growing interest. While there is currently little support for this task in mathematics found on the web, this could be an area of interest in the future.

Do you have any goals around participation or content?

Are any of your goals related to increasing participation within the Wikimedia movement, or increasing/improving the content on Wikimedia projects? If so, we ask that you look through these three metrics, and include any that are relevant to your project. Please set a numeric target against the metrics, if applicable. Remember to review the tutorial for tips on how to answer this question.

Our study will provide a reliable data set to evaluate a specific set of technologies around mathematical formulas on Wikimedia sites. This data set combined with our analysis and recommendations will help inform Wikimedia's development team on how to improve the technology stack in alignment with Wikimedia's strategic direction and the overall platform evolution.

As a result, we expect our work provide the foundation for future development at Wikimedia which will improve content with mathematical formulas on all Wikimedia sites as well as MediaWiki installations leveraging the same technology stack by enabling a more accessible rendering. This will improve Wikimedia platforms in terms of knowledge-as-a-service while ensuring knowledge equity.

Project plan

Activities

Tell us how you'll carry out your project. What will you and other organizers spend your time doing? What will you have done at the end of your project? How will you follow-up with people that are involved with your project?

Work plan

We plan around 500h of work for 2 researchers over 6 months, with an equal split between PI and CI. In addition, we plan to engage with the advisor on a regular bases. Work is split into 4 technical and one administrative workpackage. Workpackages are further broken down into tasks.

Gantt chart illustrating the work plan for the project.

WP 0: Adminstration and Dissemination

• 0.1: Set up reporting infrastructure (Month 1)
• 0.2: Engage with advisor on requirements and limitations of Wikimedia infrastructure (Month 1)
• 0.3: Engage with advisor on draft recommendations (Month 5)
• 0.4: Write-up technical report on work. Prepare scientific publication (Month 6)
• 0.5: Meet with relevant technical teams on results (Month 6)

Deliverables:

• Wiki-based reporting infrastructure (month 1),
• Technical report (month 6)

Time: PI (25h), CI (25h)

WP 1: Test infrastructure

• 1.1: Collation of non-visual components (Month 2)
• 1.2: Collation of visual components (Month 2)
• 1.3: Assembling of test environment (Month 2)
• 1.4: Update of test environment after initial tests (Month 4)

Deliverable: Test Environment (Month 2),

Time: PI (75h), CI (50h)

WP 2: Sample set design

• 2.1: Initial collection of test data (month 2)
• 2.2: Update collection of test data after initial test phase (month 4)

Deliverable: Public sample set (month 4)

Time: PI (50h), CI (75h)

WP 3: Test phase

• 3.1: Initial test (month 3)
• 3.2: Final tests (month 5)

Time: PI (50h), CI (50h)

WP 4: Recommendations

• 4.1: Initial draft recommendations (month 5)
• 4.2: Final recommendations (month 6)

Time: PI (50h), CI (50h)

Deliverable: Report (month 6)

Budget

How you will use the funds you are requesting? List bullet points for each expense. (You can create a table later if needed.) Don’t forget to include a total amount, and update this amount in the Probox at the top of your page too!

• Principal Investigator @ 250h @ EUR 120
• Co-Investigator @ 250h @ EUR 100

Total: EUR 55,000.00

For the detailed breakdown, please see the workplan above.

Community engagement

Community input and participation helps make projects successful. How will you let others in your community know about your project? Why are you targeting a specific audience? How will you engage the community you’re aiming to serve during your project?

We will seek community input and contributions to our sample set both during our initial design and test phases. Our work will be exposed in publicly available repository (most likely github) and we will give regular updates in the relevant Wikimedia fora.

In particular, we will also seek input from math community members [32] and community members using assistive technologies (in particular via the WikiProject Accessibility [33]); given sufficient interest, we will add a short UX survey.

We will also accept any contributed test data from the community once the test framework and evaluation procedure are in place. The study will not depend on such contributions.

As further outreach we plan to disseminate not only as a recommendation for the technical teams at the wikimedia foundation and the volunteers working on equation rendering but we also intend to publish our scientific results. In particular, we will publish a technical report as well as submit scientific papers to relevant conferences such as CHI, Assets and W4A.

Get involved

Participants

Please use this section to tell us more about who is working on this project. For each member of the team, please describe any project-related skills, experience, or other background you have that might help contribute to making this idea a success.

• Grantees
  • Volker Sorge (Zorkow (talk)) is Reader at Birmingham University (UK) and founder of Progressive Access Solutions. He maintains Speech Rule Engine [34], an open source JavaScript library for semantic analysis of mathematical formulas. Volker is also a core developer at MathJax [35], the leading, open-source JavaScript library for formula rendering on the web.
  • Peter Krautzberger (pkra (talk)) is an independent developer and consultant working primarily with STEM publishers to improve their conversion processes for web-based formats (e.g., full-text articles, reflowable ebooks). He co-chairs the W3C MathOnWebPages Community Group [36] and was manager of the MathJax [37] Consortium from 2012 through 2017.
• Advisors
  • Moritz Schubotz (physikerwelt (talk)) is a post-doctoral researcher at Universität Wuppertal. Moritz maintains the MediaWiki Math Extension.
  • Marko Obrovac (Mobrovac-WMF (talk)) is part of the WMF Core Platform team.
    • Formulae rendering is done on the back-end via the Mathoid service. As part of the Platform Evolution CDP, we envisage major changes in the way our software works, and having a comprehensive study such as this one inform the possibilities we have in serving content (in this case formulae) to users will be invaluable. As someone who has been involved in Mathoid for 3-4 years now as well as someone involved in the PE CDP, I think I can provide input and guidance to the grant proposal authors throughout the project's lifetime on various technical aspects. Mobrovac-WMF (talk) 20:48, 28 November 2018 (UTC)

Community notification

Please paste links below to where relevant communities have been notified of your proposal, and to any other relevant community discussions. You are responsible for notifying relevant communities of your proposal, so that they can help you! Depending on your project, notification may be most appropriate on a Village Pump, talk page, mailing list, etc. Need notification tips?

• Phabricator "Math Committee Thread" [38]
• WikiProject Accessibility [39]

Endorsements

Do you think this project should be selected for a Project Grant? Please add your name and rationale for endorsing this project below! (Other constructive feedback is welcome on the discussion page).

•  Support This project will define the foundations for the advancing the math extension. Moreover, it will evaluate which method is suitable to grant access for people with special needs to mathematics in Wikipedia and other Wikimedia projects. Physikerwelt (talk) 08:55, 19 November 2018 (UTC)
•  Support This money would be well spend and improve accessibility not just for Math on Wikipedia, but accessibility of math in general, since a lot of websites use Media-Wiki and even more use MathJax. The answer to my question here shows that in Wikipedia there is much room for improvement. The rendering of non-latin characters mentioned is another area which desperately needs improvement, cf. this. Debenben (talk) 21:55, 26 November 2018 (UTC)
•  Support We at Elhuyar Fundazioa work, among other things, in speech solutions for accessibility, and we have worked in various projects regarding the Basque Wikipedia with the Basque WikiPedia Users Group, the most prominent of them being Wikispeech. We think that the ability for assistive technologies (such as Wikispeech or others) to read out equations properly in any language is crucial for universal accessibility. We belive that this project will be a great step towards the proper reading reading of equations in Wikipedia and other sites. Elhuyar Fundazioa (talk) 12:48, 29 November 2018 (UTC)
• for far to long education has been able to by pass students with disability's saying the technology is not there yet for them to have equal access this study will help say this is not true and help find the technology that does the job 136.152.143.189 22:01, 3 December 2018 (UTC)
•  Support I support this approach! Andreg-p (talk) 08:36, 5 December 2018 (UTC)