Community Wishlist Survey 2022/Better diff handling of paragraph splits/Test strategy

Project documentation

• Project page: Community_Wishlist_Survey_2022/Better_diff_handling_of_paragraph_splits
• Phabricator tag: https://phabricator.wikimedia.org/tag/better-diffs/

Feature documentation

• General help: mw:Help:Diff and Help:Diff
• Wikidiff2, which is used on production and beta wikis: mw:Wikidiff2

Test documentation

Where to test it

This needs to be figured out. How will we test the latest version of wikidiff2 and compare its output to previous versions?

How to install locally

By default, local wikis will use an "internal" diff engine.

In order to test wikidiff2 locally, you have to install it on your server. Doc to be written.

Coverage criteria and factors

• Diff engines
  • We may want to test the "internal" PHP diff engine in case UI changes cause problems
• Data
  • Different types of wikitext (e.g. tables)
  • Unicode, zero-width, non-printing
• Different content models
  • JSON, Javascript, CSS, etc.
• Different languages
  • Especially languages which have word tokenisation/segmentation different to English, such as Thai
• Different types of diffs and places where they appear
  • See /Types of diffs
• Different combinations of Wikidiff2 and VisualEditor
  • Wikidiff2 installed but not VE
  • VE installed but not Wikidiff2
  • Neither installed
• No JavaScript?
• Actions on diffs
  • Copy and paste diffs

Oracles

• Previous version of wikidiff2
  • We use this in the scripts #Random and #Selected
  • Could also use historical data
• Other diff engines
  • MediaWiki's internal diff engine
  • External diff programs (e.g. we use Python's difflib in #Random and #Selected)
• Hand-crafted "gold standards"
• UX mockups
• Self-verifying data?
• Metamorphic oracles?
• "Explainability", "Desirability" and "Purpose" oracles

Risks

• Vandals can craft their edits in order to avoid detection

Quality criteria

• Usability/understandability
• Accessibility
• Performance
• Other than the intended changes, diffs behave the same as before

Techniques

Nouns and Verbs

Generate sentences combining the nouns, verbs, adverbs and adjectives below to generate ideas for tests (please feel free to expand this list):

• Verbs
  • Add
  • Remove
  • Replace
  • Modify
  • Move
  • Split
  • Combine
  • Rearrange
  • Copy
• Nouns
  • Characters
  • Words
  • Sentences
  • Paragraphs
  • Newline
  • Whitespace
• Adverbs
  • Repeat(edly)
  • Sequentially
  • In parallel
  • From one <noun> to another
• Adjectives
  • Large
  • Small
  • Multiple
  • Similar
  • Extra

For example, "move sentences from one paragraph to another" gave me the idea for this test case.

Or "combine multiple paragraphs" for this.

Test data

See Community_Wishlist_Survey_2022/Better_diff_handling_of_paragraph_splits/Test_cases.

and w:User:Cacycle/diff_test_cases.

Further examples could be generated:

• Taken from real-life examples (e.g. from production wikis), see #Random
• Hand-crafted (see #Techniques)
• Created programmatically and/or at random

Test tools and scripts

See https://gitlab.wikimedia.org/dwalden/betterdiffs-testing

Random

• Script which finds random diffs from a wiki, runs them through the current MediaWiki diff and the wikidiff2 demo and compares outputs: https://gitlab.wikimedia.org/dwalden/betterdiffs-testing/-/blob/main/random_diffs.py

Selected

• Script which takes diffs you have selected, runs them through the current MediaWiki diff and the wikidiff2 demo and compares outputs: https://gitlab.wikimedia.org/dwalden/betterdiffs-testing/-/blob/main/all_all_diffs.py

Current testing status