User:IIgwilo (WMF)/Real Time Preview for Wikitext

Test strategy for the Real Time Preview for Wikitext wish.

Project Documentation and High-level Requirements

• Original wish: Community_Wishlist_Survey_2021/Editing/Live_preview
• Project page: Real_Time_Preview_for_Wikitext

Quality Criteria

• The preview should be a faithful representation of how the main content article will look once published
• The preview box should render invalid wikitext output in a way that helps the user recognise and correct errors
• Edits the user makes to the wikitext should be displayed in the preview in a reasonable amount of time
• Users should be happy with this feature from a usability perspective

Risks

• The real-time preview viewport is small. It might look different compared to viewing the article full-screen.

Oracles

• The real-time preview should match the "normal" preview
• The real-time preview should match the actual article once it is published

Coverage Criteria

• Coverage of wikitext (including invalid wikitext)
  • multimedia
  • templates
  • things which change geometry radically (e.g. tables, images)
  • things which change position on the screen (e.g. images when you float them)
  • Parser tests have lots of examples of wikitext we could use: gerrit:plugins/gitiles/mediawiki/core/+/refs/heads/master/tests/parser/ (every .txt file in that directory)
• Different content models, wikis, namespaces
  • To find different wikis on production and beta (resp.) you can go to:
    • https://commons.wikimedia.org/wiki/Special:SiteMatrix
    • https://commons.wikimedia.beta.wmflabs.org/wiki/Special:SiteMatrix
  • To find different namespaces you can go to Special:AllPages on whichever wiki you are testing
  • A query to find pages using different content models on a wiki: /Finding_content_models
• Section edits
• Browser coverage
• How this feature interacts with other features of the wikitext editor
• Types of users/personas and use-cases
• Languages and RTL
• Screen size

Techniques

• HTML comparison
  • Compare the HTML of the preview with the HTML of the actual article (using a diff tool, for example)
• Stress test
  • What happens when we try to preview a huge amount of wikitext
    • E.g. try using preview on an article like en:Barack_Obama (no doubt there are even bigger articles)
  • How often can we update the preview in a short period of time? Can we overload it?
• Regression Test
  • Existing related features (like show preview, publish, and show changes) still works/compliments new feature?
• State-based testing
  • There are several ways to reload RTP (e.g. shortcut vs. automatic) and several states it can be in (e.g. automatic vs. manual mode) which might affect how it reloads (this script will generate random sequences of RTP reloads)

Test Environments

TBD. Probably beta primarily.

Open Questions

• Will a user be able to use this feature alongside other edit previewing features such as "Show changes"?
• Will they be able to use this feature when resolving edit conflicts?
• What is the deployment/roll out plan to production
• Are there any major infrastructure set ups to do when we roll out to production? I am guessing no, however want to confirm

Release Testing

See /Release_Testing.