Toolhub/Progress reports/2022-03-18
Report on activities in the Toolhub project for the week ending 2022-03-18.
Release did not go smoothly, but we survive
editWhile attempting to deploy a newer version of Toolhub to Wikimedia's 'staging' cluster, we ran into a fatal error that required a new python library to be installed. The django-prometheus library that Toolhub is using to produce some metrics for production reporting introduced a breaking change in their v2.2.0 release. SemVer (semantic versioning) states that breaking changes are indicated by increasing the major number (indicating high risk). If they were actually following SemVer, django-prometheus would have labeled this release v3.0.0. As it was they didn't even bother to list the breaking change in the release notes.
After changing the libraries and successfully deploying into the staging cluster we proceeded to deploy to the 'codfw' and 'eqiad' clusters. The eqiad cluster is the important one for Toolhub. This is where all of our production traffic is actually processed. Unfortunately we found T303889 in this environment and did not understand this was the problem until after we had applied database changes which would be difficult to rollback. At this point https://toolhub.wikimedia.org/ was returning an ugly HTTP 500 error page to all visitors, and had been doing so for nearly an hour.
The most expedient path back to a likely working deployment was to undo the changes from gerrit:770987 and also pin the django-prometheus library to an older version (2.1.0) which worked with the python-memcached that was also known to work with our mcrouter sidecar. Preparing the git commit to rollback gerrit:770987 and add pinning took more time than Bryan hoped. Poetry had some failures in trying to install python-memcached, but eventually worked. Approximately 1.5 hours after initially deploying the broken version of Toolhub to the eqiad cluster a working version was deployed.
Production release made
editA number of feature enhancements and bug fixes were deployed to the production https://toolhub.wikimedia.org site on 2022-03-15. This was the first deployment of the production service since 2022-01-05.
Features added:
- Implement text analysis to support stemming
- Implement UI for viewing list revision history, content diffs, and patrolling actions
- Add autocomplete/lookahead search to home page search bar
- Tags and author should link to filtered search queries
- Allow multiple authors in toolinfo.json
- [Tech spike] Experiment with storage implementations for toolinfo annotations
- UI for viewing and searching public lists
- Implement search functionality for published lists
- UI to toggle "featured" status of a published list
Bugs fixed:
- Sortable tables are currently misleading (sorts page, not full dataset)
- Alert dialogs can have close button placed off screen
- Toolhub has duplicate identifiers for GPL family of licenses
- Changing languages does not always change the language across the whole page
- The LineChart component is not reactive
- i18n reactivity of 'this field is required' text not working
- Toolhub URL field doesn't accept Cyrillic characters
- Tool list editing and display has unexpected/non-deterministic ordering for tools in list
- Change GET /api/lists/{list_pk}/revisions/{id}/ to emit full info on tools contained in the list
- Create new serializer for list diff generation
- Unexpected list diff api endpoint behavior when diffing lists of tools
- Structured diff UI assumes all members are scalars or arrays
- Historic revisions of toolinfo records with "author" changes fail after schema changes
- Building "web" container failing during `poetry install` in multiple environments
- Monitor changes to tool/list revisions without using watchers
- Unexpected "isTrusted" value showing up in some author diff output
- Exclude 'featured' field from List diffs
- MyLists.vue can call vuex before user is known
- Add config to disallow extra fields in some json schema objects
- Diffs involving annotations seem to be blowing up consistently
- Toolhub broken in prod by memcached client library change
Technical debt:
- Improve API error handling and display in Vue frontend
- urlRegex frontend validation is fragile
- Upgrade Toolhub ui container from nodejs10 to nodejs12 (or newer)
- Maintenance environment needed for running one-off commands
- Add optimized Elasticsearch field mappings for autocomplete searches
- Add Sphinx extension for copying code snippets in docs to clipboard
- Create a better dev experience for editing docs
- Make all vuex actions that make API requests return a promise
- Improve Javascript test coverage to >=85% and enforce same for future patches
Features Added/In The Works
editTool Lists
edit- Slavina refactored all components/views used to display toollist(s) information into extending from one base component.
- Slavina submitted a patch making it easy to edit and delete a ToolList.
Annotations
edit- Bryan made improvements to the Annotations feature by adding 14 new fields to annotations and connecting it all to elasticsearch.
- Raymond started working on the UI side of the annotations, submitting a patch that refactored the tool edit form so more fields from annotations can be added without making UX worse.
Recent Changes
edit- The Recent Changes feature patch by Raymond aimed at unifying the content moderation and patrolling of all lists and tools on toolhub on a single view has been merged.
- Raymond submitted a couple of patches aimed at limiting the number of requests made while performing certain content moderation and patrolling actions like undoing an edit.
Multi-author support for Toolinfo records
editRaymond and Bryan made improvements to the UI of the multi-author support for toolinfo records feature by submitting and merging a patch that allows for the display of extended author information on tool detail screen.
Bugs Fixed/ Development Improvements
editGuard against missing prior values in revision diffs
editBug fix submitted by Bryan.
Delete list document from search index when list is unpublished
editBug fix submitted by Raymond.