Gulp

This is a proposal based on T231891

Generic Unified List Processor (GULP)

This aims to define the capabilities of a to-be-created API to handle generic lists of pages/items.

Definitions

list

a set of revisions for a site

revision

a set of page entries

entry

Consists of a title, a namespace, and optional metadata

site

A wiki

Essentials

• Create a new list
• Add/remove page entries to/from the current revision of a list. This would not create a new revision automatically!
• Create a snapshot, ie, freeze the current revision of a list
• Retrieve a (current or snapshotted) revision of a list

Minimum viable product

• Delete a revision or list (?)
• Import from various sources
  • All sources offered in PagePile
  • Wiki pages
• Export to various places
  • All consumers offered in PagePile
  • Listeria (V2?)
  • Wiki pages
• Combine lists (subset, union, diff, etc.)
• Filter lists

Nice-to-have

• list and/or revision with optional expiration date (automatically updated lists)
  • possibly "temporary" lists with ~1h expiration date?
• maximum number of revisions (delete oldest one if capacity is reached)
• visual interface for pipelines (generate/filter/combine/output) based on lists
• Diff functionality between revisions
  • For the interface (what has changed?)
  • For storage (less space required)
• Meta-data per entry (store/update/remove/query)

Extended version

What is it's not just "page lists", but any (general, of one of pre-defined types) tables?

• One table type would be "page title/page namespace", giving us the above lists.
• Others could be, say, Mix'n'match catalogs ("external ID/url/name/description/instance of")

Technical notes

Storage options for revisions

• sqlite files on disk, a la PagePile (or use PagePile transparently in the background). Works for large lists, especially for combination/filtering of lists.
• Commons Data: namespace (aka ".tab files"). Size limited.
• (MySQL) database. Might eventually outgrow capacity
• Disk-backed object store

Any combination of the above could be used transparently, based on the list (large lists=>sqlite, small lists=>MySQL etc). Storage could even switch between revisions.

Identifiers

• One ID (number) for lists, another for revisions (like MediaWiki)
• One ID (number) for both. Using the list ID when asking for a revision would automatically use the latest revision. Simple but might confuse users.
• Combined ID (String), eg "123.456" or "123/456" (the latter could be useful in URLs; missing revision would automatically use the latest revision)

Data structures

List

ID

Name

Description?

Site

User who created it (special privileges?)

Creation timestamp

Last update timestamp

Last revision

Optional:Maximum number of revisions (more => oldest gets deleted)

Optional:Maximum age of revisions (old ones get deleted)

Optional:Maximum age of list (auto-destruct for temporary lists)

Optional:Source

Revision

ID

Date of creation

entries

Is current or snapshot?

Previous Revision ID

Entry

Page name

Page namespace

Metadata (possibly JSON object for flexibility)