Data dumps/2006 notes

Clusters edit

The wikis hosted in our Korean cluster will have a separate host, at http://download-yaseo.wikimedia.org/

Reporting edit

The backup runner script will generate some pretty HTML pages showing status as each file completes, so it should be easier to see what's done, what's in progress, and what failed.

I'm about to code up this part, shouldn't be too hard I hope. :)

File layout edit

This basic layout of file generation is complete in the script:

public/
- dbname/
  - YYYYMMDD/
    - dbname-YYYYMMDD-all-titles-in-ns0.gz
      list of page names for BBC
    - dbname-YYYYMMDD-table.gz
      SQL table dumps
    - dbname-YYYYMMDD-pages-type.xml.bz2
    - dbname-YYYYMMDD-pages-type.xml.7z
      XML page text dumps
    - dbname-YYYYMMDD-abstract.xml.gz
      page extracts for Yahoo

Static URLs edit

There will probably also be a directory with symbolic links for a static URL to whatever the latest version is of each file. Will likely look like this:

public/
- dbname/
  - latest/
    - dbname-all-titles-in-ns0.gz
      list of page names for BBC
    - dbname-table.gz
      SQL table dumps
    - dbname-pages-type.xml.bz2
    - dbname-pages-type.xml.7z
      XML page text dumps
    - dbname-abstract.xml.gz
      page extracts for Yahoo

Images/uploads edit

Not yet included, this may change in near future.