Grants:Project/Stephane/Kiwix/Offline medical/Layman

The below is a glossary / explainer of things we are trying to achieve. Please centralize comments and questions on the Grant's talk page

Indexing, OPDS and OpenSearch support

The basic problem we currently have with Kiwix is that the library.xml we are using is not a standard (remember that Kiwix is old: it was developed as such because there was no standard to work from at the time). What this does basically is a catalog of metadata. But it doesn't just sit there and require a bit of maintenance: while it was develop for a couple of hundred items, now we're dealing with thousands upon thousands of them. Hence the need to get something more robust.

Merge/revamp zimpatch/zimdiff

Basically zimpatch is exactly what is says: a piece of code (a patch) written a while back (2-3 years) during a Google Summer of code. No one ever got around implementing it properly, but we still need it to generate zim files: Zimdiff is important because it is also the first step towards implementing incremental updates: at the moment, people need to re-download the full app (up to 1.2 Gb in English) if they want to have updated articles. When one tries to offer content for people who are (i) offline and (ii) with limited, patchy or expensive internet connections, this is not optimal.

Also, just to be clear: this is a first (necessary) step towards versioning, but we won't have a final version before another couple iterations (but we've got to start somewhere).

Create test bases

The more complex the code, the more complex the testing. We reached the point where we need to build software to test improvements before releasing. We currently have several issues with the latest Kiwix 2.0 android release, which at the moment we can only deal with post hoc. This is ungrateful work that no volunteer has ever volunteered to do.

Support fulltext search

At the moment Kiwix only allows search on article names. We want to implement search on article content.

Add content seek for video streaming

Zimlib and Kiwixserve need to process a full video before being able to navigate it. On desktop, this is partially compensated by the fact that computers are powerful enough to manage the whole thing as one chunk. On Android, that's more of an issue, particularly when one considers that a fair amount of our users are not exactly using the latest technology.

Finish the one pass zim file generation, incl. adding metadata to zim

Zim generation is/was a two-step process, which is by definition a bit resource intensive: we need to improve this if we want to automate and simplify updates and content generation.

Implement the last kiwix library improvement in the iOS app;

It's all in the title. Necessary step to get the iOS app going.

Add search in zim file; Add a multi-zim research

We recently integrated indexes directly into their zim files. There are, however, constraints in terms of quality of the search that can currently be performed (see above, e.g. fulltext search) and the size of said index. We're using xapian and want to improve/fine-tune it: this is another necessary improvement if we want to cater to low-end devices (20% of our users are on Android 4.4 or older, and that's only for those we could track).

Industrialize the releasing process for zim and custom apps

Right now zim and app generation is a fairly labor-intensive (and manual) endeavour: with the number of apps growing (7 now, probably 10-15 by mid-2017), we want to start automating this process so that users can get regular content updates.