Grants:IEG/Wikimaps Atlas/Timeline

This project is funded by an Individual Engagement Grant

Timeline for Wikimaps Atlas

Timeline	Date
Detail tasks and responsibilities for Yug and Planemad for the entire project	2014/03/25
Create project website and placeholder frontend	2014/04/28
Recreating locator and topography maps through automation scripts (makefiles)	2014/04/7
Creation of master postgres GIS database importing naturalearth and openstreetmap vectors	2014/04/14
Automatic conversion of postgres data to topojson file by admin region	2014/04/24
Create labs account with postgres privileges	2014/05/12
Automation of topographic+admin maps using bbox options	2014/05/14
Public beta of Wikimaps Atlas website to browse and download generated maps	2014/06/30
Move automation scripts, maps and website to Labs server	2014/07/14
Website ready for Wikimania showcase	2014/07/30
Wikiatlas API to get topojson files of requested admin area	2014/11/10
D3 frontend with country dropdown selector and instant map rendering with Level 2 admin boundaries	2014/11/12
Integration of makefile map generator to render raster topographic data	2014/11/15
Working world map navigator with SVG download capabilities. Move Code to Tools server	2014/11/18
Implement D3 transitions for map styling for varying scale	2014/11/20
Implement API for OpenStreetMap>topojson requests	2014/11/25
Frontend UI improvements for automatic legend and title generation based on map data. Capabilities to show/hide map data layers	2014/11/28
Bug fixes. Public announcement of release	2014/11/30
Start project documentation. Inititate public discussion on future work.	2014/12/1
Build repository of generated locator map SVGs. Engage Commons/Wiki communities to replace existing locator maps.	2014/12/15
Submit final project report	2014/12/31

Monthly updates #2014

Please prepare a brief project update each month, in a format of your choice, to share progress and learnings with the community along the way. Submit the link below as you complete each update.

March 2014

• The project work began on Yug's arrival in India on 23rd.
• The initial days consisted on syncing our workplans, evaluating the goals to achieve and how we would do it
• Although the main focus of the project is automation scripts, we decided a good frontend to view the generated maps will greatly enhance the publicity. The website with a placeholder map browser was hosted on http://wikimapsatlas.github.io/

April 2014

• The current mapmaking workflow by wikimedia cartographers was understood and split into seperate tasks for automation
• Every task was then associated with a relevant library and converted into a command
• The commands were then chained into a makefile to enable generation of a topographic map from raw srtm data using bbox option [1]
• Using a combination of image processing using imagemagick, superior hillshade results was achieved with enhanced contrast [2]
• Postgres tables were set up to mass import open geo data, currently from openstreetmap and naturalearthdata.

May 2014

• The team attended the Zurich hackathon where the following was done:
  • Meeting of everyone interested in maps and wikimedia, there was a lot discussed on all the different projects, ideas and how to cooperate
  • One of the decisions was that a map namespace was reuired to handle map related views
  • The team successfully implemented automation of administrative and topographic maps using makefiles[3]. This marks a major milestone for the project.
  • A labs account with postgres access was created to migrate the system to WMF server
• The first topographic+admin map was automated using image composting [4]

June 2014

• A repository of administrative topojson was created [5]
• Administrative area level 1 and zoom to level 2 is demoed [6]

July 2014

Return to the project -- Pre-Wikimania

• Dataviz:
  • Fixing bitmap embedding
  • Terminal to NodeJS jointure : Done
  • Module topography.makefile for vector topography : Done
  • Labellings CSS improvement : Done
• Website :
  • webpage to search/browser for available data
  • webpage to pass parameter to script (basic)
    • Form => new string for new Terminal command => run on terminal
    • Contact: yuvipanda@gmail.com
• Community
  • Books?
  • Organize Edouard L. & Jakub K. coming to Wikimania for hackathon  : Done
• Documentation:
  • Makefiles
  • Webpage (Map generator)

Wikimania

• Topojson integration into MediaWiki -- Vega.js -- External: Lopez Edouard  : Done (tested)
• Sharemaps alignment to WP Style -- Sharemap.com -- External: Jakub : Done (local)
• Dataviz expension -- Hugo : Done
• Website rush -- Arun
• Multilanguage maps -- Arun
• Communcating about project helped by documentations -- Hugo & Arun

August 2014

• Data : completion of the data creation chain, all modules working !
  • Module water.makefile done, it adds lakes, rivers from NaturalEarthData.
  • Module data.makefile done, it download the relevant GIS file.
  • Modules improvement: Collecting output files in common place for easier pick up.
• Dataviz: improvement of location_map with cities points and cities labels prototype
• Dataviz: creation of automatic locator_globe_map prototype
• Dataviz: Test OSM-JSON API to explore OSM data exploitation.
• Dataviz: clean up Wikidata code for multilingual maps (function to finish, see trello)
• Dataviz: start to modularize the code into a wikiatlas.js library.
• Conference: decline FOSS4G 2014 invitation due to busy professional schedules.

November 2014

• Tool-labs migration engaged.
  • Finding external support : Done (Yuvipanda)
  • Training to tool-lab management for both member : Done

• Static website UI migration :
  • repository: https://github.com/WikimapsAtlas/WikimapsAtlas.github.io Done
  • "hello world" access : http://tools.wmflabs.org/wikiatlas2014/ Done

• Back-end scripts Migration-synchronisation : Done
  • repository : https://github.com/WikimapsAtlas/make-modules/ Done
  • Dependencies installation and associated troubles-shooting : Done
  • Installation and testing : Done
  • Installation process refactoring : Done
  • README.md : Done

• Back-end database design and migration : ongoing
  • repository: https://github.com/WikimapsAtlas/make-data-json
  • chosen technologies: Postgres
  • Huge refactoring : ongoing
  • Local server-style testing: ongoing

• Website server-query connector: underway
  • chosen technologies: socket.io
  • repository: <to come>

December 2014

• Back-end scripts:
  • make-modules/01_reliefs [raster]: Done
    • georeferencing compression R&D, improvement Done (H)
    • all relevant file eventually moved to ../output/ Done (H)
  • general review & improvement
    • data download process centralized Done (A)
    • iso_2 transition started Done (A)
    • path to source updated Done (A)
    • minor fix unzips target Done (H)
    • merge escaped_item and item as unnecessary for iso codes Done (A)
    • fix path of base64 source Done (A)
    • [tool lab kept up to date] Done (H)
    • add base64 for multiple files from ./01 Done (H)
    • add HEIGHT parameters, extracted via identify | cut | sed Done (H)
    • add: passing HEIGHT,WIDHT variables to nodejs, output right size svg Done (H)
  • Loop
    • Create data for all countries Done
    • Python integration of make-modules Done

Monthly updates #2015

January 2015

In January, the back end should be improved (but not over polished!), D3js & front, web dev & fluidity in the workflow should be the main focus. Paperworks: the community should be reached out before & for announcement (Important!) while documents should be provided.

• Back-end scripts:
  • push for valid svg!
  • d3js code to plug into final nodejs module
  • make: administrative module revamp Done (H)
    • switch places filter from "in {country}" to "in {area}" Done (H)
    • switch admin_1 filter from "in {country}" to "in {area}" Done (H)
    • output polished topojson with convenient structure for web usages link Done (H)
    • pass and stores relevant layers: admin_0, admin_1, disputed, places. Done (H)
    • pass and stores relevant properties : id,name,L0,L1,note Done (H)
  • make: vector hillshade
  • make: water vector topography (mirror of landmass vector topography)
  • data: add our best SRTM rasters, merge the tiles.
  • d3js improvement are needed. Color ramp, etc.
• Front-end d3js:
  • svg height auto-adjust regardless of map projection link Done (H)
    • svg autofocus now 100% automatized
  • Migrate from styles from css to pure js link Done (H)
    • required for stand alone svg.
  • Migrate "download button" into an easy to use function. + Fix css bug. Done (H)
  • Add "download button" clipping on vieabox, for lighter svg. Working. Done (H)
  • Debug and implement stroke-dasharray for international borders link Done (H)
  • Add hashing required for disputed area link Done (H)
  • Note: Location maps d3js now 100% ok. visual end-result

May 2015

• Finish the make-module here (historic) Done (H)
• Finish the make-module documentation here Done (H)
• Server serving existing files. Structure here and here (A.)

May 2015 - February 2016

	Pause : Wikiatlas in pause due to the delay affecting personal commitments

Monthly updates #2016

February - May 2016

Wikimaps Atlas - Final push

Target 1: Generate a repository of world maps for admin-0, admin-1 (Hugo)

• February 25: Retry make_module Done (H)
• Check paths and filenames Done (H.)
• Fix/escape complex country names (H.) <== currently crashing on this one.
• April 10: Run the script for all countries (H.)
• April 17: Upload files to Labs server (H.)

Target 2: Create a gallery/download website (A.)

• February 25: Finalize directory structure for map output Done (A.+H.)
• March 3: Write a Jekyll site generator to build individual map gallery pages for every admin-0 and admin-1 territory Done (A.)
• April 17: Upload Jekyll site to Labs server to serve the generated maps through the gallery website (A.)

Target 3: Documentation (Hugo, Arun)

• March 17: Polish up documentation of the make scripts for easy customization by a user experienced with command line Done (A.+H.)
• April 24: Document the Jekyll site generator and how it can be updated with an updated run of the maps (A.)

Target 4: Communications (H.+A.)

• Every week: Twitter progress update and weekly goals <== must do this more.
• April 24: Planet blog post outlining the features of the project and how it can be used by the community
• May 8: Call for review.
• May 31: 'Final report (H.+A.)

Extension request

New end date

Delivery: November 30th.

(Expected: Nov.15th, asked: November 30th.)

Rationale

R&D for the project have been 1.5 times longer than expected, and professional occupations have been conflicting with Wikiatlas. As of late August, the project advancement is as follow :

• Back end scripts (data) : 70% of the workload, done at 90%
• Rendering (dataviz): 10% of the workload, done at 80%
• Website UI & server : 10% of the workload, done at 0%.
• Automatic push to common: 10% of the workload, done at 0%.

The team's human resources are available about 5 more weeks between Sept. 15th and Nov. 15th to complete the project. Also, this extension would be much welcome. Yug (talk) 23:03, 24 August 2014 (UTC)

Hi, Yug. This extension is approved. I can see that the team has hit some roadblocks with other commitments, and also that you guys have made great progress so far! More time is an easy give, happy to see it done well (particularly not rushing automating pushes to Commons, etc!). Your new end-date is now November 30th, with your final report due 30 days afterwards (by December 30th). Feel free to hand in your final report in November as well, of course! Cheers, Siko (WMF) (talk) 00:10, 3 September 2014 (UTC)

Extension request

New end date

• February 15th. Yug (talk) 11:13, 21 January 2015 (UTC)

Rationale (2015)

• Need delay to deliver final product. Yug (talk) 11:13, 21 January 2015 (UTC)

This request is approved. Cheers, Siko (WMF) (talk) 17:14, 11 February 2015 (UTC)

Extension request

New end date (2016)

• March 31st. Yug (talk) 19:05, 17 February 2016 (UTC) Planemad (talk) 19:35, 17 February 2016 (UTC)

Rationale

• The previous difficulty due to technical challenges of a highly ambitious goal resulted in large delays to the project, and thus impacting the personal commitments of both members. We have both been able to take care of outstanding commitments and now in a position to dedicate time to revive the project and package this as a usable product for the community. Yug (talk) 19:04, 17 February 2016 (UTC) Planemad (talk) 19:35, 17 February 2016 (UTC)

Thank you for updating your project end date, Yug and Planemad. I'm approving this request for a March 31, 2016 end date. Please update your Timeline table so that it reflects your remaining milestones. The date for the last row (submission of your Final Report) should also be updated. Looking forwarding to hearing about your outcomes! --Marti (WMF) (talk) 00:11, 2 March 2016 (UTC)

Is your final report due but you need more time?

Extension request

New end date (2016 b)

• June 30th. Yug (talk) 21:05, 30 March 2016 (UTC)

Rationale

• Reviving the project and some progresses have been done, yet further steps are needed to complete, install on tool-labs and ask volunteers to review our work. Yug (talk) 21:05, 30 March 2016 (UTC)

Extension request

New end date

September 4th 2016 Planemad (talk) 18:28, 30 June 2016 (UTC)

I'm approving this extension request for a September 4 project end date. Your Final Report is due 30 days following the project end date, on October 4. I've emailed you about setting up a check-in to talk about wrapping up. Cheers, Marti (WMF) (talk) 04:55, 20 September 2016 (UTC)

Rationale

Sadly Yug and I have not been able to resume the project as expected with our personal commitments. This was one of our fears when we missed the first deadline, and it has only got increasingly hard to get back on track by treating it as a weekend project. We're going to request for a final extension to make this work, given that I will have week in August to devote to this and make a significant push towards our intended goals. Our blocker has been failing dependencies due to their updates over the last year. Its going to be a uphill effort, but we feel its our responsibility to close the project in a state that is worthy of the effort already invested, and we will have to make the best use of this time to do that Planemad (talk)