digidem / discussion Goto Github PK

Offline is important for a lot of our tools, and I've been experimenting with digidem/html5-image-cache caching images dynamically in local storage (although I recently found that a lot has already been done here and here)

For data that is dynamically loaded via xmlHttpRequest that is another challenge. To date I had approached it starting to write a custom Backbone sync adapter with js-git, but this is a complex approach. Importantly, what about external libraries that load data, like vector tiles in Mapbox-gl-js (a lightweight way to provide image tiles) or deforestation API data from CartoDB Torque?

For much of this data all we need is the ability to cache the data and potentially have the ability to "side-load" it via a USB rather than an internet connection.

My idea is to write a wrapper for XmlHttpRequest that works similarly to the image cache script: when called the first time it will save to a cache, when called a second time it will load from the cache. That way it could be transparent to any libraries like mapbox or cartodb that we use. We save all requests with a hash of the url, so we always get the same data at the same URL.

Both Mapbox-gl-js and torque use browserify, so we could write a transform to replace XmlHttpRequest with our own version, wrapping the original version and adding a caching layer.

We could also use this for our own apps, whether we use backbone or angular. We just agree that only read offline matters, then just use, for example, the standard Backbone.sync but cache at the XHR layer rather than the Backbone.sync layer.

To populate the cache, we would either need to visit everything with a connection, or call the relevant functions in each library to download the tile we want.

mbtiles would be a good format for side-loading. We would need to write a library to download the needed tiles and save to mbtiles, and have a standard way to encode the original urls into the mbtiles package. That way we could populate the cache of the webapp with the correct url-hashes as the ids, so that as far as the xmlHttpRequest wrapper is concerned, the cache has come from a previous request.

Thoughts on this? Task list:

• wrapper to XmlHttpRequest that mimics the original functionality but adds a caching layer.
• Browserify transform to replace XmlHttpRequest with our custom version
• "downloaders" for whatever tools we use.
• Library to "side-load" data from a file and populate teh XHR cache.

Thoughts?

Crazy idea, exploring feasibility. Could enable slow-but-cheap connections to remote locations.

cc @maxogden