OpenStreetMap Architecture

We are currently evaluating several approaches to handling OSM data, and each approach has advantages and disadvantages. This document is tracking each option so that we can evaluate the best path forward.

Required Functionality

As it stands, we need a system that accepts many small OSM changesets received by OpenMapKit Server. There are many OSM files, and these files are stored on the file system.

This system must also be able to ingest source OSM data from a given HTTP endpoint. This format needs to be OSM XML, and OSM PBF would be a bonus.

We must be able to filter our edited OSM data by various parameters. This ability will allow us to gather and validate data to submit for a given user, and the ability to gather certain subsets of the edited data will be very useful in the workflow of submitting finally submitting edits to the OSM Editing API.

We need to visualize our OSM data on a map. Two approaches can be taken.

1. We will render raster or vector tiles of our data using Mapnik or an equivalent tool. This approach will allow us to apply cartographic styling and have tiles appropriate for final visualization.
2. We will only serve OSM XML though a REST API. This approach will not create tiles, but it will be sufficient to provide data needed by the iD editor as well as the basic vector rendering functionality of OpenMapKit Android.

Regardless of the approach, we must serve OSM XML with a REST API compatable with the OSM 0.6 Editing API.

Database / Datastore

We have explored 3 options for handling our OSM data.

Flat OSM XML Files

Currently, Ona and OpenMapKit Server provide an aggregation of OSM edits for a given form by concatenating all of the small OSM media attachments received by the ODK server. This is only sufficient for handling tag edits of existing OSM elements, because new OSM elements are created by assigning arbitrary negative IDs consistent only within the given OSM file. As these files are read and aggregated together, new negative IDs need to be assigned. This is possible if we parse each OSM file and assign new IDs ad hoc in the aggregation operation.

Pros

• Very simple.
• No database dependency.
• Can still provided the required end result of an aggregation of OSM edits for iD.

Cons

• Very slow to parse and handle hundreds of OSM files repeatedly for each change in the underlying data.
• Filtering OSM data based on parameters is cumbersome.
• Rendering OSM XML directly in Mapnik is not recommended.

Spatialite Database

R&D efforts have focused on using Spatialite as the underlying database. Spatialite is SQLite with spatial extensions. An initial sprint that builds an OSM data model with Spatialite has been completed as OSMSpatialite. This approach was written in C++11 and is geared towards being compatible with AmigoClient. The data model is similar to the data model found in the current OpenMapKit Android.

Pros

• Unified data model that can work on a server as well as Android / iOS.
• Work already started.
• Logical choice for a low resource platform.
• Likely the most performant option.
• Mapnik works with Spatialite as a data source.

Cons

• Requires NodeJS Bindings for the REST API.
• We have not yet tested how well Spatialite works with Mapnik.
• C++11 has a learning curve. All team members may not be comfortable with using this language.
• Macrocosm currently only supports a Postgres backend. Changes would need to be made to also support Spatialite.

PostGIS Database

As a team, we have the most experience working with PostGIS. The master OSM 0.6 Editing API uses Postgres as a backend, and Macrocosm is already built to leverage PostGIS. This is a traditional RDBMS, and it is not built to run on a low resource, embedded system.

Pros

• Standard backend for Mapnik.
• Compatible with Macrocosm.
• Typical choice for OSM backend.
• Osmosis can be used to populate db with source data.

Cons

• Business logic of ingesting edits needs to be fleshed out.
• Less performant for low resource device.
• Cannot be built to also run on Android.
• Macrocosm is meant to replace OSM API, not meant to be a staging server that submits to OSM API.

Currently, our evaluation involving PostGIS is focused around learning about Macrocosm. If we use this tool, we will have to adapt it significantly to meet our needs. It will need to be reimagined for the staging use case. It doesn't look like the model takes into account OSM users, and some important attributes found in the master OSM model are missing.

It would be nice if it was possible to add vectors in an easy way, either by drawing them on the map or by using a gps track. This could be usefull to map roads, fences, etc