This repository is part of the Pelias project. Pelias is an open-source, open-data geocoder built by Mapzen that also powers Mapzen Search. Our official user documentation is here.

Work In Progress

The polyline importer facilitates importing road network data in to Pelias from a list of polyline encoded line strings.

• NodeJS 4 or newer (the latest in the Node 4 series is currently recommended)
• Elasticsearch 2.3+ (support for version 1.x has been deprecated).

Since this module is just one part of our geocoder, we'd recommend starting with our Vagrant image for quick setup, or our full installation docs to use this module.

$ git clone https://github.com/pelias/polylines.git && cd polylines;
$ npm install

We are still deciding on the best format to publish polyline data for distribution.

Currently there is a single planet-wide road network file which was cut on 8th Jan 2017, download here (1.5GB compressed, 2.3GB uncompressed).

For more information on how the extract was generated, see the wiki article: Generating polylines from Valhalla.

We also have some smaller extracts for testing purposes, a small number were manually cut from pbf for the geographies of our major contributors. See the 'Generating a custom polylines extract from a PBF extract' section below for more info on how you can generate your own extracts:

note: these extracts were generated using a different method from the planet cut above.

• Berlin (1.9MB, 49k roads)
• New York (4.2MB, 102k roads)
• Finland (7.7MB, 100k roads)
• Sweden (5.9MB, 126k roads)
• London (5.6MB, 166k roads)
• Paris (2.9MB, 81k roads)
• San Francisco (1.3MB, 27k roads)
• New Zealand (3.1MB, 52k roads)
• Chicago (3.5MB, 88k roads)

Once you have downloaded and extracted the data you will need to follow the Configuration steps below in order to tell Pelias where they can be found.

If you would like to use a different source of polyline data you might need to tweak the defaults in ./stream/pipeline.js, open an issue if you get stuck.

In order to tell the importer the location of your downloads and environmental settings you will first need to create a ~/pelias.json file.

See the config documentation for details on the structure of this file. Your relevant config info for the polyline module might look something like this:

note: the importer currently only supports a single entry in the files array.

  "imports": {
    "polyline": {
      "datapath": "/data",
      "files": [ "road_network.polylines" ]
    }
  }

Polyline data doesn't have a full administrative hierarchy (ie, country, state, county, etc. names), but it can be calculated using data from Who's on First. See the readme for pelias/wof-admin-lookup for more information. By default, adminLookup is enabled. To disable, set imports.adminLookup.enabled to false in Pelias config.

Note: Admin lookup requires loading around 5GB of data into memory.

This will start the import process, it will take around 30 seconds to prime it's in-memory data and then you should see regular debugging output in the terminal.

$ PELIAS_CONFIG=<path_to_config_json> npm start

You can use the CLI tool to run imports and for debugging purposes:

note: by default the cli tool will read from stdin and write to stdout.

$ node ./bin/cli.js --help
Usage: cli.js [options]
Options:

  --file           read from file instead of stdin
  --config         read filename from pelias config (overrides --file)
  --pretty         indent output (stdout only)
  --db             save to elasticsearch instead of printing to stdout

Run a 'dry-run' of the import process:

node ./bin/cli.js --config --pretty

Import a specific file to elasticsearch:

node ./bin/cli.js --file=/tmp/myfile.polylines --db

You can generate a custom polylines extract using this OSM PBF tool.

Note: golang 1.6+ is required, please ensure this is correctly installed before continuing.

$ go version
go version go1.6.2 linux/amd64

$ go get github.com/missinglink/pbf

$ pbf --help

$ wget https://s3.amazonaws.com/metro-extracts.mapzen.com/chicago_illinois.osm.pbf

$ pbf streets chicago_illinois.osm.pbf | head

avnfoAzllifD~NsZ?ePL}^rAetD^qpAT}i@Fmh@l@cdBLo]?sFHartzell Street
gjpfoAnq}jfD?}IOqSe@oRu@qRcAsQcAsFfCyBhCiBpBgDdAgDrAeEr@eF\sFEuEe@cF}@eEsAiCiBgD?uYjA]l@m@l@kBDyAEkBu@{@u@m@sAOkANm@z@e@lAMhB\jAd@zAd@l@zA\Hartzell Street
_bnfoAps~gfDG||AkAzzD}@nmDm@npBe@rFm@tEGz`@Ung@Gnh@U|h@O`g@Gnh@Mbf@O~]O`GHartzell Street
uidfnApqsqeDvChWrApRV`SeApQaBbQqBhWW`St@pQjAlUMorningside Avenue
elvrnA`helgDUsp@m@o{A?oIMorningside Avenue
w{vznAjm~dgDwHkAmo@yMg|@aRMorningside Avenue
sa}znApje{fDiB_fBgD}eDsAiuAmAwuAyAwvAsAgvAMorningside Avenue
{cd}nAp~}dgDmgBbGal@vDgc@bFaS`H}NnH_c@jWoSdOyWtO{PpH{U~HiXrFy[fDmZl@gI_@_IyAyGyC}J{KeEmIMorningside Avenue
sg|znA|w|dgDg_D]oeB]wpB_@Morningside Avenue
eko{nAts|dgDtsEj@Morningside Avenue

If you have any issues getting set up or the documentation is missing something, please open an issue here: https://github.com/pelias/polylines/issues

Please fork and pull request against upstream master on a feature branch.

Pretty please; provide unit tests and script fixtures in the test directory.

A .jshintrc file is provided which contains a linting config, usually your text editor will understand this config and give you inline hints on code style and readability.

These settings are strictly enforced when you do a git commit, you can execute git commit at any time to run the linter against your code.

$ npm test

Travis tests every change against Node.js 4and6`.