Hey everyone,

Is there any tilerator's benchmark? How long it takes to generate the whole planet (0-17 zoom)?

So I am testing the basic add job call to Tilerator with this command: curl -X POST 'http://localhost:16533/add?generatorId=gen&storageId=v2&zoom=0' and in response I get an error:

  "stack": "tilerator: tiles parameter not set
            at new Job (/usr/local/lib/node_modules/tilerator/lib/Job.js:125:15)

To me, it looks like the default values for idxFrom and idxBefore are not getting registered properly. When Tilerator processes the job it goes through this sequence: https://github.com/kartotherian/tilerator/blob/master/lib/Job.js#L64-L73 ... where it should set the tiles param from idxFrom & idxBefore, but it never gets the chance because the condition never passes. In checkType (https://github.com/kartotherian/kartotherian-core/blob/master/lib/core.js#L330-L342) it looks like it returns false after it sets the default value of the field... So I guess we need to either return true there or not return and allow checkType to continue to check the default value. What do ya'll think?

After 'fixing' (and getting rid of) the test for content encoding, the splicerTest unit tests now fail on Error: Unexpected layer tag #0

This is coming from splicerTest.js line #59 where it seems something inside the data sent to the tileCodec.decodeTile(result.data); leads to a layer tag 0, which is unexpected (see https://github.com/kartotherian/babel/blob/master/lib/tileCodec.js#L60 )

The full stack trace for one of the failures:

9) Tag recombination pick missing:
     Error: Unexpected layer tag #0
      at Tile.Layer._readField (lib/tileCodec.js:123:24)
      at Object.readFields (node_modules/pbf/index.js:39:13)
      at Object.Tile.Layer.read (lib/tileCodec.js:108:16)
      at Tile._readField (lib/tileCodec.js:27:36)
      at Object.readFields (node_modules/pbf/index.js:39:13)
      at Object.Tile.read (lib/tileCodec.js:22:16)
      at Object.decodeTile (lib/tileCodec.js:7:21)
      at babel.then.then.result (test/splicerTest.js:59:37)
      at Gzip.onEnd (zlib.js:227:5)
      at endReadableNT (_stream_readable.js:974:12)
      at _combinedTickCallback (internal/process/next_tick.js:80:11)
      at process._tickCallback (internal/process/next_tick.js:104:9)
  From previous event:
      at test (test/splicerTest.js:50:11)
      at Context.it (test/splicerTest.js:257:30)

I can't make sense of the data or why and how the tag is sent incorrectly. Is this a known bug? Is this indicating a deeper issue with the tile codec code? It's hard to test since babel isn't really used yet, as far as I can see, so I can't tell if the test itself is sending bad data (the code is correct) or if the test is good but its failure indicates something is wrong with the tile codec reading method (the code is broken)

From @benstadin on October 21, 2015 19:21

Is there a chance that mapbox-gl-js will return? I saw that it was included in earlier versions, wasn't it working well?

~Ben

Copied from original issue: kartotherian/kartotherian-server#1

Environment

• Node version: v8.16.0
• Lerna version: v3.14.1
• OS: Mac 10.14.4
• Mapnik version: 3.0.22 (use brew install mapnik to install)

To install the dependencies for all modules, I run the following from the project root, but it has something wrong.

lerna bootstrap --hoist --nohoist mapnik --nohoist libxmljs
lerna notice cli v3.14.1
lerna info versioning independent
lerna ERR! EWORKSPACES --hoist is not supported with --npm-client=yarn, use yarn workspaces instead
lerna ERR! EWORKSPACES A guide is available at https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/

And I try to run another command yarn install from the project root, but it still does not work.

$ yarn install                         yarn install v1.16.0
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
warning [email protected]: The engine "firefox" appears to be invalid.
warning [email protected]: The engine "chrome" appears to be invalid.
[3/4] 🔗  Linking dependencies...
warning " > @kartotherian/[email protected]" has unmet peer dependency "mapnik@~3.7.0".
warning " > @kartotherian/[email protected]" has unmet peer dependency "mapnik@^3.7.0".
warning " > @kartotherian/[email protected]" has unmet peer dependency "mapnik@~3.7.0".
warning "workspace-aggregator-1506f268-a74c-4387-90bc-dd4623300415 > @kartotherian/babel > [email protected]" has unmet peer dependency "eslint@^4.9.0".
warning "workspace-aggregator-1506f268-a74c-4387-90bc-dd4623300415 > @kartotherian/babel > [email protected]" has unmet peer dependency "eslint-plugin-import@^2.7.0".
warning "workspace-aggregator-1506f268-a74c-4387-90bc-dd4623300415 > kartotherian > [email protected]" has unmet peer dependency "@mapbox/tilelive-vector@>=2.0.0".
warning "workspace-aggregator-1506f268-a74c-4387-90bc-dd4623300415 > @kartotherian/server > [email protected]" has unmet peer dependency "[email protected] - 5.x".
[4/4] 🔨  Building fresh packages...
[7/10] ⠠ heapdump
[2/10] ⠠ mapnik
[3/10] ⠠ mapnik
[6/10] ⠠ dtrace-provider
error ./kartotherian/node_modules/canvas: Command failed.
Exit code: 1
Command: node-gyp rebuild
Arguments: 
Directory: ./kartotherian/node_modules/canvas
Output:
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | darwin | x64
gyp http GET https://nodejs.org/download/release/v8.16.0/node-v8.16.0-headers.tar.gz
gyp http 200 https://nodejs.org/download/release/v8.16.0/node-v8.16.0-headers.tar.gz
gyp http GET https://nodejs.org/download/release/v8.16.0/SHASUMS256.txt
gyp http 200 https://nodejs.org/download/release/v8.16.0/SHASUMS256.txt
gyp info spawn /usr/local/bin/python2
gyp info spawn args [ './kartotherian/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   './kartotherian/node_modules/canvas/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   './kartotherian/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '~/.node-gyp/8.16.0/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=~/.node-gyp/8.16.0',
gyp info spawn args   '-Dnode_gyp_dir=./kartotherian/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=~/.node-gyp/8.16.0/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=./kartotherian/node_modules/canvas',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libffi', required by 'gobject-2.0', not found
gyp: Call to './util/has_lib.sh freetype' returned exit status 0 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (./kartotherian/node_modules/node-gyp/lib/configure.js:345:16)
gyp ERR! stack     at emitTwo (events.js:126:13)
gyp ERR! stack     at ChildProcess.emit (events.js:214:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Darwin 18.5.0
gyp ERR! command "~/.nvm/versions/node/v8.16.0/bin/node" "./kartotherian/node_modules/.bin/node-gyp" "rebuild"
gyp ERR! cwd ./kartotherian/node_modules/canvas

Hi there,

We noticed that your repo has a medium severity vulnerability:

• Remote Memory Exposure
• Affected package: request <2.68.0 >2.2.5

Here is the test report for this repo.
If you’d like to fix this vulnerability, Snyk lets you generate a pull request that recommends the best upgrade path - there’s a link to fix this vulnerability on the test report.

Stay secure :-)
Snyk Community

Nothing's been happening here for 2 years, maybe it's time to admit the obvious and close the project to avoid people wasting their time on considering it for their usage? With Dependabot notifications piling, nobody knows if the thing still works with recent Node and is safe to use...

Hi,
Unless I'm mistaken, the https://www.wikidata.org/wiki/Q25975588 entry for kartotherian that was added today does not meet the notability requirements.
Did I miss something ?
Cheers

https://github.com/kartotherian/tilerator-jobprocessor/blob/master/lib/fileParser.js#L116 will hang on stdin if the list if empty.

Related-To: https://phabricator.wikimedia.org/T145534#2987786

Hi, I am following the readme to install kartotherian, but I found some minor issues in the installation steps:

• no need to add dependencies since they seem to be already included in package.json
• no need to add "@kartotherian/tilelive-http", to the registerSourceLibs section (already there)
• System dependencies are not installed automatically.
  I figured out I needed to install on my debian 9 x64:
  • pkg-config
  • libcairo2-dev
  • libjpeg-dev
  • libgif-dev
  • libmapnik-dev
    (only after installing I found the actual dependencies in the package.json file)

Hello,

The tiles are not cached so they have to be reloaded everytime.
It would be useful to have an Expires header.

I don't know if this is actually the right Github to report this, but looking at the fallbacks.json file included here, I noticed that West Frisian, the second official language spoken in the Friesland province of the Netherlands, is missing:

https://en.wikipedia.org/wiki/Frisian_languages

I think a "fry":"nl" needs to be added. The other two Frisian languages (North and Saterland Frisian - frr and stq) are currently listed.

I was thinking about storing the rendered PNG's in cassandra, but it seems that this is hard coded to store PBF's. Is that the only way you would recommend doing it?

Hey all, I am not sure this is the best place to post this question, if you know of someone I should rather talk to or somewhere I should rather post this, feel free to let me know.

So I am trying to use tilerator to generate map tiles z=0...15 for a map tile service. I just upgraded my hardware & it seems pretty good for the job. Planet OSM takes around 14 hours to import. Tilerator is chugging away now. It burned through the first 12 zoom levels at lightning speed. It took maybe 4 days to finish z=14, & now it is slowly working on z=15. My question is what does ya'lls production process look like @ wikimaps? How long does it take you guys to generate z=0...15 & what kind of hardware are you using? Also how are updates handled with tilerator? Is there a way to update only tiles that have changed based on OSM planet diffs? @ my current rate, it looks like it will take over two weeks to complete all the tiles... this seems like a big time investment, but I guess if I can update individual tiles over time & never lose my database it isn't too big of a deal as a one time investment. Advice most welcomed. Cheers & thankyou

Using this command: curl -X POST 'http://localhost:16533/add?generatorId=gen&storageId=v2&zoom=0&fromZoom=0&beforeZoom=16&checkZoom=-1&parts=12' to add a job to Tilerator I get an error (message in title). Stack trace says it fails here: https://github.com/kartotherian/kartotherian-core/blob/286670b135433ae4917c72f71157efa28b1284e0/lib/core.js#L408-L411. It appears to be failing on the checkZoom filter param, '-1' doesn't seem to be passing as a valid zoom. Relevant part of stack trace:

"stack": "tilerator: Invalid zoom param - an integer zoom value was expected\n 
         at Object.checkType (/usr/local/lib/node_modules/tilerator/node_modules/kartotherian-core/lib/core.js:410:23)\n
         at Job.<anonymous> (/usr/local/lib/node_modules/tilerator/lib/Job.js:96:22)

Kartotherian has outgrown its current configuration/modular system, and needs to be rethought. Existing configuration example. Ideally, we may be able to engage larger tilelive community to consider revising the existing interface towards a more generic Promise-based one.

Requirements

• Support existing tilelive ecosystem. Each tilelive module is instantiated with a URI, whose protocol identifies which constructor to use. The modules register their protocols with the common tilelive registry. Each instance has a getTile(z,x,y,callback) function that returns [data, headers] via callback.
• Tilelive interface only passes 3 params: zoom,x,y, but often more context is needed (e.g. user's language, license key, ...). The new interface should have a more generic <Promise<object>> get(<object>). If the module has old interface, the get() should be injected. If the module only supports the new interface, the old functions should be injected so that it can be used with other tilelive code.
• Some modules do not have a tilelive interface, but can consume it. For example, a snapshot module can combine multiple tiles into a single image.
• Module loader should be able to mix and match modules via configuration. Example configurations:
  • configure renderer (moduleA) with data coming from storage (moduleB)
  • same as above, but moduleB is a demultiplexer, which takes tiles from multiple different modules
  • a geodata processing module that runs some queries directly on Postgress DB and produces geojson
  • an image snapshot service module that uses a tile module to get data, and geodata module to create overlays

Implementation Ideas

• Match every existing tilelive function with an Async version:
  getTile(z, x, y, callback) → <Promise<object>> getTileAsync(object), with the assumption that object parameter will have z, x, y values. The loader will inject a wrapper function if either getTile or getTileAsync is missing.

YAML config file structure

• Tile requests: src, z, x, y, [scale], format
  /:src([A-Za-z][-A-Za-z0-9_]*)/:z(\\d+)/:x(\\d+)/:y(\\d+):scale(@[\\.\\d]+x)?.:format([\\w]+)
• Geodata requests
  /geoshape?..., /geoline?..., /shape?...
• Snapshot requests: src, zoom, lat, lon, w, h, [scale], format, [query params]. May use tile and geodata services.
  /img/:src([A-Za-z][-A-Za-z0-9_]*),:zoom(a|\\d+),:lat(a|[-\\d\\.]+),:lon(a|[-\\d\\.]+),:w(\\d+)x:h(\\d+):scale(@[\\.\\d]+x)?.:format([\\w]+)

routes:
  # List of route objects
  - route:
    # The route param is a string or a list of routes as accepted by express
    - "/img/:src([A-Za-z][-A-Za-z0-9_]*),:zoom(a|\\d+),:lat(a|[-\\d\\.]+),:lon(a|[-\\d\\.]+),:w(\\d+)x:h(\\d+):scale(@[\\.\\d]+x)?.:format([\\w]+)?"
    - "/:src([A-Za-z][-A-Za-z0-9_]*)/:z(\\d+)/:x(\\d+)/:y(\\d+):scale(@[\\.\\d]+x)?.:format([\\w]+)"
    formats: [png,json,headers,svg,jpeg]
    scales: [1.3, 1.5, 2, 2.6, 3]
    # Source could be either an ID, or the object describing the source
    source: gen

sources: # a path to a file, or an object with source configurations
  gen:
    protocol: bridge
    xml: "/path/to/file.xml"
    params:
      # Just as before, each param could be a string/number/boolean,
      # or an object with a single key, e.g. "ref" for a source ID, or "source" for in-place source initialization
      backend:
        source:
          ...

variables: # a path to a file, or an object with variables
  varA: value

hi,
( this is most-likely not a bug, but a head-scratching case. i don't get it...)
( also, this points to the "production" version maps.wikimedia.org -- as used in wikidata.org .. )

working on
https://www.wikidata.org/wiki/Q183104
the geo shows this map
https://maps.wikimedia.org/img/osm-intl,13,47.238055555556,9.5983333333334,310x180.png?lang=en

The name of the city shows "Sogn Pieder", but should be "Feldkirch"

this seems to be the "räteromanisch" ( = a language) name of the city. ( https://de.wikipedia.org/wiki/Feldkirch )
This makes no sense. Räterromanisch is a small, valid language in Switzerland (nearby), but no official one in Austria.

For me this looks like: at as some-point-in-time this was as-is in the data.
With "current" i cant re-create this behavior. ( i can also switch the "?lang=en" to "de", "fr".. nothing changes )

Maybe some "very stable" server side caching issue ?

.ka7

Greetings everyone,

I have a strange problem when generation job finishes in less 1 second.

Example POST request:

curl -X POST "http://HOSTNAME/add?generatorId=substbasemap&storageId=basemap&zoom=11&fromZoom=11&beforeZoom=12&parts=10&deleteEmpty=true&keepJob=true&checkZoom=-1"

What is the possible reason?

@nyurik, any objections to if I tag 0.0.31 later today?

Hi,

I tried to use yarn to install JS deps and run kartotherian. I am running node v8.14.0, yarn v1.12.3 on Ubuntu 18.04 LTS. Everything runs fine when using npm.

When running yarn install, no error is shown. However, when starting kartotherian (with the config.external.yaml file), I get the following trace:

$ yarn run start
yarn run v1.12.3
$ service-runner
[2019-01-04T13:36:58.177Z]  INFO: kartotherian/3275 on ubuntu-bionic: Loading variables from the config file (levelPath=info)
[2019-01-04T13:36:58.184Z]  INFO: kartotherian/3275 on ubuntu-bionic: Loading sources from /tmp/kartotherian/sources.external.yaml (levelPath=info)
[2019-01-04T13:36:59.026Z] FATAL: kartotherian/3275 on ubuntu-bionic: /tmp/kartotherian/node_modules/@mapbox/blend/node_modules/mapnik/lib/binding/mapnik.node: undefined symbol: _ZN6mapnik19MAPNIK_LONGLAT_PROJE
    Error: /tmp/kartotherian/node_modules/@mapbox/blend/node_modules/mapnik/lib/binding/mapnik.node: undefined symbol: _ZN6mapnik19MAPNIK_LONGLAT_PROJE
        at Object.Module._extensions..node (module.js:682:18)
        at Module.load (module.js:566:32)
        at tryModuleLoad (module.js:506:12)
        at Function.Module._load (module.js:498:3)
        at Module.require (module.js:597:17)
        at require (internal/module.js:11:18)
        at Object.<anonymous> (/tmp/kartotherian/node_modules/@mapbox/blend/node_modules/mapnik/lib/mapnik.js:31:15)
        at Module._compile (module.js:653:30)
        at Object.Module._extensions..js (module.js:664:10)
        at Module.load (module.js:566:32)
        at tryModuleLoad (module.js:506:12)
        at Function.Module._load (module.js:498:3)
        at Module.require (module.js:597:17)
        at require (internal/module.js:11:18)
        at Object.<anonymous> (/tmp/kartotherian/node_modules/@mapbox/blend/index.js:1:76)
        at Module._compile (module.js:653:30)
error Command failed with exit code 1.

It looks like a mapnik build issue with yarn.

Best,

It seems the env params in airbnb-base are setting up node:true,es6:true and those cannot be overridden.

The static/main.js file is browser facing, and must be linted with browser:true and es5 rules.

As a test, changing the "extends" rule in .eslintrc.js to "wikimedia" fixes the es6 linting. The airbnb-base has an explicit es6:true that cannot be overridden.

eslint has a bug that the environment variable cannot be overridden inline, so that solution doesn't work either.

We can't serve browser-facing javascript containing es6 syntax, so this linting must be fixed.

It has been fairly difficult to manage tens of separate repos that contain small semi-related packages, especially when migrating one package required updates to another one. It seems a good solution to this is a common practice of combining code into a mono-repo. All code is placed inside a single repository as separate "package" sub-directories, but each package is maintained and published separately. Also, there are tools that help with managing it, e.g. Lerna, which handles NPM linking, common versioning, etc.

Plan

• Rename kartotherian/master branch to kartotherian/legacy
• Using git merge, combine all repositories into a single repository kartotherian with the following structure:
  • /README.md
  • /lerna.json
  • /packages/kartotherian
  • /packages/tilerator
  • /packages/babel
  • Root dir will contain scripts, build tools, styles, docs, and anything else related to the system as a whole rather than an individual package.
• Archive other repos once everything works, redirecting them to the corresponding package

CC: @mateusbs17 @MaxSem @mdholloway

What is the format of the served JSON files?

The wiki page says:

you can also use "json" to see the GeoJSON data

but it is clearly not GeoJSON, but some custom JSON data. Is it documented somewhere?

Several jobs on zoom level 14 are invalid. But after restarting the tilerator, including tileratorUI and tilerator-workers, these jobs can be processed successfully. Some parts of error log are shown as below:

�[36mtilerator_1         |�[0m Error: job "185" is invalid
�[36mtilerator_1         |�[0m     at Command.callback (/opt/tilerator/node_modules/kue/lib/queue/job.js:182:17)
�[36mtilerator_1         |�[0m     at normal_reply (/opt/tilerator/node_modules/redis/index.js:721:21)
�[36mtilerator_1         |�[0m     at RedisClient.return_reply (/opt/tilerator/node_modules/redis/index.js:819:9)
�[36mtilerator_1         |�[0m     at JavascriptRedisParser.returnReply (/opt/tilerator/node_modules/redis/index.js:192:18)
�[36mtilerator_1         |�[0m     at JavascriptRedisParser.execute (/opt/tilerator/node_modules/redis-parser/lib/parser.js:574:12)
�[36mtilerator_1         |�[0m     at Socket.<anonymous> (/opt/tilerator/node_modules/redis/index.js:274:27)
�[36mtilerator_1         |�[0m     at emitOne (events.js:116:13)
�[36mtilerator_1         |�[0m     at Socket.emit (events.js:211:7)
�[36mtilerator_1         |�[0m     at addChunk (_stream_readable.js:263:12)
�[36mtilerator_1         |�[0m     at readableAddChunk (_stream_readable.js:250:11)
�[36mtilerator_1         |�[0m     at Socket.Readable.push (_stream_readable.js:208:10)
�[36mtilerator_1         |�[0m     at TCP.onread (net.js:601:20)
�[36mtilerator_1         |�[0m Error: job "202" is invalid
�[36mtilerator_1         |�[0m     at Command.callback (/opt/tilerator/node_modules/kue/lib/queue/job.js:182:17)
�[36mtilerator_1         |�[0m     at normal_reply (/opt/tilerator/node_modules/redis/index.js:721:21)
�[36mtilerator_1         |�[0m     at RedisClient.return_reply (/opt/tilerator/node_modules/redis/index.js:819:9)
�[36mtilerator_1         |�[0m     at JavascriptRedisParser.returnReply (/opt/tilerator/node_modules/redis/index.js:192:18)
�[36mtilerator_1         |�[0m     at JavascriptRedisParser.execute (/opt/tilerator/node_modules/redis-parser/lib/parser.js:574:12)
�[36mtilerator_1         |�[0m     at Socket.<anonymous> (/opt/tilerator/node_modules/redis/index.js:274:27)
�[36mtilerator_1         |�[0m     at emitOne (events.js:116:13)
�[36mtilerator_1         |�[0m     at Socket.emit (events.js:211:7)
�[36mtilerator_1         |�[0m     at addChunk (_stream_readable.js:263:12)
�[36mtilerator_1         |�[0m     at readableAddChunk (_stream_readable.js:250:11)
�[36mtilerator_1         |�[0m     at Socket.Readable.push (_stream_readable.js:208:10)
�[36mtilerator_1         |�[0m     at TCP.onread (net.js:601:20)
�[36mtilerator_1         |�[0m Error: job "206" is invalid
�[36mtilerator_1         |�[0m     at Command.callback (/opt/tilerator/node_modules/kue/lib/queue/job.js:182:17)
�[36mtilerator_1         |�[0m     at normal_reply (/opt/tilerator/node_modules/redis/index.js:721:21)
�[36mtilerator_1         |�[0m     at RedisClient.return_reply (/opt/tilerator/node_modules/redis/index.js:819:9)
�[36mtilerator_1         |�[0m     at JavascriptRedisParser.returnReply (/opt/tilerator/node_modules/redis/index.js:192:18)
�[36mtilerator_1         |�[0m     at JavascriptRedisParser.execute (/opt/tilerator/node_modules/redis-parser/lib/parser.js:574:12)
�[36mtilerator_1         |�[0m     at Socket.<anonymous> (/opt/tilerator/node_modules/redis/index.js:274:27)
�[36mtilerator_1         |�[0m     at emitOne (events.js:116:13)
�[36mtilerator_1         |�[0m     at Socket.emit (events.js:211:7)
�[36mtilerator_1         |�[0m     at addChunk (_stream_readable.js:263:12)
�[36mtilerator_1         |�[0m     at readableAddChunk (_stream_readable.js:250:11)
�[36mtilerator_1         |�[0m     at Socket.Readable.push (_stream_readable.js:208:10)
�[36mtilerator_1         |�[0m     at TCP.onread (net.js:601:20)
�[36mtilerator_1         |�[0m Error: job "196" is invalid
�[36mtilerator_1         |�[0m     at Command.callback (/opt/tilerator/node_modules/kue/lib/queue/job.js:182:17)
�[36mtilerator_1         |�[0m     at normal_reply (/opt/tilerator/node_modules/redis/index.js:721:21)
�[36mtilerator_1         |�[0m     at RedisClient.return_reply (/opt/tilerator/node_modules/redis/index.js:819:9)
�[36mtilerator_1         |�[0m     at JavascriptRedisParser.returnReply (/opt/tilerator/node_modules/redis/index.js:192:18)
�[36mtilerator_1         |�[0m     at JavascriptRedisParser.execute (/opt/tilerator/node_modules/redis-parser/lib/parser.js:574:12)
�[36mtilerator_1         |�[0m     at Socket.<anonymous> (/opt/tilerator/node_modules/redis/index.js:274:27)
�[36mtilerator_1         |�[0m     at emitOne (events.js:116:13)
�[36mtilerator_1         |�[0m     at Socket.emit (events.js:211:7)
�[36mtilerator_1         |�[0m     at addChunk (_stream_readable.js:263:12)
�[36mtilerator_1         |�[0m     at readableAddChunk (_stream_readable.js:250:11)
�[36mtilerator_1         |�[0m     at Socket.Readable.push (_stream_readable.js:208:10)
�[36mtilerator_1         |�[0m     at TCP.onread (net.js:601:20)

Right now the code and instructions include not just the tile serving stack, but are coupled to a specific style.

The software should not contain instructions that require

• A specific .tm2 style
• A specific .tm2source style
• Shapefiles
• SQL functions
• PostgreSQL or PostGIS

A lot of the documentation should be pushed down to https://github.com/kartotherian/osm-bright.tm2source

There are a few repos that were merged with #89 that were initially forked. I am having second thoughts about merging those, mostly because that makes them hard to PR into the originals, and also because they no longer show up in the fork network.

I propose to remove them (rewrite history), and keep them independent:

• abaculus
• tilelive-tmsource
• tilelive-vector

Do you have a process for update ? How invalidate vector tiles on update and on web cache (Varnish in you case) ?

Please, can you write the readme to explain the install process. Maybe explain the requires osm-bright
subprojects as sub git module.

npm installation is missing in apt command

This is visible through the failing unit tests; the headers get 'Content-Encoding': 'gzip' regardless of the gzip option that's passed.

I've tried to follow the route of the code through but it seems like somewhere in the stack when we unpack the tile and repack it, the 'Content-Encoding': 'gzip' is added indiscriminately.

@nyurik is this a known issue that we should fix in the code, or is this a bad expectation of the unit tests?

If the tests are correct in expecting 'Content-Encoding': 'gzip' only when requested, can you point me towards where? I've went into @kartotherian/core down the rabbit hole, and am still not sure what adds it in, and I have a sense this isn't even in babel. If that's the case, maybe the tests should be corrected?

Tilerator (and Kartotherian) hardcodes a number of packages that shouldn't really be part of it, e.g. @kartotherian/cassandra, @kartotherian/autogen, @kartotherian/babel and many other. These packages are important for specific configurations, but not in the general case. Additionally, Tilerator and other deps still use an old version of Mapnik (current is 4.21, we use 3.7).

I would like to propose a different packaging that will be both more flexible and leaner:

• package.json will not contain any optional packages
• specific targets, e.g. WMF build, would use additional npm install <extra packages> as part of the build process right after running npm install without any params.
• Docker-based build and deployment

This fits well into a more common modern packaging scenario - Docker/Kubernetes. Tilerator will be published as a "core tilerator" docker image in the packages.docker.com, plus the WMF-specific Dockerfile would use npm install X to add more components and publish it to the WMF-specific docker repository.

P.S. This will also allow us to have a more flexible, secure and smaller packages, e.g. by building Mapnik libraries during the build, but only copy the needed results to the resulting docker image. Also, this method is still compatible with the original, because the resulting build results can still be pushed to the deployment git repo.

CC: @gehel @stephanebisson @mdholloway @mateusbs17

Is there a chance that mapbox-gl-js will return? I saw that it was included in earlier versions, wasn't it working well?

~Ben

Ubuntu 18.04, Node.JS v8.12.0, npm 6.4.1, Kernel: 4.15.0-36-generic

npm log attached.

Commands:

git clone https://github.com/kartotherian/kartotherian.git
cd kartotherian
npm install

Output:

npm WARN deprecated [email protected]: This module is now under the @mapbox namespace: install @mapbox/sphericalmercator instead
npm WARN deprecated [email protected]: renamed to d3-queue
npm WARN deprecated [email protected]: protozero should no longer be used via npm, install instead via https://github.com/mapbox/mason
npm WARN deprecated [email protected]: maki is now namespaced, install @mapbox/maki instead
npm WARN deprecated [email protected]: Use uuid module instead
npm WARN deprecated [email protected]: Please upgrade to @kadenceproject/kadence - See https://kadence.github.io
npm WARN deprecated [email protected]: Please upgrade to @kadenceproject/kadence - See https://kadence.github.io

[email protected] install /home/user/projects/kartotherian/node_modules/canvas
node-gyp rebuild

./util/has_lib.sh: 31: ./util/has_lib.sh: pkg-config: not found
gyp: Call to './util/has_lib.sh freetype' returned exit status 0 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: gyp failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:345:16)
gyp ERR! stack at emitTwo (events.js:126:13)
gyp ERR! stack at ChildProcess.emit (events.js:214:7)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Linux 4.15.0-36-generic
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/user/projects/kartotherian/node_modules/canvas
gyp ERR! node -v v8.12.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm WARN [email protected] requires a peer of @mapbox/tilelive-vector@>=2.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of ajv@^5.0.0 but none is installed. You must install peer dependencies yourself.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/user/.npm/_logs/2018-10-15T18_37_17_984Z-debug.log

2018-10-15T18_37_17_984Z-debug.log

If we need to serve pbf tiles, we need to add in the sources.*.yaml configuration (in the style section)

pbfsource: gen

It should be added in some doc file.

(It works as a server for MapboxGL :-) )

There are many name tags in OSM, yet for some reason we are handling "name" as a special case. I think it should be treated the same as any other language, using the same fallback mechanism.

For example, a full names_ could be

{
  "en": "name:en",
  "fr": "name:fr",
  "*name": "name",
  "*intl": "name_intl",
...
}

While in theory *name could be defined by OSM editor, there are so few of them we can safely ignore them. The fallback file could define the order for each type, or we could even have a generic fallback when specific language fallback fails (try en, and if not en, try name_intl, and if not, try name)

cc: @pnorman @mooeypoo

Hello,

At higher zoom levels, the wrong tiles are displayed:

(https://maps.wikimedia.org/#17/48.58548/7.75050)

I have tried a few combinations already, but I can't figure how to setup and start the server using a different db name ("gis_bw"), user ("mapuser") and providing a password for the db user. I'm using MD5 as authentication in pg_hba.conf.

~Ben

I would like to implement dynamically updateable style and island map support. The use-case I'm thinking of is like this:

• Given some external tool, a user can extract part of a map. This part will be copied into an own database for isolated editing and viewing.
• Given another external style editing tool, a user makes changes to an existing style - or creates a new named style.
• The map can then be viewed via http://mymap/?source=default/?s=osm

How can I change the config loading and related modules so that I could then for example use a simple scheme to store source and style configs in a directors structure like:
configs/sourceName/styles/styleName

Example content:
configs/default/source.yaml
configs/default/styles/osm <-- osm-bright-style copy
configs/myMapOfNY/source.yaml
configs/myMapOfNY/styles/myNYStyle/ <-- editable copy of osm-bright-style

Is it possible?

~Ben

During the last sync, the following topics/goals were establish:

• Agreed to move to monorepo - components will remain separately published to NPM, but all code will reside in a single git repo
• Migrate non-core component usage from package.json to the user-specific deployment build
  • E.g. to here - https://github.com/wikimedia/maps-kartotherian-package/blob/master/package.json#L36
• Improve testing coverage
• Do not rely on master, should always use package versions
• Add Dockerfiles to simplify "core builds" - e.g. Ubuntu and Debian, possibly including Mapnik master build, etc
  • Should help simplify testing
• Adopt package-lock.json (will stick with NPM for now, but YARN migration is desirable in the future)

CC: @mateusbs17 @mdholloway @gehel (please add those missing)

Using hstore has up to 10% storage and 10% performance penalty.

Is it required for the stock style? The original OSM Bright doesn't use it. Is it more for future proofing?

Please, can you explicite how to use the gl vector view in the readme ? It's working by default ?
(I successfully setup a running kartotherian with png tiles)

I'm lost trying to build the docker.
I finally figure the Dockefile was generated.
The committed Dokerfile is useful or not? Can be removed?