kartotherian / kartotherian Goto Github PK
View Code? Open in Web Editor NEWMap Tile Server
Home Page: https://www.mediawiki.org/wiki/Maps
License: Apache License 2.0
Map Tile Server
Home Page: https://www.mediawiki.org/wiki/Maps
License: Apache License 2.0
Right now the code and instructions include not just the tile serving stack, but are coupled to a specific style.
The software should not contain instructions that require
A lot of the documentation should be pushed down to https://github.com/kartotherian/osm-bright.tm2source
Nothing's been happening here for 2 years, maybe it's time to admit the obvious and close the project to avoid people wasting their time on considering it for their usage? With Dependabot notifications piling, nobody knows if the thing still works with recent Node and is safe to use...
Using this command: curl -X POST 'http://localhost:16533/add?generatorId=gen&storageId=v2&zoom=0&fromZoom=0&beforeZoom=16&checkZoom=-1&parts=12'
to add a job to Tilerator I get an error (message in title). Stack trace says it fails here: https://github.com/kartotherian/kartotherian-core/blob/286670b135433ae4917c72f71157efa28b1284e0/lib/core.js#L408-L411. It appears to be failing on the checkZoom filter param, '-1' doesn't seem to be passing as a valid zoom. Relevant part of stack trace:
"stack": "tilerator: Invalid zoom param - an integer zoom value was expected\n
at Object.checkType (/usr/local/lib/node_modules/tilerator/node_modules/kartotherian-core/lib/core.js:410:23)\n
at Job.<anonymous> (/usr/local/lib/node_modules/tilerator/lib/Job.js:96:22)
So I am testing the basic add job call to Tilerator with this command: curl -X POST 'http://localhost:16533/add?generatorId=gen&storageId=v2&zoom=0'
and in response I get an error:
"stack": "tilerator: tiles parameter not set
at new Job (/usr/local/lib/node_modules/tilerator/lib/Job.js:125:15)
To me, it looks like the default values for idxFrom
and idxBefore
are not getting registered properly. When Tilerator processes the job it goes through this sequence: https://github.com/kartotherian/tilerator/blob/master/lib/Job.js#L64-L73 ... where it should set the tiles
param from idxFrom
& idxBefore
, but it never gets the chance because the condition never passes. In checkType (https://github.com/kartotherian/kartotherian-core/blob/master/lib/core.js#L330-L342) it looks like it returns false after it sets the default value of the field... So I guess we need to either return true there or not return and allow checkType to continue to check the default value. What do ya'll think?
npm installation is missing in apt command
It seems the env params in airbnb-base are setting up node:true,es6:true and those cannot be overridden.
The static/main.js file is browser facing, and must be linted with browser:true and es5 rules.
As a test, changing the "extends" rule in .eslintrc.js to "wikimedia" fixes the es6 linting. The airbnb-base has an explicit es6:true that cannot be overridden.
eslint has a bug that the environment variable cannot be overridden inline, so that solution doesn't work either.
We can't serve browser-facing javascript containing es6 syntax, so this linting must be fixed.
There are a few repos that were merged with #89 that were initially forked. I am having second thoughts about merging those, mostly because that makes them hard to PR into the originals, and also because they no longer show up in the fork network.
I propose to remove them (rewrite history), and keep them independent:
Ubuntu 18.04, Node.JS v8.12.0, npm 6.4.1, Kernel: 4.15.0-36-generic
npm log attached.
Commands:
git clone https://github.com/kartotherian/kartotherian.git
cd kartotherian
npm install
Output:
npm WARN deprecated [email protected]: This module is now under the @mapbox namespace: install @mapbox/sphericalmercator instead
npm WARN deprecated [email protected]: renamed to d3-queue
npm WARN deprecated [email protected]: protozero should no longer be used via npm, install instead via https://github.com/mapbox/mason
npm WARN deprecated [email protected]: maki is now namespaced, install @mapbox/maki instead
npm WARN deprecated [email protected]: Use uuid module instead
npm WARN deprecated [email protected]: Please upgrade to @kadenceproject/kadence - See https://kadence.github.io
npm WARN deprecated [email protected]: Please upgrade to @kadenceproject/kadence - See https://kadence.github.io
[email protected] install /home/user/projects/kartotherian/node_modules/canvas
node-gyp rebuild
./util/has_lib.sh: 31: ./util/has_lib.sh: pkg-config: not found
gyp: Call to './util/has_lib.sh freetype' returned exit status 0 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: gyp
failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:345:16)
gyp ERR! stack at emitTwo (events.js:126:13)
gyp ERR! stack at ChildProcess.emit (events.js:214:7)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Linux 4.15.0-36-generic
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/user/projects/kartotherian/node_modules/canvas
gyp ERR! node -v v8.12.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm WARN [email protected] requires a peer of @mapbox/tilelive-vector@>=2.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of ajv@^5.0.0 but none is installed. You must install peer dependencies yourself.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /home/user/.npm/_logs/2018-10-15T18_37_17_984Z-debug.log
Hey all, I am not sure this is the best place to post this question, if you know of someone I should rather talk to or somewhere I should rather post this, feel free to let me know.
So I am trying to use tilerator to generate map tiles z=0...15 for a map tile service. I just upgraded my hardware & it seems pretty good for the job. Planet OSM takes around 14 hours to import. Tilerator is chugging away now. It burned through the first 12 zoom levels at lightning speed. It took maybe 4 days to finish z=14, & now it is slowly working on z=15. My question is what does ya'lls production process look like @ wikimaps? How long does it take you guys to generate z=0...15 & what kind of hardware are you using? Also how are updates handled with tilerator? Is there a way to update only tiles that have changed based on OSM planet diffs? @ my current rate, it looks like it will take over two weeks to complete all the tiles... this seems like a big time investment, but I guess if I can update individual tiles over time & never lose my database it isn't too big of a deal as a one time investment. Advice most welcomed. Cheers & thankyou
brew install mapnik
to install)To install the dependencies for all modules, I run the following from the project root, but it has something wrong.
lerna bootstrap --hoist --nohoist mapnik --nohoist libxmljs
lerna notice cli v3.14.1
lerna info versioning independent
lerna ERR! EWORKSPACES --hoist is not supported with --npm-client=yarn, use yarn workspaces instead
lerna ERR! EWORKSPACES A guide is available at https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/
And I try to run another command yarn install
from the project root, but it still does not work.
$ yarn install yarn install v1.16.0
[1/4] 🔍 Resolving packages...
[2/4] 🚚 Fetching packages...
warning [email protected]: The engine "firefox" appears to be invalid.
warning [email protected]: The engine "chrome" appears to be invalid.
[3/4] 🔗 Linking dependencies...
warning " > @kartotherian/[email protected]" has unmet peer dependency "mapnik@~3.7.0".
warning " > @kartotherian/[email protected]" has unmet peer dependency "mapnik@^3.7.0".
warning " > @kartotherian/[email protected]" has unmet peer dependency "mapnik@~3.7.0".
warning "workspace-aggregator-1506f268-a74c-4387-90bc-dd4623300415 > @kartotherian/babel > [email protected]" has unmet peer dependency "eslint@^4.9.0".
warning "workspace-aggregator-1506f268-a74c-4387-90bc-dd4623300415 > @kartotherian/babel > [email protected]" has unmet peer dependency "eslint-plugin-import@^2.7.0".
warning "workspace-aggregator-1506f268-a74c-4387-90bc-dd4623300415 > kartotherian > [email protected]" has unmet peer dependency "@mapbox/tilelive-vector@>=2.0.0".
warning "workspace-aggregator-1506f268-a74c-4387-90bc-dd4623300415 > @kartotherian/server > [email protected]" has unmet peer dependency "[email protected] - 5.x".
[4/4] 🔨 Building fresh packages...
[7/10] ⠠ heapdump
[2/10] ⠠ mapnik
[3/10] ⠠ mapnik
[6/10] ⠠ dtrace-provider
error ./kartotherian/node_modules/canvas: Command failed.
Exit code: 1
Command: node-gyp rebuild
Arguments:
Directory: ./kartotherian/node_modules/canvas
Output:
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | darwin | x64
gyp http GET https://nodejs.org/download/release/v8.16.0/node-v8.16.0-headers.tar.gz
gyp http 200 https://nodejs.org/download/release/v8.16.0/node-v8.16.0-headers.tar.gz
gyp http GET https://nodejs.org/download/release/v8.16.0/SHASUMS256.txt
gyp http 200 https://nodejs.org/download/release/v8.16.0/SHASUMS256.txt
gyp info spawn /usr/local/bin/python2
gyp info spawn args [ './kartotherian/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args 'binding.gyp',
gyp info spawn args '-f',
gyp info spawn args 'make',
gyp info spawn args '-I',
gyp info spawn args './kartotherian/node_modules/canvas/build/config.gypi',
gyp info spawn args '-I',
gyp info spawn args './kartotherian/node_modules/node-gyp/addon.gypi',
gyp info spawn args '-I',
gyp info spawn args '~/.node-gyp/8.16.0/include/node/common.gypi',
gyp info spawn args '-Dlibrary=shared_library',
gyp info spawn args '-Dvisibility=default',
gyp info spawn args '-Dnode_root_dir=~/.node-gyp/8.16.0',
gyp info spawn args '-Dnode_gyp_dir=./kartotherian/node_modules/node-gyp',
gyp info spawn args '-Dnode_lib_file=~/.node-gyp/8.16.0/<(target_arch)/node.lib',
gyp info spawn args '-Dmodule_root_dir=./kartotherian/node_modules/canvas',
gyp info spawn args '-Dnode_engine=v8',
gyp info spawn args '--depth=.',
gyp info spawn args '--no-parallel',
gyp info spawn args '--generator-output',
gyp info spawn args 'build',
gyp info spawn args '-Goutput_dir=.' ]
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libffi', required by 'gobject-2.0', not found
gyp: Call to './util/has_lib.sh freetype' returned exit status 0 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (./kartotherian/node_modules/node-gyp/lib/configure.js:345:16)
gyp ERR! stack at emitTwo (events.js:126:13)
gyp ERR! stack at ChildProcess.emit (events.js:214:7)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Darwin 18.5.0
gyp ERR! command "~/.nvm/versions/node/v8.16.0/bin/node" "./kartotherian/node_modules/.bin/node-gyp" "rebuild"
gyp ERR! cwd ./kartotherian/node_modules/canvas
Several jobs on zoom level 14 are invalid. But after restarting the tilerator, including tileratorUI and tilerator-workers, these jobs can be processed successfully. Some parts of error log are shown as below:
�[36mtilerator_1 |�[0m Error: job "185" is invalid
�[36mtilerator_1 |�[0m at Command.callback (/opt/tilerator/node_modules/kue/lib/queue/job.js:182:17)
�[36mtilerator_1 |�[0m at normal_reply (/opt/tilerator/node_modules/redis/index.js:721:21)
�[36mtilerator_1 |�[0m at RedisClient.return_reply (/opt/tilerator/node_modules/redis/index.js:819:9)
�[36mtilerator_1 |�[0m at JavascriptRedisParser.returnReply (/opt/tilerator/node_modules/redis/index.js:192:18)
�[36mtilerator_1 |�[0m at JavascriptRedisParser.execute (/opt/tilerator/node_modules/redis-parser/lib/parser.js:574:12)
�[36mtilerator_1 |�[0m at Socket.<anonymous> (/opt/tilerator/node_modules/redis/index.js:274:27)
�[36mtilerator_1 |�[0m at emitOne (events.js:116:13)
�[36mtilerator_1 |�[0m at Socket.emit (events.js:211:7)
�[36mtilerator_1 |�[0m at addChunk (_stream_readable.js:263:12)
�[36mtilerator_1 |�[0m at readableAddChunk (_stream_readable.js:250:11)
�[36mtilerator_1 |�[0m at Socket.Readable.push (_stream_readable.js:208:10)
�[36mtilerator_1 |�[0m at TCP.onread (net.js:601:20)
�[36mtilerator_1 |�[0m Error: job "202" is invalid
�[36mtilerator_1 |�[0m at Command.callback (/opt/tilerator/node_modules/kue/lib/queue/job.js:182:17)
�[36mtilerator_1 |�[0m at normal_reply (/opt/tilerator/node_modules/redis/index.js:721:21)
�[36mtilerator_1 |�[0m at RedisClient.return_reply (/opt/tilerator/node_modules/redis/index.js:819:9)
�[36mtilerator_1 |�[0m at JavascriptRedisParser.returnReply (/opt/tilerator/node_modules/redis/index.js:192:18)
�[36mtilerator_1 |�[0m at JavascriptRedisParser.execute (/opt/tilerator/node_modules/redis-parser/lib/parser.js:574:12)
�[36mtilerator_1 |�[0m at Socket.<anonymous> (/opt/tilerator/node_modules/redis/index.js:274:27)
�[36mtilerator_1 |�[0m at emitOne (events.js:116:13)
�[36mtilerator_1 |�[0m at Socket.emit (events.js:211:7)
�[36mtilerator_1 |�[0m at addChunk (_stream_readable.js:263:12)
�[36mtilerator_1 |�[0m at readableAddChunk (_stream_readable.js:250:11)
�[36mtilerator_1 |�[0m at Socket.Readable.push (_stream_readable.js:208:10)
�[36mtilerator_1 |�[0m at TCP.onread (net.js:601:20)
�[36mtilerator_1 |�[0m Error: job "206" is invalid
�[36mtilerator_1 |�[0m at Command.callback (/opt/tilerator/node_modules/kue/lib/queue/job.js:182:17)
�[36mtilerator_1 |�[0m at normal_reply (/opt/tilerator/node_modules/redis/index.js:721:21)
�[36mtilerator_1 |�[0m at RedisClient.return_reply (/opt/tilerator/node_modules/redis/index.js:819:9)
�[36mtilerator_1 |�[0m at JavascriptRedisParser.returnReply (/opt/tilerator/node_modules/redis/index.js:192:18)
�[36mtilerator_1 |�[0m at JavascriptRedisParser.execute (/opt/tilerator/node_modules/redis-parser/lib/parser.js:574:12)
�[36mtilerator_1 |�[0m at Socket.<anonymous> (/opt/tilerator/node_modules/redis/index.js:274:27)
�[36mtilerator_1 |�[0m at emitOne (events.js:116:13)
�[36mtilerator_1 |�[0m at Socket.emit (events.js:211:7)
�[36mtilerator_1 |�[0m at addChunk (_stream_readable.js:263:12)
�[36mtilerator_1 |�[0m at readableAddChunk (_stream_readable.js:250:11)
�[36mtilerator_1 |�[0m at Socket.Readable.push (_stream_readable.js:208:10)
�[36mtilerator_1 |�[0m at TCP.onread (net.js:601:20)
�[36mtilerator_1 |�[0m Error: job "196" is invalid
�[36mtilerator_1 |�[0m at Command.callback (/opt/tilerator/node_modules/kue/lib/queue/job.js:182:17)
�[36mtilerator_1 |�[0m at normal_reply (/opt/tilerator/node_modules/redis/index.js:721:21)
�[36mtilerator_1 |�[0m at RedisClient.return_reply (/opt/tilerator/node_modules/redis/index.js:819:9)
�[36mtilerator_1 |�[0m at JavascriptRedisParser.returnReply (/opt/tilerator/node_modules/redis/index.js:192:18)
�[36mtilerator_1 |�[0m at JavascriptRedisParser.execute (/opt/tilerator/node_modules/redis-parser/lib/parser.js:574:12)
�[36mtilerator_1 |�[0m at Socket.<anonymous> (/opt/tilerator/node_modules/redis/index.js:274:27)
�[36mtilerator_1 |�[0m at emitOne (events.js:116:13)
�[36mtilerator_1 |�[0m at Socket.emit (events.js:211:7)
�[36mtilerator_1 |�[0m at addChunk (_stream_readable.js:263:12)
�[36mtilerator_1 |�[0m at readableAddChunk (_stream_readable.js:250:11)
�[36mtilerator_1 |�[0m at Socket.Readable.push (_stream_readable.js:208:10)
�[36mtilerator_1 |�[0m at TCP.onread (net.js:601:20)
Please, can you explicite how to use the gl vector view in the readme ? It's working by default ?
(I successfully setup a running kartotherian with png tiles)
@nyurik, any objections to if I tag 0.0.31 later today?
I don't know if this is actually the right Github to report this, but looking at the fallbacks.json file included here, I noticed that West Frisian, the second official language spoken in the Friesland province of the Netherlands, is missing:
https://en.wikipedia.org/wiki/Frisian_languages
I think a "fry":"nl" needs to be added. The other two Frisian languages (North and Saterland Frisian - frr and stq) are currently listed.
Do you have a process for update ? How invalidate vector tiles on update and on web cache (Varnish in you case) ?
Hi,
I tried to use yarn
to install JS deps and run kartotherian. I am running node v8.14.0, yarn v1.12.3 on Ubuntu 18.04 LTS. Everything runs fine when using npm
.
When running yarn install
, no error is shown. However, when starting kartotherian (with the config.external.yaml file), I get the following trace:
$ yarn run start
yarn run v1.12.3
$ service-runner
[2019-01-04T13:36:58.177Z] INFO: kartotherian/3275 on ubuntu-bionic: Loading variables from the config file (levelPath=info)
[2019-01-04T13:36:58.184Z] INFO: kartotherian/3275 on ubuntu-bionic: Loading sources from /tmp/kartotherian/sources.external.yaml (levelPath=info)
[2019-01-04T13:36:59.026Z] FATAL: kartotherian/3275 on ubuntu-bionic: /tmp/kartotherian/node_modules/@mapbox/blend/node_modules/mapnik/lib/binding/mapnik.node: undefined symbol: _ZN6mapnik19MAPNIK_LONGLAT_PROJE
Error: /tmp/kartotherian/node_modules/@mapbox/blend/node_modules/mapnik/lib/binding/mapnik.node: undefined symbol: _ZN6mapnik19MAPNIK_LONGLAT_PROJE
at Object.Module._extensions..node (module.js:682:18)
at Module.load (module.js:566:32)
at tryModuleLoad (module.js:506:12)
at Function.Module._load (module.js:498:3)
at Module.require (module.js:597:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/tmp/kartotherian/node_modules/@mapbox/blend/node_modules/mapnik/lib/mapnik.js:31:15)
at Module._compile (module.js:653:30)
at Object.Module._extensions..js (module.js:664:10)
at Module.load (module.js:566:32)
at tryModuleLoad (module.js:506:12)
at Function.Module._load (module.js:498:3)
at Module.require (module.js:597:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/tmp/kartotherian/node_modules/@mapbox/blend/index.js:1:76)
at Module._compile (module.js:653:30)
error Command failed with exit code 1.
It looks like a mapnik
build issue with yarn.
Best,
Greetings everyone,
I have a strange problem when generation job finishes in less 1 second.
Example POST request:
What is the possible reason?
Hi, I am following the readme to install kartotherian, but I found some minor issues in the installation steps:
"@kartotherian/tilelive-http",
to the registerSourceLibs
section (already there)This is visible through the failing unit tests; the headers get 'Content-Encoding': 'gzip'
regardless of the gzip
option that's passed.
I've tried to follow the route of the code through but it seems like somewhere in the stack when we unpack the tile and repack it, the 'Content-Encoding': 'gzip'
is added indiscriminately.
@nyurik is this a known issue that we should fix in the code, or is this a bad expectation of the unit tests?
If the tests are correct in expecting 'Content-Encoding': 'gzip'
only when requested, can you point me towards where? I've went into @kartotherian/core down the rabbit hole, and am still not sure what adds it in, and I have a sense this isn't even in babel. If that's the case, maybe the tests should be corrected?
Is there a chance that mapbox-gl-js will return? I saw that it was included in earlier versions, wasn't it working well?
~Ben
It has been fairly difficult to manage tens of separate repos that contain small semi-related packages, especially when migrating one package required updates to another one. It seems a good solution to this is a common practice of combining code into a mono-repo
. All code is placed inside a single repository as separate "package" sub-directories, but each package is maintained and published separately. Also, there are tools that help with managing it, e.g. Lerna, which handles NPM linking, common versioning, etc.
kartotherian/master
branch to kartotherian/legacy
git merge
, combine all repositories into a single repository kartotherian
with the following structure:
/README.md
/lerna.json
/packages/kartotherian
/packages/tilerator
/packages/babel
CC: @mateusbs17 @MaxSem @mdholloway
Hi there,
We noticed that your repo has a medium severity vulnerability:
Here is the test report for this repo.
If you’d like to fix this vulnerability, Snyk lets you generate a pull request that recommends the best upgrade path - there’s a link to fix this vulnerability on the test report.
Stay secure :-)
Snyk Community
From @benstadin on October 21, 2015 19:21
Is there a chance that mapbox-gl-js will return? I saw that it was included in earlier versions, wasn't it working well?
~Ben
Copied from original issue: kartotherian/kartotherian-server#1
Kartotherian has outgrown its current configuration/modular system, and needs to be rethought. Existing configuration example. Ideally, we may be able to engage larger tilelive community to consider revising the existing interface towards a more generic Promise-based one.
getTile(z,x,y,callback)
function that returns [data, headers] via callback.<Promise<object>> get(<object>)
. If the module has old interface, the get()
should be injected. If the module only supports the new interface, the old functions should be injected so that it can be used with other tilelive code.getTile(z, x, y, callback)
→ <Promise<object>> getTileAsync(object)
, with the assumption that object parameter will have z, x, y
values. The loader will inject a wrapper function if either getTile
or getTileAsync
is missing./:src([A-Za-z][-A-Za-z0-9_]*)/:z(\\d+)/:x(\\d+)/:y(\\d+):scale(@[\\.\\d]+x)?.:format([\\w]+)
/geoshape?...
, /geoline?...
, /shape?...
/img/:src([A-Za-z][-A-Za-z0-9_]*),:zoom(a|\\d+),:lat(a|[-\\d\\.]+),:lon(a|[-\\d\\.]+),:w(\\d+)x:h(\\d+):scale(@[\\.\\d]+x)?.:format([\\w]+)
routes:
# List of route objects
- route:
# The route param is a string or a list of routes as accepted by express
- "/img/:src([A-Za-z][-A-Za-z0-9_]*),:zoom(a|\\d+),:lat(a|[-\\d\\.]+),:lon(a|[-\\d\\.]+),:w(\\d+)x:h(\\d+):scale(@[\\.\\d]+x)?.:format([\\w]+)?"
- "/:src([A-Za-z][-A-Za-z0-9_]*)/:z(\\d+)/:x(\\d+)/:y(\\d+):scale(@[\\.\\d]+x)?.:format([\\w]+)"
formats: [png,json,headers,svg,jpeg]
scales: [1.3, 1.5, 2, 2.6, 3]
# Source could be either an ID, or the object describing the source
source: gen
sources: # a path to a file, or an object with source configurations
gen:
protocol: bridge
xml: "/path/to/file.xml"
params:
# Just as before, each param could be a string/number/boolean,
# or an object with a single key, e.g. "ref" for a source ID, or "source" for in-place source initialization
backend:
source:
...
variables: # a path to a file, or an object with variables
varA: value
I was thinking about storing the rendered PNG's in cassandra, but it seems that this is hard coded to store PBF's. Is that the only way you would recommend doing it?
If we need to serve pbf tiles, we need to add in the sources.*.yaml configuration (in the style section)
pbfsource: gen
It should be added in some doc file.
(It works as a server for MapboxGL :-) )
Hello,
At higher zoom levels, the wrong tiles are displayed:
(https://maps.wikimedia.org/#17/48.58548/7.75050)
https://github.com/kartotherian/tilerator-jobprocessor/blob/master/lib/fileParser.js#L116 will hang on stdin if the list if empty.
Related-To: https://phabricator.wikimedia.org/T145534#2987786
After 'fixing' (and getting rid of) the test for content encoding, the splicerTest unit tests now fail on Error: Unexpected layer tag #0
This is coming from splicerTest.js line #59 where it seems something inside the data sent to the tileCodec.decodeTile(result.data);
leads to a layer tag 0, which is unexpected (see https://github.com/kartotherian/babel/blob/master/lib/tileCodec.js#L60 )
The full stack trace for one of the failures:
9) Tag recombination pick missing:
Error: Unexpected layer tag #0
at Tile.Layer._readField (lib/tileCodec.js:123:24)
at Object.readFields (node_modules/pbf/index.js:39:13)
at Object.Tile.Layer.read (lib/tileCodec.js:108:16)
at Tile._readField (lib/tileCodec.js:27:36)
at Object.readFields (node_modules/pbf/index.js:39:13)
at Object.Tile.read (lib/tileCodec.js:22:16)
at Object.decodeTile (lib/tileCodec.js:7:21)
at babel.then.then.result (test/splicerTest.js:59:37)
at Gzip.onEnd (zlib.js:227:5)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickCallback (internal/process/next_tick.js:104:9)
From previous event:
at test (test/splicerTest.js:50:11)
at Context.it (test/splicerTest.js:257:30)
I can't make sense of the data or why and how the tag is sent incorrectly. Is this a known bug? Is this indicating a deeper issue with the tile codec code? It's hard to test since babel isn't really used yet, as far as I can see, so I can't tell if the test itself is sending bad data (the code is correct) or if the test is good but its failure indicates something is wrong with the tile codec reading method (the code is broken)
What is the format of the served JSON files?
The wiki page says:
you can also use "json" to see the GeoJSON data
but it is clearly not GeoJSON, but some custom JSON data. Is it documented somewhere?
I would like to implement dynamically updateable style and island map support. The use-case I'm thinking of is like this:
How can I change the config loading and related modules so that I could then for example use a simple scheme to store source and style configs in a directors structure like:
configs/sourceName/styles/styleName
Example content:
configs/default/source.yaml
configs/default/styles/osm <-- osm-bright-style copy
configs/myMapOfNY/source.yaml
configs/myMapOfNY/styles/myNYStyle/ <-- editable copy of osm-bright-style
Is it possible?
~Ben
Tilerator (and Kartotherian) hardcodes a number of packages that shouldn't really be part of it, e.g. @kartotherian/cassandra
, @kartotherian/autogen
, @kartotherian/babel
and many other. These packages are important for specific configurations, but not in the general case. Additionally, Tilerator and other deps still use an old version of Mapnik (current is 4.21, we use 3.7).
I would like to propose a different packaging that will be both more flexible and leaner:
package.json
will not contain any optional packagesnpm install <extra packages>
as part of the build process right after running npm install
without any params.This fits well into a more common modern packaging scenario - Docker/Kubernetes. Tilerator will be published as a "core tilerator" docker image in the packages.docker.com
, plus the WMF-specific Dockerfile would use npm install X
to add more components and publish it to the WMF-specific docker repository.
P.S. This will also allow us to have a more flexible, secure and smaller packages, e.g. by building Mapnik libraries during the build, but only copy the needed results to the resulting docker image. Also, this method is still compatible with the original, because the resulting build results can still be pushed to the deployment
git repo.
CC: @gehel @stephanebisson @mdholloway @mateusbs17
Hey everyone,
Is there any tilerator's benchmark? How long it takes to generate the whole planet (0-17 zoom)?
There are many name tags in OSM, yet for some reason we are handling "name" as a special case. I think it should be treated the same as any other language, using the same fallback mechanism.
For example, a full names_ could be
{
"en": "name:en",
"fr": "name:fr",
"*name": "name",
"*intl": "name_intl",
...
}
While in theory *name
could be defined by OSM editor, there are so few of them we can safely ignore them. The fallback file could define the order for each type, or we could even have a generic fallback when specific language fallback fails (try en, and if not en, try name_intl, and if not, try name)
During the last sync, the following topics/goals were establish:
CC: @mateusbs17 @mdholloway @gehel (please add those missing)
Hello,
The tiles are not cached so they have to be reloaded everytime.
It would be useful to have an Expires
header.
I have tried a few combinations already, but I can't figure how to setup and start the server using a different db name ("gis_bw"), user ("mapuser") and providing a password for the db user. I'm using MD5 as authentication in pg_hba.conf.
~Ben
I'm lost trying to build the docker.
I finally figure the Dockefile was generated.
The committed Dokerfile is useful or not? Can be removed?
Using hstore has up to 10% storage and 10% performance penalty.
Is it required for the stock style? The original OSM Bright doesn't use it. Is it more for future proofing?
hi,
( this is most-likely not a bug, but a head-scratching case. i don't get it...)
( also, this points to the "production" version maps.wikimedia.org -- as used in wikidata.org .. )
working on
https://www.wikidata.org/wiki/Q183104
the geo shows this map
https://maps.wikimedia.org/img/osm-intl,13,47.238055555556,9.5983333333334,310x180.png?lang=en
The name of the city shows "Sogn Pieder", but should be "Feldkirch"
this seems to be the "räteromanisch" ( = a language) name of the city. ( https://de.wikipedia.org/wiki/Feldkirch )
This makes no sense. Räterromanisch is a small, valid language in Switzerland (nearby), but no official one in Austria.
For me this looks like: at as some-point-in-time this was as-is in the data.
With "current" i cant re-create this behavior. ( i can also switch the "?lang=en" to "de", "fr".. nothing changes )
Maybe some "very stable" server side caching issue ?
.ka7
Hi,
Unless I'm mistaken, the https://www.wikidata.org/wiki/Q25975588 entry for kartotherian that was added today does not meet the notability requirements.
Did I miss something ?
Cheers
Please, can you write the readme to explain the install process. Maybe explain the requires osm-bright
subprojects as sub git module.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.