protomaps / go-pmtiles Goto Github PK
View Code? Open in Web Editor NEWSingle-file executable tool for working with PMTiles archives
License: BSD 3-Clause "New" or "Revised" License
Single-file executable tool for working with PMTiles archives
License: BSD 3-Clause "New" or "Revised" License
I'll admit up front that I am least familiar with Go apps generally, so please excuse any very basic misses here. I love the concept of PMTiles and hope to test, initially focusing on python, but it seems this Go server is the more robust option.
I had hoped to take a shortcut to free deployment via Heroku for testing (following this workaround for compiled app https://medium.com/ki-labs-engineering/deploying-a-native-go-binary-on-heroku-6d4c955819d8), but the lack of configuration of port seems to be the main blocker. There appears to be no way to utilize a dynamic port currently.
Please disregard if this is an edge case, or if there is generally a better way to approach deployment of a precompiled Go app such as this. Excited to figure this out...
How can I compile and run go-pmtiles
on Ubuntu? I would like to mess around a bit with the source code...
If your /tmp
is mounted on a separate partition that doesn't have enough space to hold the pmtiles, the conversion of a mbtiles can never succeed.
like extract, but the whole thing.
Initial implementation here: https://github.com/protomaps/go-pmtiles/blob/main/caddy/pmtiles_proxy.go
Outstanding tasks:
github.com/protomaps/go-pmtiles
. It fails to make a build when I click download.
go.mod
is at the root of the repo, and is the module also for the CLI program.go.mod
just for Caddy. I would also prefer not to add caddy
as a dependency of go-pmtiles
(unless the parts needed by the plugin are extremely lightweight) how should I re-organize this repo?here is an example Caddyfile:
localhost:2019 {
route /* {
pmtiles_proxy {
bucket https://example.com
cache_size 64
}
}
}
A request for https://localhost:2019/foo/0/0/0.mvt
will fetch https://example.com/foo.pmtiles
. Great!
route /*
. If we did route /prefix/*
, we need to pass this information into the plugin so it can strip that off when it makes a fetch to the pmtiles file. Is this configuration state something I can access in Provision
via caddy.Context
? This overlaps with #61localhost:2019
information in order to generate correct TileJSON. I hardcoded this to https://example.com
for now. Is this something we can access in Provision
or do we need to trust the host of the request struct?I was using the writer interface of v2 combined with github.com/paulmach/orb to create my pmtiles file.
writer := pmtiles.NewWriter("./base.pmtiles")
for z := minZoom; z <= maxZoom; z++ {
covering := tilecover.Bound(bound, z)
for tile := range covering {
layers := mvt.Layers{.....layers.....}
tileData, _ := mvt.Marshal(layers)
writer.WriteTile(
pmtiles.Zxy{
Z: uint8(tile.Z),
X: tile.X,
Y: tile.Y,
},
tileData,
)
}
}
metadata, err := json.Marshal(pmtiles.Metadata{
...metadata...
})
if err != nil {
return err
}
writer.Finalize(metadata)
Is something similar still possible using v3?
To create a better separation of concerns, it might be a good idea to split the pmtiles folder. The pmtiles folder can hold all the library code for reading and writing pmtiles. Next to a cmd folder that can hold the cli related commands and logic for handling flags, etc.
Hi,
Please could you add examples of how to use pmtiles via docker to build or upload pmtiles
Beleive it something like docker run -v $(pwd)/data:/var/ ghcr.io/protomaps/go-pmtiles:main convert radon.pmtiles radon.mbtiles
, however this doesn't work for me.
Please could you also provide more detailed instructions to install on linux - I can see the binary but don't know how to install
I'll happily add to the readme
and submit a PR once I know how.
Kind regards,
Edd
I think if you use the pmtiles
command line tool to convert an mbtiles
into a pmtiles
file, and the mbtiles
doesn't have bounds/center metadata, then you get a successfully-generated pmtiles
file which the JS library (at least) will silently ignore.
I'm 80% sure this is what's happening, but I don't have the time to dig into the code to confirm at the moment (sorry).
Visualize a covering on a map
Makes it much easier to keep things up to date
https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request
https://docs.brew.sh/Formula-Cookbook
Here's the tippecanoe formula: https://github.com/Homebrew/homebrew-core/blob/master/Formula/tippecanoe.rb
Since this is open source it should be acceptable in homebrew-core
: https://docs.brew.sh/Acceptable-Formulae
edit: looks like there's even a tool for this: https://jkawamoto.github.io/fgo/ and goreleaser
also can generate a formula: https://goreleaser.com/customization/homebrew/
I'm experimenting with generating pmtiles (with MVT) from Rust. https://github.com/dabreegster/od2net/blob/tippecanwho/tippe/src/main.rs is my initial attempt. It doesn't produce usable output yet, but I'm having trouble figuring out where the problem is.
https://www.dropbox.com/scl/fi/25bikp6xvb3l6we182zt9/bad_output.pmtiles?rlkey=wn19lcqeramyxfv93jtmwdjt5&dl=0 is a bad output I'm producing right now. pmtiles verify
shows no errors and seems happy. Loading the file with https://protomaps.github.io/PMTiles doesn't show anything on the map, and the console has an error:
e24e0201-5cdb-4bcb-8107-86ee3d56b440:2 Uncaught Error: unknown command 4
at wc.loadGeometry (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:90342)
at sa (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:64684)
at f0.populate (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:100722)
at C.parse (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:217921)
at e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:220887
at e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:220047
at u.Actor.processTask (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:194145)
at D5.process [as _callback] (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:193365)
at D5.MessageChannel._channel.port2.onmessage (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:159810)
The results of pmtiles show
:
pmtiles spec version: 3
tile type: Vector Protobuf (MVT)
bounds: 0.000000,0.000000 0.000000,0.000000
min zoom: 0
max zoom: 0
center: 0.000000,0.000000
center zoom: 0
addressed tiles count: 1
tile entries count: 1
tile contents count: 1
clustered: true
internal compression: 2
tile compression: 1
vector_layers <object...>
I'm pretty certain I'm failing to populate some of the metadata correctly, or doing something very wrong with encoding geometry. My ask here is if it makes sense for go-pmtiles verify
to check for possible problems more.
I've downloaded the sample 2.4gb level 10 zoom map and wanted to try and make one that only zooms to level 8
I'm trying to run the command to extract like this on a windows command line
pmtiles.exe extract --maxzoom=8 z10.pmtiles z8.pmtiles
But I just get the error
panic: extract <input> <output>
goroutine 1 [running]:
main. Main()
/Users/bdon/workspace/protomaps/go-pmtiles/main.go:166 +0xc1e
Is there anything I'm doing wrong there, note that /Users/bdon/workspace is not my directory location, so not sure where that's coming from
Underlying data is OK but upload is producing corrupted archives past a certain point compared to rclone
If the directory does not fit in only the root, right now the code defaults to creating a root with leaves of a fixed size 4, 8, 16... kB.
It should be smart enough for two cases:
./pmtiles verify xyz.pmtiles
panic: verify <input>
goroutine 1 [running]:
main.main()
/Users/bdon/workspace/protomaps/go-pmtiles/main.go:166 +0xc19
pmtiles 1.7.0, commit dfb4fe68e5c6b25eec8703864863fcefb2effb3d, built at 2023-02-06T03:33:16Z
related to backslashes.
This should "just work" with one request, but verify it works
I'm pre-rendering millions of raster-files on multiple processes. Since sqlite-based databases still aren't any good on this, I'm directly writing them to the filesystem. It would be nice to bulk import fresh & updated tiles.
pmtiles import
I can try to do it on my own and contribute, but I'm not into GO.
This resolves one of the main drawbacks of PMTiles, which is that they can't have arbitrary Content-Encoding compression applied when delivered directly to the browser.
Here are the logs:
2023/04/25 05:38:45 convert.go:362: # of addressed tiles: 5147885
2023/04/25 05:38:45 convert.go:363: # of tile entries (after RLE): 3117056
2023/04/25 05:38:45 convert.go:364: # of tile contents: 2840164
panic: runtime error: index out of range [3117056] with length 3117056
goroutine 1 [running]:
github.com/protomaps/go-pmtiles/pmtiles.build_roots_leaves({0xc021480000, 0x2f9000, 0x35e2aa}, 0x1000)
/Users/bdon/workspace/protomaps/go-pmtiles/pmtiles/directory.go:287 +0x31f
github.com/protomaps/go-pmtiles/pmtiles.optimize_directories({0xc021480000, 0x2f9000, 0x35e2aa}, 0x3f81)
/Users/bdon/workspace/protomaps/go-pmtiles/pmtiles/directory.go:310 +0xad
github.com/protomaps/go-pmtiles/pmtiles.finalize(0xc000186400?, 0xc00011a2a0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
/Users/bdon/workspace/protomaps/go-pmtiles/pmtiles/convert.go:376 +0x30e
github.com/protomaps/go-pmtiles/pmtiles.ConvertMbtiles(0xc00063fdc8?, {0xc000124570, 0x24}, {0xc000045ec0, 0x3b}, 0x0?, 0x4?)
/Users/bdon/workspace/protomaps/go-pmtiles/pmtiles/convert.go:353 +0xfa8
github.com/protomaps/go-pmtiles/pmtiles.Convert(0x0?, {0xc000124570?, 0x24?}, {0xc000045ec0?, 0x3b?}, 0x7?, 0x58?)
/Users/bdon/workspace/protomaps/go-pmtiles/pmtiles/convert.go:110 +0x97
main.main()
/Users/bdon/workspace/protomaps/go-pmtiles/main.go:150 +0xb19
I'm currently working on isolating the problem on my end (it could be a malformed mbtiles archive), but at the very least this error should be actionable. The index out of range .. of length is the same number, so maybe there's an off by one bug somewhere? Has anyone run in to this problem?
Look into S3 retries
Downstream requirements:
Hi first thanks for this project.
I'm moved my project from mbtiles to a dedicated kv solution and finally to pmtiles which is way better, problem is the actual go-pmtiles is not directly usable as an API.
I had to copy some private part of your code, to create a different http handler https://github.com/akhenakh/kvtiles/blob/master/storage/pmtiles/pmtiles.go
Are you planning on making a more Goish API?
./pmtiles
2022/10/27 14:16:01 main.go:22: Command is required.
There doesn't seem to be any way to get the available commands listed. ./pmtiles --help
doesn't work.
Put go-pmtiles docker image on GitHub registry
It'd be great to allow converting between any pair of supported tile types (mvt, png, jpeg, or webp).
Maybe this could be an option for pmtiles convert --format png <a> <b>
?
Here's what happened to me (and I imagine other users not super familiar with mapping tools can easily run into similar problems):
react-leaflet
, pointing a TileLayer
at https://<mydomain>/<pmtiles name>/{z}/{x}/{y}.mvt
. This resulted in a CORB error due to the fact that TileLayer
was creating img
objects with mvt
files that the Cloudflare server was serving with application/x-protobuf
.protomaps
npm library to create the layer from mvt
files.There are a few options that would make this easier for new users:
react-leaflet
(I created a separate issue for this)How do i start to debug a 'Missing row' error? Not much information is returned.
main.go:154: Failed to convert /home/ubuntu/tilemaker/planet.osm.mbtiles, Missing row
I'm using v1.8.0, with tilemaker v2.4.0
My script to generate the planet is....
aws s3 cp s3://osm-pds/2023/planet-230529.osm.pbf /data/planet-latest.osm.pbf
wget https://osmdata.openstreetmap.de/download/water-polygons-split-4326.zip -O /home/ubuntu/tilemaker/water-polygons-split-4326.zip
unzip -o water-polygons-split-4326.zip
ln -fs water-polygons-split-4326 coastline
wget https://osmdata.openstreetmap.de/download/land-polygons-complete-4326.zip -O /home/ubuntu/tilemaker/land-polygons-complete-4326.zip
unzip -o land-polygons-complete-4326.zip
ln -fs land-polygons-complete-4326 landcover
tilemaker --input /data/planet-latest.osm.pbf --output planet.osm.mbtiles --skip-integrity --store /tmp --compact
go-pmtiles convert planet.osm.mbtiles planet.pmtiles
I spent the weekend trying to get a demo of v3 PMtiles up and running, but ran into an error with my own encoded pmtile.
Here is the pipeline I’ve developed:
osm.pbf -> tilemaker (with openmaptiles config) -> osm.mbtiles -> go_pmtiles -> osm.pmtiles
tilemaker --input hawaii-latest.osm.pbf \
--output hawaii.mbtiles \
--config config-openmaptiles.json \
--process process-openmaptiles.lua
pmtiles hawaii.mbtiles hawaii.pmtiles
This produces the following v3 .pmtile:
$ pmtiles show hawaii.pmtiles
pmtiles spec version: 3
total size: 14 MB
tile type: Vector Protobuf (MVT)
bounds: -179.848300,10.835750 -137.522800,29.096240
min zoom: 0
max zoom: 14
center: -158.685550,19.965995
center zoom: 7
addressed tiles count: 5189
tile entries count: 5066
tile contents count: 4894
clustered: true
internal compression: 2
tile compression: 2
maxzoom 14
minzoom 0
name Tilemaker to OpenMapTiles schema
type baselayer
vector_layers <object...>
version 3.0
description Tile config based on OpenMapTiles schema
format pbf
Which I serve:
$ pmtiles serve .
2023/02/13 21:25:46 main.go:118: Serving . on port 8080 with Access-Control-Allow-Origin:
And then request a pbf over Hawaii with the following url:
http://localhost:8080/hawaii/10/301/384.pbf
The logs print:
2023/02/13 21:26:44 server.go:139: fetching hawaii 0-16384
2023/02/13 21:26:44 server.go:180: fetched hawaii 0-0
2023/02/13 21:26:44 main.go:115: served /hawaii/10/301/384.pbf in 1.64375ms
But in the browser I get a 400 error path mismatch: archive is type MVT (.mvt)
.
Any idea what that error means? Does the pipeline and process look correct?
Additionally, which frontend codebase is best for testing v3 pmtiles:
Comment in this thread with your reports on version 1.9's extract
command.
pmtiles extract https://r2-public.protomaps.com/protomaps-sample-datasets/protomaps-basemap-opensource-20230408.pmtiles out.pmtiles --region myregion.json
pmtiles extract
example run, extracting z0-z15 US + Mexico from a planet OSM basemap hosted on Source Cooperative, medium EC2 instance in same AWS region
./pmtiles extract protomaps/openstreetmap/tiles/v2/protomaps-basemap-opensource-20230408.pmtiles us_and_mexico.pmtiles --bucket=s3://us-west-2.opendata.source.coop --region=us_and_mexico.json
WARNING: extract is an experimental feature and results may not be suitable for production use.
fetching 303 dirs, 303 chunks, 15 requests
Region tiles 26196418, result tile entries 15282945
fetching 15282945 tiles, 4261 chunks, 339 requests
fetching chunks 100% |█████████████████████████████████████████████████████████████████████████████████████| (17/17 GB, 33 MB/s)
Completed in 9m24.19785483s seconds with 1 download thread.
Extract required 357 total requests.
Extract transferred 18 GB (overfetch 0.1) for an archive size of 17 GB
Verify your extract is usable at https://protomaps.github.io/PMTiles/
Feedback wanted! report your success or failure to https://github.com/protomaps/go-pmtiles/issues
That's 28,000 tiles a second
Right now the download is unoptimized and single threaded - the priority right now is to ensure that the result is correct, we'll make a multithreaded extractor after.
--overfetch=0.1
By default, the downloader will batch nearby requests by downloading slightly more data, by default 10%. --overfetch=1.0
lets it download twice the final size of data.--dry-run
prints out stats without actually downloading tile dataThis should work against a local file only, verifying that:
Etc...
It'd be great if I can configure options using environment variables to have deployments be easily managed via docker, etc.
PMTILES_PATH=https://storage.googleapis.com/my-tiles
PMTILES_CORS=*
PMTILES_CACHE=2000
like extract, but the whole thing.
When trying to convert mbtiles an error occurs:
main.go:159: Failed to convert xxx.mbtiles, Failed to convert MBTiles to header JSON, strconv.ParseFloat: parsing " 34.30084411": invalid syntax
The thing is that in the metadata the value for center
was 28.33709717, 34.30084411, 18
, i.e. contained spaces between values.
TL;DR I can't get tiles out of go-pmtiles v0.5 for some files which work well through pmtiles-js.
I suspect it is linked with the file having leaf directories.
File | Size | JS - test URL | GO - test URL |
---|---|---|---|
Bugianen_Cervino | 0.2 GB | 🆗 | 🆗 wtracks C-go |
Bugianen | 3.5 GB (sorry!) | 🆗 wtracks B-js | ⛔ wtracks B-go |
eslo14_walps | 1.5 G | 🆗 | ⛔ |
I am integrating some pmtiles maps in https://opoto.github.io/wtracks/
After merging pmtiles support (opoto/wtracks#21), I have working, but excruciatingly slow pmtiles-js-on-HTTP1 maps (that we discussed in protomaps/PMTiles#16). For example IT Bugianen (find it in More... menu or though the custom link above)
Now I'm trying to evaluate performance on go-pmtiles on the same map, but it does not work (metadata works 404 on tiles I tried).
See the links to download/test the files above. I including a valid file to show that the issue is not how my server setup on montagne.top.
To test on wtracks, search for a covered area for both maps eg "Aosta".
Or test with curl:
curl -i https://www.montagne.top/maki/Bugianen_Cervino/16/34100/23382.jpg
HTTP/1.1 200 OK
curl -i https://www.montagne.top/maki/Bugianen/16/34100/23382.jpg
HTTP/1.1 404 Not Found
curl -I https://www.montagne.top/tile/eslo14_walps/16/34100/23382.png
HTTP/1.1 404 Not Found
I can try to debug it myself at some point, but figured maybe it's a known issue with big files?
It seems that tiles are not cached (trying on latest chrome, linux)
Looking at others they all set cache-control:
cache-control: max-age=604800, stale-while-revalidate=604800, stale-if-error=604800
Cache-Control: max-age=439803
cache-control: public, must-revalidate, max-age=3600
access-control-max-age: 43200
I'd like to combine two (or more) .pmtiles
files. Browsing the docs I don't see a way to do this.
As a workaround I also considered converting the .pmtiles
files back to .mbtiles
files and using https://github.com/mapbox/tippecanoe 's tile-join
tool to join them into a single .mbtiles
file, and then back to a .pmtiles
file. But I don't see a way to convert .pmtiles
to .mbtiles
.
Let me know if I'm missing something. Otherwise its just a feature request. Thank you!
It would be useful to be able to extract tiles from a pmtiles file for a certain area and put the extracted tiles into a new pmtiles file.
It could look something like this:
pmtiles extract input.pmtiles output.pmtiles --bounds "47.5,7.5,48.0,8.5"
A usecase could be to have a global tileset and allow users to download an extract for a city or so.
Otherwise these will default to 0 if not present.
Requires using https://gocloud.dev/concepts/as/
define a JSON schema like
{
'tile_type':0,
'metadata':{
'my_key':'my_value'
}
}
to allow modification of pmtiles by writing a new file.
pmtiles show --header-metadata-json
would also be useful to output this.
should show an error
/pmtiles show file:// public/catch.pmtiles
total size: 13 MB
tile type: Vector Protobuf (MVT)
bounds: 140.962409,-39.136654 149.974995,-33.981390
min zoom: 0
max zoom: 15
center: 142.476196,-38.371809
center zoom: 15
addressed tiles count: 322258
tile entries count: 55994
tile contents count: 42135
clustered: true
Surprised it doesn't list what spec version the .pmtiles file is - that would be helpful.
Used go-pmtiles to convert a MBTiles with compressed vector tiles. It did not work in maplibre. Then I made the MBTiles with uncompressed vector tiles and converted again. Then it worked just fine. Am I doing something wrong?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.