mapbox / untiler Goto Github PK
View Code? Open in Web Editor NEWStitch image tiles into larger composite TIFs
License: MIT License
Stitch image tiles into larger composite TIFs
License: MIT License
Tests fail on Python 3.6 - see this Travis run.
def test_extract_mbtiles(): with TestTiler() as tt: testpath = tt.path testmbtiles = os.path.join(os.path.dirname(__file__), 'fixtures/testtiles.mbtiles') runner = CliRunner() result = runner.invoke(cli, [ 'streammbtiles', testmbtiles, testpath, '-z', '16', '-x', '-s', '{z}-{x}-{y}-mbtiles.tif', '--co', 'compress=lzw']) assert result.exit_code == 0 expected_checksums = [[13858, 8288, 51489, 31223], [17927, 52775, 411, 9217]] for o, c in zip(result.output.rstrip().split('\n'), expected_checksums): with rio.open(o) as src: checksums = [src.checksum(i) for i in src.indexes] > assert checksums == c E assert [17927, 52775, 411, 9217] == [13858, 8288, 51489, 31223] E At index 0 diff: 17927 != 13858 E Use -v to get the full diff
Noticed this while working on the autodeploy branch. Initially thought this was failing due to changes I'd made, but I think this is some sort of config/dependency rot that's happening otherwise.
$ python -m pydocstyle --count --select=D1 --add-ignore=D105 .
...
29
Open Source is the Best Source
At the risk of ๐ฒ ๐ก ... @dnomadb we have some conflicts with traditional options (-c
for one) and I'd like to suggest --foo-bar
instead of --foobar
.
A few things to change (that I see now):
mbutil
is (a lot farther) from 3.5 https://github.com/mapbox/untiler/blob/master/untiler/scripts/mbtiles_extract.py#L5. I think removing it entirely and reading tiles using sqlite3 is preferable.cc @perrygeo
As per chat w/ @yhahn (https://mapbox.slack.com/archives/satellite/p1441040393007215 + many other convos), we want to able to have modules such as this testable on travis w/ sudo: false. However, I am not sure how to accomplish this on a repo that uses rasterio, which has dependencies that are installed via sudo apt-get install ...
.
@sgillies can you help guide me in this?
Hi, I recently installed untiler. When I run any of the untiler commands, I get the error attached below. I am using Python 3.9.1.
I'm not sure if it's an issue with the python version, I am currently trying to downgrade my python version, and will update this issue accordingly.
in __exit__
Traceback (most recent call last):
File "/usr/lib/python3.9/sre_parse.py", line 1039, in parse_template
this = chr(ESCAPES[this][1])
KeyError: '\\d'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/c_abraham/.local/bin/untiler", line 8, in <module>
sys.exit(cli())
File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/c_abraham/.local/lib/python3.9/site-packages/untiler/scripts/cli.py", line 53, in streammbtiles
untiler.stream_dir(input_tile_dir, output_dir, compositezoom, maxzoom, None, readtemplate, scenetemplate, workers, creation_options, no_fill)
File "/home/c_abraham/.local/lib/python3.9/site-packages/untiler/__init__.py", line 225, in stream_dir
template, readTemplate, separator = tile_utils.parse_template("%s/%s" % (inputDir, read_template))
File "/home/c_abraham/.local/lib/python3.9/site-packages/untiler/scripts/tile_utils.py", line 151, in parse_template
return valPattern.sub('\d+', template), valPattern.sub('%s', template), separator[0]
File "/usr/lib/python3.9/re.py", line 327, in _subx
template = _compile_repl(template, pattern)
File "/usr/lib/python3.9/re.py", line 318, in _compile_repl
return sre_parse.parse_template(repl, pattern)
File "/usr/lib/python3.9/sre_parse.py", line 1042, in parse_template
raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \d at position 0
Using rasterio for input tile reading should be faster for reading input images. To implement:
Use syntax:
with rasterio.drivers():
with rasterio.open(...) as src:
imdata = src.read()
Update tiler
to handle (depth, height, width)
arrays vs (height, width, depth)
and (height, depth)
arrays
cc @sgillies
I am having trouble getting untiler
to work with a directory of 512x512 tiles. Support for this does not seem to be documented, but it sounds like a typical use case given Mapbox's 512x512 default in Studio. Is this supported? Is it a rasterio
issue, like this?
This is what happens when trying to untile with 512x512 images. (An equivalent run with 256x256 tiles succeeds.)
$ untiler streamdir -t {z}/{x}/{y}.png ./tiles ./untiles
/usr/local/lib/python3.6/site-packages/rasterio/__init__.py:240: NotGeoreferencedWarning: Dataset has no geotransform set. Default transform will be applied (Affine.identity())
s = DatasetReader(fp, driver=driver, **kwargs)
/usr/local/lib/python3.6/site-packages/rasterio/__init__.py:240: NotGeoreferencedWarning: Dataset has no geotransform set. Default transform will be applied (Affine.identity())
s = DatasetReader(fp, driver=driver, **kwargs)
tiles/18/66179/97359.png errored
tiles/18/66179/97403.png errored
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/local/lib/python3.6/site-packages/untiler/__init__.py", line 201, in streaming_tile_worker
raise e
File "/usr/local/lib/python3.6/site-packages/untiler/__init__.py", line 192, in streaming_tile_worker
dst.write(imdata, window=window)
File "rasterio/_io.pyx", line 1234, in rasterio._io.DatasetWriterBase.write
ValueError: Source shape is inconsistent with given indexes
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/untiler", line 11, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/untiler/scripts/cli.py", line 32, in streamdir
untiler.stream_dir(input_dir, output_dir, compositezoom, maxzoom, logdir, readtemplate, scenetemplate, workers, creation_options, no_fill)
File "/usr/local/lib/python3.6/site-packages/untiler/__init__.py", line 255, in stream_dir
for p in pool.imap_unordered(streaming_tile_worker, tiler.get_sub_tiles(allTiles, superTiles)):
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 735, in next
raise value
ValueError: Source shape is inconsistent with given indexes
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.