mapbox / untiler Goto Github PK

View Code? Open in Web Editor NEW

38.0 109.0 11.0 765 KB

Stitch image tiles into larger composite TIFs

License: MIT License

Python 100.00%

satellite pxm imagery

untiler's Issues

Tests broken on master

Tests fail on Python 3.6 - see this Travis run.

    def test_extract_mbtiles():
        with TestTiler() as tt:
            testpath = tt.path
            testmbtiles = os.path.join(os.path.dirname(__file__), 'fixtures/testtiles.mbtiles')
            runner = CliRunner()
            result = runner.invoke(cli, [
                'streammbtiles', testmbtiles, testpath, '-z', '16', '-x', '-s',
                '{z}-{x}-{y}-mbtiles.tif', '--co', 'compress=lzw'])
            assert result.exit_code == 0
            expected_checksums = [[13858, 8288, 51489, 31223], [17927, 52775, 411, 9217]]
            for o, c in zip(result.output.rstrip().split('\n'), expected_checksums):
                with rio.open(o) as src:
                    checksums = [src.checksum(i) for i in src.indexes]
>                   assert checksums == c
E                   assert [17927, 52775, 411, 9217] == [13858, 8288, 51489, 31223]
E                     At index 0 diff: 17927 != 13858
E                     Use -v to get the full diff

Noticed this while working on the autodeploy branch. Initially thought this was failing due to changes I'd made, but I think this is some sort of config/dependency rot that's happening otherwise.

pydocstyle reports 29 missing docstrings

$ python -m pydocstyle --count --select=D1 --add-ignore=D105 .
...
29

Open source tile-stitcher

Open Source is the Best Source

cc @camillacaros @sgillies @yhahn

Revisit command line options

At the risk of 🚲 🏡 ... @dnomadb we have some conflicts with traditional options (-c for one) and I'd like to suggest --foo-bar instead of --foobar.

Python 3.6 compatibility

A few things to change (that I see now):

A few integer division changes: https://github.com/mapbox/untiler/blob/master/untiler/scripts/tile_utils.py#L49
Bigger prob: mbutil is (a lot farther) from 3.5 https://github.com/mapbox/untiler/blob/master/untiler/scripts/mbtiles_extract.py#L5. I think removing it entirely and reading tiles using sqlite3 is preferable.

cc @perrygeo

sudo: false with rasterio deps

As per chat w/ @yhahn (https://mapbox.slack.com/archives/satellite/p1441040393007215 + many other convos), we want to able to have modules such as this testable on travis w/ sudo: false. However, I am not sure how to accomplish this on a repo that uses rasterio, which has dependencies that are installed via sudo apt-get install ....

@sgillies can you help guide me in this?

KeyError: '\\d'

Hi, I recently installed untiler. When I run any of the untiler commands, I get the error attached below. I am using Python 3.9.1.

I'm not sure if it's an issue with the python version, I am currently trying to downgrade my python version, and will update this issue accordingly.

in __exit__
Traceback (most recent call last):
  File "/usr/lib/python3.9/sre_parse.py", line 1039, in parse_template
    this = chr(ESCAPES[this][1])
KeyError: '\\d'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/c_abraham/.local/bin/untiler", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/c_abraham/.local/lib/python3.9/site-packages/untiler/scripts/cli.py", line 53, in streammbtiles
    untiler.stream_dir(input_tile_dir, output_dir, compositezoom, maxzoom, None, readtemplate, scenetemplate, workers, creation_options, no_fill)
  File "/home/c_abraham/.local/lib/python3.9/site-packages/untiler/__init__.py", line 225, in stream_dir
    template, readTemplate, separator = tile_utils.parse_template("%s/%s" % (inputDir, read_template))
  File "/home/c_abraham/.local/lib/python3.9/site-packages/untiler/scripts/tile_utils.py", line 151, in parse_template
    return valPattern.sub('\d+', template), valPattern.sub('%s', template), separator[0]
  File "/usr/lib/python3.9/re.py", line 327, in _subx
    template = _compile_repl(template, pattern)
  File "/usr/lib/python3.9/re.py", line 318, in _compile_repl
    return sre_parse.parse_template(repl, pattern)
  File "/usr/lib/python3.9/sre_parse.py", line 1042, in parse_template
    raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \d at position 0

Use rasterio for tile reading

Using rasterio for input tile reading should be faster for reading input images. To implement:

Use syntax:

 with rasterio.drivers():
     with rasterio.open(...) as src:
         imdata = src.read()

Update tiler to handle (depth, height, width) arrays vs (height, width, depth) and (height, depth) arrays

cc @sgillies

How to get this to work with QGIS

So it doesn't seem to work with QGIS Tiles, and the things it works with output nonsense.

these were originally grey, and you can see they didn't even stitch properly, any insight on how to make this work?

Support for 512x512 tiles: ValueError: Source shape is inconsistent with given indexes

I am having trouble getting untiler to work with a directory of 512x512 tiles. Support for this does not seem to be documented, but it sounds like a typical use case given Mapbox's 512x512 default in Studio. Is this supported? Is it a rasterio issue, like this?

This is what happens when trying to untile with 512x512 images. (An equivalent run with 256x256 tiles succeeds.)

$ untiler streamdir -t {z}/{x}/{y}.png ./tiles ./untiles 
/usr/local/lib/python3.6/site-packages/rasterio/__init__.py:240: NotGeoreferencedWarning: Dataset has no geotransform set. Default transform will be applied (Affine.identity())
  s = DatasetReader(fp, driver=driver, **kwargs)
/usr/local/lib/python3.6/site-packages/rasterio/__init__.py:240: NotGeoreferencedWarning: Dataset has no geotransform set. Default transform will be applied (Affine.identity())
  s = DatasetReader(fp, driver=driver, **kwargs)
tiles/18/66179/97359.png errored
tiles/18/66179/97403.png errored
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.6/site-packages/untiler/__init__.py", line 201, in streaming_tile_worker
    raise e
  File "/usr/local/lib/python3.6/site-packages/untiler/__init__.py", line 192, in streaming_tile_worker
    dst.write(imdata, window=window)
  File "rasterio/_io.pyx", line 1234, in rasterio._io.DatasetWriterBase.write
ValueError: Source shape is inconsistent with given indexes
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/untiler", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/untiler/scripts/cli.py", line 32, in streamdir
    untiler.stream_dir(input_dir, output_dir, compositezoom, maxzoom, logdir, readtemplate, scenetemplate, workers, creation_options, no_fill)
  File "/usr/local/lib/python3.6/site-packages/untiler/__init__.py", line 255, in stream_dir
    for p in pool.imap_unordered(streaming_tile_worker, tiler.get_sub_tiles(allTiles, superTiles)):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
ValueError: Source shape is inconsistent with given indexes

mapbox / untiler Goto Github PK

untiler's Issues

Tests broken on master

pydocstyle reports 29 missing docstrings

Open source tile-stitcher

Revisit command line options

Python 3.6 compatibility

sudo: false with rasterio deps

KeyError: '\\d'

Use rasterio for tile reading

How to get this to work with QGIS

Support for 512x512 tiles: ValueError: Source shape is inconsistent with given indexes

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent