Git Product home page Git Product logo

cogdumper's Introduction

COG Dumper

Build Status codecov

A python (3.6) utility to extract a tile from a Cloud Optimized GeoTIFF (COG) without decompressing the contained data. Tiff data can be hosted locally, on a web server or S3.

This can be useful for serving compressed tiles from a TIFF without invoking Rasterio and GDAL. This utility has been tested with Tiff that have JPEG compression.

Tiled Tiff is an extension to TIFF 6.0 and more detail can be found here

Note that tiles are padded at the edge of an image. This requires an image mask to be defined if tile sizes do not align with the image width / height (as in the test data which demonstrates this effect).

Data Preparation

Read the document on COG and create a tiled pyramid GeoTIFF.

For example;

gdal_translate SENTINEL2_L1C:S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441.SAFE/MTD_MSIL1C.xml:TCI:EPSG_32630 \
               S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI.tif \
               -co TILED=YES -co COMPRESS=DEFLATE
gdaladdo -r average  S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI.tif 2 4 8 16 32
gdal_translate S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI.tif \
               S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif \
               -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co COPY_SRC_OVERVIEWS=YES

This library also works with a file hosted in an S3 bucket.

Installation

Python 3.6 is required.

pip install cogdumper

or from source

git clone https://github.com/mapbox/COGDumper
cd COGDumper
pip install .

Command line interface

$ cogdumper --help
Usage: cogdumper [OPTIONS] COMMAND [ARGS]...

  Command line interface for COGDumper.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  file  COGDumper cli for local dataset.
  http  COGDumper cli for web hosted dataset.
  s3    COGDumper cli for AWS S3 hosted dataset
local files
cogdumper file --help
Usage: cogdumper file [OPTIONS]

  COGDumper cli for local dataset.

Options:
  --file PATH       input file  [required]
  --output PATH     local output directory
  --xyz INTEGER...  xyz tile coordinate where z is the overview level
  --version         Show the version and exit.
  --help            Show this message and exit.

e.g. cogdumper file --file data/cog.tif --xyz 0 0 0

web files
cogdumper http --help
Usage: cogdumper http [OPTIONS]

  COGDumper cli for web hosted dataset.

Options:
  --server TEXT       server e.g. http://localhost:8080  [required]
  --path TEXT         server path
  --resource TEXT     server resource
  --output DIRECTORY  local output directory
  --xyz INTEGER...    xyz tile coordinates where z is the overview level
  --version           Show the version and exit.
  --help              Show this message and exit.

e.g. cogdumper http --server http://localhost:8080 --path data --resource cog.tif

S3 files
cogdumper s3 --help
Usage: cogdumper s3 [OPTIONS]

  COGDumper cli for AWS S3 hosted dataset

Options:
  --bucket TEXT       AWS S3 bucket  [required]
  --key TEXT          AWS S3 key  [required]
  --output DIRECTORY  local output directory
  --xyz INTEGER...    xyz tile coordinates where z is the overview level
  --help              Show this message and exit.

e.g. cogdumper s3 --bucket bucket_name --key key_name/image.tif --xyz 0 0 0

cogdumper's People

Contributors

matiskay avatar normanb avatar vincentsarago avatar yoninachmany avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cogdumper's Issues

COGDumper downloads the whole header

COGDumper downloads the whole header. See this example which is a BIGTIFF with a 400mb header!

cogdumper http --server https://s3-ap-southeast-2.amazonaws.com --path linz-test-data --resource auckland_urban_2017_0.075m_cog.tif --xyz 0 0 0

Note GDAL when built with internal libtiff only downloads the parts of the header required for data or overview tiles requests

Implement chunk or stream read to reduce the number of `read` call

Right now when reading the header we have to loop through each FID to get each FID metadata.

# for JPEG we need to read all IFDs, they are at the front of the file
for ifd in self.ifds():

While this permit to read only part we need to determine what will be the offsets for the data and mask part, this is also not efficient, resulting in 10s of small read calls.

cogdumper s3 --bucket mapbox --key playground/vincent/y.tif --xyz 0 0 0
Read Header
read 0 3
read 4 7
read 8 9
read 10 251
read 516 771
read 260 515
read 826 967
read 250 253

Read IFD
read 1378 1379
read 1380 1549
read 1808 2063
read 1552 1807
read 1548 1551

Read IFD
read 2064 2065
read 2066 2259
read 2332 2395
read 2268 2331
read 2450 2591
read 2258 2261

Read IFD
read 2592 2593
read 2594 2787
read 2812 2827
read 2796 2811
read 2882 3023
read 2786 2789

Read IFD
read 3024 3025
read 3026 3219
read 3282 3423
read 3218 3221

Read IFD
read 3424 3425
read 3426 3619
read 3682 3823
read 3618 3621

Read IFD
read 3824 3825
read 3826 4019
read 4082 4223
read 4018 4021

Read IFD
read 4224 4225
read 4226 4419
read 4482 4623
read 4418 4421

Read IFD
read 4624 4625
read 4626 4795
read 4862 4925
read 4798 4861
read 4794 4797

Read IFD
read 4926 4927
read 4928 5097
read 5116 5131
read 5100 5115
read 5096 5099

Read IFD
read 5132 5133
read 5134 5303
read 5302 5305

Read IFD
read 5306 5307
read 5308 5477
read 5476 5479

Read IFD
read 5480 5481
read 5482 5651
read 5650 5653

Read IFD
read 5654 5655
read 5656 5825
read 5824 5827

Read IFD
read 883331 908790
read 2341520 2341571

I don't remember exactly but it seems that GDAL is reading the first 16ko of the file and then determine all it needs. I think applying the same logic could be nice.

cc @normanb @sgillies

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.