remotepixel / amazonlinux Goto Github PK

This still need to be done, but I'm looking at a nice way to publish the layers via CircleCI only if some tests pass

Hi @vincentsarago this repo is awesome, thanks v much.

After reading some of your COG medium posts and trying a few things out I would really like to use GDAL3.1's new COG driver to get Lambda to generate COGs for every file I load into an S3 bucket.

You Dockerfiles uses https://github.com/OSGeo/gdal/archive/ but there doesn't seem to be one for the latest dev version.

Do you know a way or point me in the direction how I could modify your Dockerfile to use the dev version?

Many thank for your help

https://github.com/OSGeo/gdal/blob/v3.0.4/gdal/NEWS
https://github.com/OSGeo/gdal/blob/v2.4.4/gdal/NEWS

I am having problems when trying to use gml drivers for OGR. It seems that this build does not include Expat or Xerces. See the following error when running ogrinfo into a sample file from Sentinel 2:

$ curl -s https://pastebin.com/raw/DWfrMySu > out.gml
$ ogrinfo out.gml 
ERROR 1: Unable to create Xerces C++ or Expat based GML reader, Xerces or Expat support not configured into GDAL/OGR.
ERROR 1: File out.gml appears to be GML but the GML reader can't
be instantiated, likely because Xerces or Expat support was
not configured in.
FAILURE:
Unable to open datasource `out.gml' with the following drivers.
  -> PDS4
  -> JP2OpenJPEG
  -> PDF
  -> MBTiles
  -> EEDA
  -> ESRI Shapefile
...

This was tested in both versions, 2.4 and 3.0.

After checking the dockerfiles, in both versions, 2.4 and 3.0, I saw that gdal was built explicitly without both libraries.

RUN cd /tmp/gdal \
      ...
      --without-expat \
      ...
      --without-xerces \
      ...

Is there a reason for it? I know that the intetion of this repo is to be a lightweight build. But it seems weird for me. For example, Sentinel-2 has all its masks stored in GML format in the open registry.

Would it be possible to include, at least, one of them to the build?

Thanks in advance.

I did add the grid for GDAL 3 image but not for GDAL 2 🤦‍♂

PROJ_LIB need to be set for rasterio

3.0.3 && 2.4.4 https://twitter.com/EvenRouault/status/1216716727159005184

The aim of this project is to create usuable and light GDAL docker image and AWS Lambda layer. Following this statement, I'd like to make some change to the available drivers we currently allow by following https://github.com/OSGeo/gdal/blob/master/gdal/docker/alpine-ultrasmall/Dockerfile#L110-L196

@meyer1994 I'll make sure to keep GML driver but if you are using others please let me know.

Hi @vincentsarago, thanks for your work on this (and other things I use).

I'm trying to install a module for use alongside the lambda layer, following the steps outlined in the respective README section.

I've ran into some issues with dependencies that are available in the docker image (so no issues during build or package) but they are missing in the actual lambda function.

Here's an specific example:

The module I'm installing (replacing mercantile in your example) depends among other things on pyyaml. When running pip install I get the corresponding log message:

Requirement already satisfied: pyyaml in /var/lang/lib/python3.7/site-packages ...

and everything works fine locally. But the contents of /var/lang/lib/python3.7/site-packages don't make it into the zip, and they are not available in the lambda function which includes the layer.

Of course I could just try to find and install all dependencies into /var/task so they get zipped up. But that seems to be a sub-optimal solution. Any ideas? Apologies if I'm missing something obvious.

Thanks!

let's remove this layer here and make a new repo

After PR #12 got merged, I tried to create a layer with the script located here. But I still got the same errors:

Xerces or Expat support not configured into GDAL/OGR

After some intense search I've stumbled upon this doc which led me to ldd.

From inside the docker image I ran $ ldd /opt/lib/libgdal.so. Lo and behold, lots of info came out of it:

bash-4.2# ldd /opt/lib/libgdal.so
	linux-vdso.so.1 =>  (0x00007ffc645e8000)
	libcrypto.so.10 => /var/lang/lib/libcrypto.so.10 (0x00007fb9df7f1000)
	libgeos_c.so.1 => /opt/lib/libgeos_c.so.1 (0x00007fb9df5be000)
	libwebp.so.7 => /opt/lib/libwebp.so.7 (0x00007fb9df351000)
	libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x00007fb9df09e000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007fb9dee75000)  # BOOOM!!!
	libopenjp2.so.7 => /opt/lib/libopenjp2.so.7 (0x00007fb9dec1b000)
	libjpeg.so.62 => /opt/lib/libjpeg.so.62 (0x00007fb9de98a000)
	libpng16.so.16 => /opt/lib/libpng16.so.16 (0x00007fb9de758000)
	libzstd.so.1 => /opt/lib/libzstd.so.1 (0x00007fb9de4d9000)
	libproj.so.15 => /opt/lib/libproj.so.15 (0x00007fb9de04d000)
	libz.so.1 => /lib64/libz.so.1 (0x00007fb9dde37000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb9ddc1b000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fb9dda13000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fb9dd80f000)
	libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007fb9dd588000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fb9dd203000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fb9dcf01000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb9dcceb000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fb9dc91e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fb9e0e6e000)
	libgeos-3.7.2.so => /opt/lib/libgeos-3.7.2.so (0x00007fb9dc56e000)
	libnghttp2.so.14 => /usr/lib64/libnghttp2.so.14 (0x00007fb9dc349000)
	libidn2.so.0 => /usr/lib64/libidn2.so.0 (0x00007fb9dc128000)
	libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x00007fb9dbf00000)
	libpsl.so.0 => /usr/lib64/libpsl.so.0 (0x00007fb9dbc8b000)
	libssl.so.10 => /var/lang/lib/libssl.so.10 (0x00007fb9dba1a000)
	libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007fb9db7cd000)
	libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007fb9db4e4000)
	libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007fb9db2c9000)
	libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007fb9db0c6000)
	libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007fb9dae73000)
	liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007fb9dac64000)
	libunistring.so.0 => /usr/lib64/libunistring.so.0 (0x00007fb9da94e000)
	libicuuc.so.50 => /usr/lib64/libicuuc.so.50 (0x00007fb9da5d8000)
	libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007fb9da3c9000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fb9da1c6000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fb9d9fad000)
	libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x00007fb9d9d92000)
	libssl3.so => /usr/lib64/libssl3.so (0x00007fb9d9b40000)
	libsmime3.so => /usr/lib64/libsmime3.so (0x00007fb9d9919000)
	libnss3.so => /usr/lib64/libnss3.so (0x00007fb9d95f4000)
	libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007fb9d93c5000)
	libplds4.so => /lib64/libplds4.so (0x00007fb9d91c1000)
	libplc4.so => /lib64/libplc4.so (0x00007fb9d8fbc000)
	libnspr4.so => /lib64/libnspr4.so (0x00007fb9d8d7e000)
	libicudata.so.50 => /usr/lib64/libicudata.so.50 (0x00007fb9d77ab000)
	libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007fb9d758a000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fb9d7353000)
	libfreebl3.so => /lib64/libfreebl3.so (0x00007fb9d7151000)

It appear that the build process for gdal does not place every library inside the /opt/lib dir. So, when creating the layer, we should add the libraries by hand (unfortunately).

The addition of the following line to the create-lambda-layer.sh file fixed the error for me:

zip -r9 --symlinks /tmp/${PACKAGE_NAME}.zip /lib64/libexpat.so*

However, I think there are more problems that may appear from this problem. For example, libsqlite3.so.0 not included by the script as well. I have not tested it. But it is entirely possible.

I did not make a PR because I am not sure how to fix this in a more generic way. Maybe adding each lib by hand is the best option, but I am not sure.

The latest commit created lambda package exceeding the 50Mb limit 😢

Starting in #10, we found a huge performance downgrade when doing reprojection (Warp) of dataset.
As pointed in #10 (comment) and #10 (comment) this only happens in the containers we are building (the performance in gdal official images are fine).

To Do

• Check PROJ compilation
• Check GDAL compilation

https://github.com/OSGeo/gdal/releases/tag/v3.0.1
https://github.com/OSGeo/gdal/releases/tag/v2.4.3

I've created a layer using the script. It throws an error when trying to import fiona.

[ERROR] Runtime.ImportModuleError: Unable to import module 'app': /opt/python/fiona/ogrext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: OSRFixup

I've created this layer using the script in the scripts directory. I used this image:

remotepixel/amazonlinux:gdal3.0-py3.7-full

To create the error I used a simple chalice lambda function:

import subprocess

# Testing imports
import fiona
import rasterio
import shapely
from osgeo import gdal, ogr, osr

from chalice import Chalice

app = Chalice(app_name='test')

@app.lambda_function()
def index(event, context):
    # Testing bin
    result = subprocess.run(['gdalinfo', '--formats'])
    assert result.returncode == 0
    return {'it': 'worked'}

Everything works fine when removing the fiona import.

Edit:

I've just read that fiona does not yet support gdal 3. So there is the problem. But, rasterio works and it does not support it yet aswell. Any ideas how to fix?

FYI, I'm going to remove all old GDAL 3.0 layers

I recently stumbled across this repo and it's exactly what I've been looking for to use ogr2ogr with Lambda. I've been able to get it running for some relatively conventional use cases (including reading a file in directly from an S3 bucket) but it looks like the Postgres driver isn't included in this build. Any tips on the best way to include this? I've been using the us-east-1 Python 3.7 layer

Here's my function:

def lambda_handler(event, context):
        # s3 upload triggers function
        srcBucket = os.environ['STORAGE_BUCKETNAME']   # get bucket from env
        keyURI = event['Records'][0]['s3']['object']['key']   # get key from event
        srcKey = unquote(keyURI)
        filename = keyURI.split('/')[-1]
        path = '/vsis3/'+srcBucket+'/'+srcKey   # create s3 input url for ogr2ogr
        commands = [
           'ogr2ogr', '-f', '"PostgreSQL"', 'PG:"host=<HOST> user=<USER> dbname=<DB> password=<PW>"', path
        ]
        subprocess.check_output(commands)
     
    return {
        'statusCode': 200,
        'body': json.dumps('Success)
    }

Specific error message is: Unable to find driver 'PostgreSQL'.

If I can get this to work, it would be phenomenal