Git Product home page Git Product logo

Comments (18)

meyer1994 avatar meyer1994 commented on August 19, 2024 1

Weird. The error persists :(

Just tried with the latest layer:

arn:aws:lambda:eu-central-1:524387336408:layer:gdal30-py37-geo:5

When trying to open .gml it returns None. So it fails right after. I couldn't reproduce the first bug though. The one with:

Xerces or Expat support not configured into GDAL/OGR

Sample lambda:

from osgeo import gdal, ogr

gdal.UseExceptions()
ogr.UseExceptions()

gdal.SetConfigOption('AWS_DEFAULT_REGION', 'eu-central-1')
gdal.SetConfigOption('AWS_REQUEST_PAYER', 'requester')

BUCKET = 'sentinel-s2-l1c'
KEY = 'tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.gml'


def handler(event, context):
    key = f'/vsis3/{BUCKET}/{KEY}'
    mask = ogr.Open(key)

    print('Layers:', mask.GetLayerCount())

    layer = mask.GetLayer()
    print('Features:', layer.GetFeatureCount())

from amazonlinux.

vincentsarago avatar vincentsarago commented on August 19, 2024 1

I've try with remotepixel/amazonlinux:gdal3.0-py3.7-geo

docker run \
--name dockerimage \
-p 8000:8000 \
--env AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID} \
--env AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
--env AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
--env AWS_REGION=${AWS_REGION} \
--volume $(pwd)/:/local \
--rm -it remotepixel/amazonlinux:gdal3.0-py3.7-geo bash

bash-4.2# python

from osgeo import gdal, ogr

gdal.UseExceptions()
ogr.UseExceptions()

gdal.SetConfigOption('AWS_DEFAULT_REGION', 'eu-central-1')
gdal.SetConfigOption('AWS_REQUEST_PAYER', 'requester')

BUCKET = 'sentinel-s2-l1c'
KEY = 'tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.gml'

key = f'/vsis3/{BUCKET}/{KEY}'
mask = ogr.Open(key)

print('Layers:', mask.GetLayerCount())
Layers: 1

layer = mask.GetLayer()

print('Features:', layer.GetFeatureCount())
Features: 1

from amazonlinux.

vincentsarago avatar vincentsarago commented on August 19, 2024

@meyer1994 thanks again for the issue.
I should have check that, and you are also right about libsqlite3.

The only solution I can think of is to move or build those libraries at build time to /opt/lib

I don't have time this week to work on this sadly

from amazonlinux.

vincentsarago avatar vincentsarago commented on August 19, 2024

@meyer1994 in fact it goes down to understand what libraries are available in lambda runtime.

we are using lambci/lambda-base:build docker image to make sure we compile GDAL is the exact same environement as in lambda so if libsqlite3.so.0 is link to gdal.so it means it might be available in lambda ....

Thus we just need to copy in /opt/lib the libraries that are not in lambda (proj, goes ...)

I'll continue to investigate

from amazonlinux.

vincentsarago avatar vincentsarago commented on August 19, 2024

@meyer1994 just run ldd libgdal.so on lambda runtime directly and seems that everything is almost here

	linux-vdso.so.1 =>  (0x00007ffee3f9b000)
	libcrypto.so.10 => /var/lang/lib/libcrypto.so.10 (0x00007f6407733000)
	libgeos_c.so.1 => /opt/lib/libgeos_c.so.1 (0x00007f6407500000)
	libwebp.so.7 => /opt/lib/libwebp.so.7 (0x00007f6407293000)
	libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x00007f6406fe0000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f6406db7000)
	libopenjp2.so.7 => /opt/lib/libopenjp2.so.7 (0x00007f6406b5d000)
	libjpeg.so.62 => /opt/lib/libjpeg.so.62 (0x00007f64068cc000)
	libpng16.so.16 => /opt/lib/libpng16.so.16 (0x00007f640669a000)
	libzstd.so.1 => /opt/lib/libzstd.so.1 (0x00007f640641b000)
	libproj.so.15 => /opt/lib/libproj.so.15 (0x00007f6405f8f000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f6405d79000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6405b5d000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f6405955000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f6405751000)
	libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007f64054ca000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f6405145000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f6404e43000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6404c2d000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f6404860000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f6408db0000)
	libgeos-3.7.2.so => /opt/lib/libgeos-3.7.2.so (0x00007f64044b0000)
	libnghttp2.so.14 => /usr/lib64/libnghttp2.so.14 (0x00007f640428b000)
	libidn2.so.0 => /usr/lib64/libidn2.so.0 (0x00007f640406a000)
	libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x00007f6403e42000)
	libpsl.so.0 => /usr/lib64/libpsl.so.0 (0x00007f6403bcd000)
	libssl.so.10 => /var/lang/lib/libssl.so.10 (0x00007f640395c000)
	libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f640370f000)
	libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f6403426000)
	libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f640320b000)
	libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f6403008000)
	libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f6402db5000)
	liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f6402ba6000)
	libunistring.so.0 => /usr/lib64/libunistring.so.0 (0x00007f6402890000)
	libicuuc.so.50 => /usr/lib64/libicuuc.so.50 (0x00007f640251a000)
	libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f640230b000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f6402108000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f6401eef000)
	libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x00007f6401cd4000)
	libssl3.so => /usr/lib64/libssl3.so (0x00007f6401a82000)
	libsmime3.so => /usr/lib64/libsmime3.so (0x00007f640185b000)
	libnss3.so => /usr/lib64/libnss3.so (0x00007f6401536000)
	libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f6401307000)
	libplds4.so => /lib64/libplds4.so (0x00007f6401103000)
	libplc4.so => /lib64/libplc4.so (0x00007f6400efe000)
	libnspr4.so => /lib64/libnspr4.so (0x00007f6400cc0000)
	libicudata.so.50 => /usr/lib64/libicudata.so.50 (0x00007f63ff6ed000)
	libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f63ff4cc000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f63ff295000)
	libfreebl3.so => /lib64/libfreebl3.so (0x00007f63ff093000)

the only differences I see when running this in remotepixel/amazonlinux:gdal3.0-py3.7 are:

libpcre.so.0
libxml2.so.2
liblzma.so.5

What we can see is first, we might be missing libxml in runtime and Xerces is definitely not in lambda

from amazonlinux.

vincentsarago avatar vincentsarago commented on August 19, 2024

@meyer1994 I cannot reproduce the error you mentioned, can you share more about how you created your layer or pacakge ?

from amazonlinux.

meyer1994 avatar meyer1994 commented on August 19, 2024

I am trying to reproduce it, unsuccessfully. I will try to create a reproducible example in the next couple of days.

Weirdly, my stack is working based on waht I said. It wasn't working without copying the libexpat file to the layer...

Weirdly, again, I have not copied the libxml2.so.2. It seems that it would be necessairy, GML is XML based.

from amazonlinux.

vincentsarago avatar vincentsarago commented on August 19, 2024

after other checks it seems that libxml2 is present on lambda runtime env 🤔.

from amazonlinux.

vincentsarago avatar vincentsarago commented on August 19, 2024

@meyer1994 with the latest update I don't see how this could still be a problem. Can you test with the latest lambda layer or docker images and close here if everything works?

from amazonlinux.

vincentsarago avatar vincentsarago commented on August 19, 2024

@meyer1994 thanks
I'm not sure what's going on.
When running the code on the docker image it works just file but on the lambda it fails.
both libgdal.so are linked to the same libs so it should work
there is maybe some environement variables that needs to be configured ?

from amazonlinux.

meyer1994 avatar meyer1994 commented on August 19, 2024

You sure it works fine?

I am running it on remotepixel/amazonlinux:gdal3.0-py3.7 and it is returning None when opening the .gml.

from amazonlinux.

meyer1994 avatar meyer1994 commented on August 19, 2024

I tried somethings around, with no avail...

It seems to me that adding the libraries to the layer by hand, using:

zip -r9 --symlinks /tmp/${PACKAGE_NAME}.zip /lib64/libexpat.so*

Is the best solution for now.

If you want you can close this issue and reopen it when it seems fit.

from amazonlinux.

vincentsarago avatar vincentsarago commented on August 19, 2024

🤔 but it seems to me that /lib64/libexpat.so is already in the environment!

are you using python3.7 ?

from amazonlinux.

meyer1994 avatar meyer1994 commented on August 19, 2024

I am using python3.7.

but it seems to me that /lib64/libexpat.so is already in the environment!

Yes, it is. But it is in the wrong place. When you add the file to the layer, it will be unzipped in /opt/lib or /opt/lib64 when the layer is deployed. I think your built version of gdal searches for it on /opt.

Using this command in the layer creation:

$ zip -r9 --symlinks /tmp/${PACKAGE_NAME}.zip /lib64/libexpat.so*

After the layer is deployed into the lambda, the library will be in /opt/lib64.

from amazonlinux.

vincentsarago avatar vincentsarago commented on August 19, 2024

When you add the file to the layer

we don't ship libexpact with the layer.

I think your built version of gdal searches for it on /opt.

ldd libgdal.so shows it is linked to /lib64/libexpat.so so I'm not sure why it can't find it 🤔

from amazonlinux.

meyer1994 avatar meyer1994 commented on August 19, 2024

we don't ship libexpact with the layer.

Yes yes. I was trying to say when someone adds the file to the layer (like I did). My bad.

ldd libgdal.so shows it is linked to /lib64/libexpat.so so I'm not sure why it can't find it 🤔

I don't know what else to do. For now, my version is working with the copied libexpat. Feel free to close it if you think is not worth it.

from amazonlinux.

meyer1994 avatar meyer1994 commented on August 19, 2024

Ok. I am ashamed.

The new layer works!!! The problem was with IAM roles... Adding permissions to access sentinel bucket made it work fine.

See serverless.yml:

service: layertest

provider:
  name: aws
  runtime: python3.7
  stage: dev
  versionFunctions: false
  region: eu-central-1

  environment:
    PROJ_LIB: '/opt/share/proj'
    GEOS_CONFIG: '/opt/bin/geos-config'
    GDAL_DATA: '/opt/share/gdal'
    GDAL_CONFIG: '/opt/bin/gdal-config'
    GDAL_CACHEMAX: '512'
    GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: 'YES'
    GDAL_HTTP_MULTIPLEX: 'YES'
    GDAL_HTTP_VERSION: '2'
    GDAL_DISABLE_READDIR_ON_OPEN: 'EMPTY_DIR'
    VSI_CACHE: 'true'
    VSI_CACHE_SIZE: '536870912'
    CPL_TMPDIR: '/tmp'
    CPL_VSIL_CURL_ALLOWED_EXTENSIONS: '.tif,.jp2,.gml'

  iamRoleStatements:
    - Effect: Allow
      Action:
        - s3:GetObject
      Resource:
        - arn:aws:s3:::sentinel-s2-l1c*

package:
  include:
    - handler.py

functions:
  layer:
    handler: handler.handler
    layers:
      - arn:aws:lambda:eu-central-1:524387336408:layer:gdal30-py37-geo:5

And handler.py:

import glob

from osgeo import gdal, ogr

gdal.UseExceptions()
ogr.UseExceptions()

gdal.SetConfigOption('AWS_DEFAULT_REGION', 'eu-central-1')
gdal.SetConfigOption('AWS_REQUEST_PAYER', 'requester')

BUCKET = 'sentinel-s2-l1c'
KEY = 'tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.gml'

def handler(event, context):
    libs = glob.glob('/opt/libs*/*')
    libs = sorted(libs)
    print('\n'.join(libs))

    key = f'/vsis3/{BUCKET}/{KEY}'
    mask = ogr.Open(key)

    print('Layers:', mask.GetLayerCount())

    layer = mask.GetLayer()
    print('Features:', layer.GetFeatureCount())

After deploying:

$ sls invoke -f layer -l
null
--------------------------------------------------------------------
START RequestId: a4121a73-5241-40ab-b867-62f4c3e4f735 Version: $LATEST

Warning 1: HTTP response code on https://sentinel-s2-l1c.s3.amazonaws.com/tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.resolved.gml: 403
Layers: 1
Features: 1
END RequestId: a4121a73-5241-40ab-b867-62f4c3e4f735
REPORT RequestId: a4121a73-5241-40ab-b867-62f4c3e4f735	Duration: 265.57 ms	Billed Duration: 300 ms	Memory Size: 1024 MB	Max Memory Used: 111 MB	Init Duration: 358.54 ms

from amazonlinux.

vincentsarago avatar vincentsarago commented on August 19, 2024

🎉 nothing to be ashamed, this happened to me so many time 😄
Glad you figured out and can't wait to see what you are building!

from amazonlinux.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.