Comments (18)
Weird. The error persists :(
Just tried with the latest layer:
arn:aws:lambda:eu-central-1:524387336408:layer:gdal30-py37-geo:5
When trying to open .gml
it returns None
. So it fails right after. I couldn't reproduce the first bug though. The one with:
Xerces or Expat support not configured into GDAL/OGR
Sample lambda:
from osgeo import gdal, ogr
gdal.UseExceptions()
ogr.UseExceptions()
gdal.SetConfigOption('AWS_DEFAULT_REGION', 'eu-central-1')
gdal.SetConfigOption('AWS_REQUEST_PAYER', 'requester')
BUCKET = 'sentinel-s2-l1c'
KEY = 'tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.gml'
def handler(event, context):
key = f'/vsis3/{BUCKET}/{KEY}'
mask = ogr.Open(key)
print('Layers:', mask.GetLayerCount())
layer = mask.GetLayer()
print('Features:', layer.GetFeatureCount())
from amazonlinux.
I've try with remotepixel/amazonlinux:gdal3.0-py3.7-geo
docker run \
--name dockerimage \
-p 8000:8000 \
--env AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID} \
--env AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
--env AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
--env AWS_REGION=${AWS_REGION} \
--volume $(pwd)/:/local \
--rm -it remotepixel/amazonlinux:gdal3.0-py3.7-geo bash
bash-4.2# python
from osgeo import gdal, ogr
gdal.UseExceptions()
ogr.UseExceptions()
gdal.SetConfigOption('AWS_DEFAULT_REGION', 'eu-central-1')
gdal.SetConfigOption('AWS_REQUEST_PAYER', 'requester')
BUCKET = 'sentinel-s2-l1c'
KEY = 'tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.gml'
key = f'/vsis3/{BUCKET}/{KEY}'
mask = ogr.Open(key)
print('Layers:', mask.GetLayerCount())
Layers: 1
layer = mask.GetLayer()
print('Features:', layer.GetFeatureCount())
Features: 1
from amazonlinux.
@meyer1994 thanks again for the issue.
I should have check that, and you are also right about libsqlite3.
The only solution I can think of is to move or build those libraries at build time to /opt/lib
I don't have time this week to work on this sadly
from amazonlinux.
@meyer1994 in fact it goes down to understand what libraries are available in lambda runtime.
we are using lambci/lambda-base:build
docker image to make sure we compile GDAL is the exact same environement as in lambda so if libsqlite3.so.0 is link to gdal.so it means it might be available in lambda ....
Thus we just need to copy in /opt/lib
the libraries that are not in lambda (proj, goes ...)
I'll continue to investigate
from amazonlinux.
@meyer1994 just run ldd libgdal.so
on lambda runtime directly and seems that everything is almost here
linux-vdso.so.1 => (0x00007ffee3f9b000)
libcrypto.so.10 => /var/lang/lib/libcrypto.so.10 (0x00007f6407733000)
libgeos_c.so.1 => /opt/lib/libgeos_c.so.1 (0x00007f6407500000)
libwebp.so.7 => /opt/lib/libwebp.so.7 (0x00007f6407293000)
libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x00007f6406fe0000)
libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f6406db7000)
libopenjp2.so.7 => /opt/lib/libopenjp2.so.7 (0x00007f6406b5d000)
libjpeg.so.62 => /opt/lib/libjpeg.so.62 (0x00007f64068cc000)
libpng16.so.16 => /opt/lib/libpng16.so.16 (0x00007f640669a000)
libzstd.so.1 => /opt/lib/libzstd.so.1 (0x00007f640641b000)
libproj.so.15 => /opt/lib/libproj.so.15 (0x00007f6405f8f000)
libz.so.1 => /lib64/libz.so.1 (0x00007f6405d79000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6405b5d000)
librt.so.1 => /lib64/librt.so.1 (0x00007f6405955000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f6405751000)
libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007f64054ca000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f6405145000)
libm.so.6 => /lib64/libm.so.6 (0x00007f6404e43000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6404c2d000)
libc.so.6 => /lib64/libc.so.6 (0x00007f6404860000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6408db0000)
libgeos-3.7.2.so => /opt/lib/libgeos-3.7.2.so (0x00007f64044b0000)
libnghttp2.so.14 => /usr/lib64/libnghttp2.so.14 (0x00007f640428b000)
libidn2.so.0 => /usr/lib64/libidn2.so.0 (0x00007f640406a000)
libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x00007f6403e42000)
libpsl.so.0 => /usr/lib64/libpsl.so.0 (0x00007f6403bcd000)
libssl.so.10 => /var/lang/lib/libssl.so.10 (0x00007f640395c000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f640370f000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f6403426000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f640320b000)
libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f6403008000)
libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f6402db5000)
liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f6402ba6000)
libunistring.so.0 => /usr/lib64/libunistring.so.0 (0x00007f6402890000)
libicuuc.so.50 => /usr/lib64/libicuuc.so.50 (0x00007f640251a000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f640230b000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f6402108000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f6401eef000)
libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x00007f6401cd4000)
libssl3.so => /usr/lib64/libssl3.so (0x00007f6401a82000)
libsmime3.so => /usr/lib64/libsmime3.so (0x00007f640185b000)
libnss3.so => /usr/lib64/libnss3.so (0x00007f6401536000)
libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f6401307000)
libplds4.so => /lib64/libplds4.so (0x00007f6401103000)
libplc4.so => /lib64/libplc4.so (0x00007f6400efe000)
libnspr4.so => /lib64/libnspr4.so (0x00007f6400cc0000)
libicudata.so.50 => /usr/lib64/libicudata.so.50 (0x00007f63ff6ed000)
libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f63ff4cc000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f63ff295000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007f63ff093000)
the only differences I see when running this in remotepixel/amazonlinux:gdal3.0-py3.7
are:
libpcre.so.0
libxml2.so.2
liblzma.so.5
What we can see is first, we might be missing libxml
in runtime and Xerces
is definitely not in lambda
from amazonlinux.
@meyer1994 I cannot reproduce the error you mentioned, can you share more about how you created your layer or pacakge ?
from amazonlinux.
I am trying to reproduce it, unsuccessfully. I will try to create a reproducible example in the next couple of days.
Weirdly, my stack is working based on waht I said. It wasn't working without copying the libexpat
file to the layer...
Weirdly, again, I have not copied the libxml2.so.2
. It seems that it would be necessairy, GML is XML based.
from amazonlinux.
after other checks it seems that libxml2 is present on lambda runtime env 🤔.
from amazonlinux.
@meyer1994 with the latest update I don't see how this could still be a problem. Can you test with the latest lambda layer or docker images and close here if everything works?
from amazonlinux.
@meyer1994 thanks
I'm not sure what's going on.
When running the code on the docker image it works just file but on the lambda it fails.
both libgdal.so are linked to the same libs so it should work
there is maybe some environement variables that needs to be configured ?
from amazonlinux.
You sure it works fine?
I am running it on remotepixel/amazonlinux:gdal3.0-py3.7
and it is returning None
when opening the .gml
.
from amazonlinux.
I tried somethings around, with no avail...
It seems to me that adding the libraries to the layer by hand, using:
zip -r9 --symlinks /tmp/${PACKAGE_NAME}.zip /lib64/libexpat.so*
Is the best solution for now.
If you want you can close this issue and reopen it when it seems fit.
from amazonlinux.
🤔 but it seems to me that /lib64/libexpat.so
is already in the environment!
are you using python3.7 ?
from amazonlinux.
I am using python3.7.
but it seems to me that /lib64/libexpat.so is already in the environment!
Yes, it is. But it is in the wrong place. When you add the file to the layer, it will be unzipped in /opt/lib
or /opt/lib64
when the layer is deployed. I think your built version of gdal searches for it on /opt
.
Using this command in the layer creation:
$ zip -r9 --symlinks /tmp/${PACKAGE_NAME}.zip /lib64/libexpat.so*
After the layer is deployed into the lambda, the library will be in /opt/lib64
.
from amazonlinux.
When you add the file to the layer
we don't ship libexpact
with the layer.
I think your built version of gdal searches for it on /opt.
ldd libgdal.so
shows it is linked to /lib64/libexpat.so
so I'm not sure why it can't find it 🤔
from amazonlinux.
we don't ship libexpact with the layer.
Yes yes. I was trying to say when someone adds the file to the layer (like I did). My bad.
ldd libgdal.so
shows it is linked to/lib64/libexpat.so
so I'm not sure why it can't find it 🤔
I don't know what else to do. For now, my version is working with the copied libexpat
. Feel free to close it if you think is not worth it.
from amazonlinux.
Ok. I am ashamed.
The new layer works!!! The problem was with IAM roles... Adding permissions to access sentinel bucket made it work fine.
See serverless.yml
:
service: layertest
provider:
name: aws
runtime: python3.7
stage: dev
versionFunctions: false
region: eu-central-1
environment:
PROJ_LIB: '/opt/share/proj'
GEOS_CONFIG: '/opt/bin/geos-config'
GDAL_DATA: '/opt/share/gdal'
GDAL_CONFIG: '/opt/bin/gdal-config'
GDAL_CACHEMAX: '512'
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: 'YES'
GDAL_HTTP_MULTIPLEX: 'YES'
GDAL_HTTP_VERSION: '2'
GDAL_DISABLE_READDIR_ON_OPEN: 'EMPTY_DIR'
VSI_CACHE: 'true'
VSI_CACHE_SIZE: '536870912'
CPL_TMPDIR: '/tmp'
CPL_VSIL_CURL_ALLOWED_EXTENSIONS: '.tif,.jp2,.gml'
iamRoleStatements:
- Effect: Allow
Action:
- s3:GetObject
Resource:
- arn:aws:s3:::sentinel-s2-l1c*
package:
include:
- handler.py
functions:
layer:
handler: handler.handler
layers:
- arn:aws:lambda:eu-central-1:524387336408:layer:gdal30-py37-geo:5
And handler.py
:
import glob
from osgeo import gdal, ogr
gdal.UseExceptions()
ogr.UseExceptions()
gdal.SetConfigOption('AWS_DEFAULT_REGION', 'eu-central-1')
gdal.SetConfigOption('AWS_REQUEST_PAYER', 'requester')
BUCKET = 'sentinel-s2-l1c'
KEY = 'tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.gml'
def handler(event, context):
libs = glob.glob('/opt/libs*/*')
libs = sorted(libs)
print('\n'.join(libs))
key = f'/vsis3/{BUCKET}/{KEY}'
mask = ogr.Open(key)
print('Layers:', mask.GetLayerCount())
layer = mask.GetLayer()
print('Features:', layer.GetFeatureCount())
After deploying:
$ sls invoke -f layer -l
null
--------------------------------------------------------------------
START RequestId: a4121a73-5241-40ab-b867-62f4c3e4f735 Version: $LATEST
Warning 1: HTTP response code on https://sentinel-s2-l1c.s3.amazonaws.com/tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.resolved.gml: 403
Layers: 1
Features: 1
END RequestId: a4121a73-5241-40ab-b867-62f4c3e4f735
REPORT RequestId: a4121a73-5241-40ab-b867-62f4c3e4f735 Duration: 265.57 ms Billed Duration: 300 ms Memory Size: 1024 MB Max Memory Used: 111 MB Init Duration: 358.54 ms
from amazonlinux.
🎉 nothing to be ashamed, this happened to me so many time 😄
Glad you figured out and can't wait to see what you are building!
from amazonlinux.
Related Issues (20)
- Bug ? Rasterio + WarpVRT bad performances for GDAL3.0 + Proj6 HOT 20
- Error when using GML driver HOT 3
- automatically publish lambda versions
- Using GDAL 3.1Dev HOT 4
- GDAL3.0 performances issue! HOT 2
- use`ultrasmall` compiling options from official GDAL images HOT 2
- Clearing old layer for GDAL 3.0
- add PROJ grid in GDAL 2
- PostgreSQL driver not found HOT 3
- new GDAL Versions HOT 1
- Package is too big for GDAL 3.0 HOT 1
- Gdal 3.04 and 2.4.4
- remove geo layer
- Dependency issues HOT 5
- Cannot import fiona. Urecognized symbol OSRFixup HOT 3
- Publish layers
- only ship python 3.7
- add layer size and version in docs HOT 1
- new GDAL release HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazonlinux.