ryansb / sklearn-build-lambda Goto Github PK
View Code? Open in Web Editor NEWBuild the numpy/scipy/scikitlearn packages and strip them down to run in Lambda
License: MIT License
Build the numpy/scipy/scikitlearn packages and strip them down to run in Lambda
License: MIT License
Whenever I run $ docker run -v $(pwd):/outputs -it amazonlinux:2016.09 \ /bin/bash /outputs/build.sh
most things seem to work, but then I get the message
no such option: --use-wheel
Then I try pip install wheel
while in the sklearn-build-lambda directory (not sure that matters), and I get the response
Requirement already satisfied: wheel in /Users/spencer/anaconda3/lib/python3.6/site-packages
So wheel is on my local machine, but not in the Docker container. I've researched wheel and how to install python modules within Docker, but no luck. Any help is much appreciated!
I followed well until this step:
$ docker run -v $(pwd):/outputs -it amazonlinux:2016.09
/bin/bash /outputs/build.sh
My local path is e:/Science_git_sklearn-build-lambda. What is the right value for $(pwd)?
Thank you.
Hello,
can you explain the sentence
"and add the lib directory so it can be used for shared libs"
what exactly is "lib"?
thanks.
Hey there, I was having trouble with numpy in my serverless project and came across this repo/blog post. I lifted the version of numpy from your zip file, along with the C dependencies included in your lib folder. However, when I deploy my serverless function I'm getting this error:
Traceback (most recent call last):
File "/var/task/add-return-items/handler.py", line 20, in handler
ctypes.cdll.LoadLibrary(os.path.join(d, f))
File "/usr/lib64/python2.7/ctypes/__init__.py", line 438, in LoadLibrary
return self._dlltype(name)
File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libquadmath.so.0: cannot open shared object file: No such file or directory
which seems to be related to this for loop:
import ctypes
for d, _, files in os.walk('vendored/lib'):
for f in files:
if f.endswith('.a'):
continue
ctypes.cdll.LoadLibrary(os.path.join(d, f))
As far as I can tell, it seems like ctypes needs libquadmath.so.0
, however this is one of the files included in the lib I'm trying to load. Any ideas on how to address this?
This would be a welcome addition to https://github.com/Miserlou/lambda-packages
I tried running the script on an Amazon Linux instance but the strip
part failed with a "Not enough room for program headers, try linking with -N" error.
BFD: /home/ec2-user/sklearn_build/lib64/python2.7/site-packages/numpy/.libs/stV4bcuq: Not enough room for program headers, try linking with -N
strip:/home/ec2-user/sklearn_build/lib64/python2.7/site-packages/numpy/.libs/stV4bcuq[.note.gnu.build-id]: Bad value
Do you have an idea how to fix this problem?
Thanks
find . -name 'test*' -exec rm {} \;
and now the zip file seems small enough. would it be an idea to add this in the bash script? id be happy to have a go at it (but i have little experience with docker)Hi, thanks for setting this up it looks very useful and promising for my teams
lambda use. I've been the below blog post, but I am getting a few errors when I run the docker command.
https://serverlesscode.com/post/scikitlearn-with-amazon-linux-container/
docker run -v $(pwd):/outputs -it amazonlinux:2017.09 \ /bin/bash /outputs/build.sh
Here is the stacktrace
One of the configured repositories failed (amzn-main-Base),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Disable the repository, so yum won't use it by default. Yum will then
just ignore the repository until you permanently enable it again or use
--enablerepo for temporary usage:
yum-config-manager --disable amzn-main
4. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=amzn-main.skip_if_unavailable=true
failure: repodata/filelists.sqlite.bz2 from amzn-main: [Errno 256] No more mirrors to try.
http://packages.us-west-1.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: [Errno 12] Timeout on http://packages.us-west-1.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 5 seconds')
http://packages.eu-west-1.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: [Errno 12] Timeout on http://packages.eu-west-1.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 5 seconds')
http://packages.ap-northeast-1.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: [Errno 12] Timeout on http://packages.ap-northeast-1.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 5 seconds')
http://packages.ap-northeast-2.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: [Errno 12] Timeout on http://packages.ap-northeast-2.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 5 seconds')
http://packages.sa-east-1.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: [Errno 12] Timeout on http://packages.sa-east-1.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 5 seconds')
http://packages.ap-southeast-2.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: [Errno 12] Timeout on http://packages.ap-southeast-2.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 5 seconds')
http://packages.eu-central-1.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: [Errno 12] Timeout on http://packages.eu-central-1.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 5 seconds')
http://packages.us-west-2.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: [Errno 12] Timeout on http://packages.us-west-2.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 5 seconds')
http://packages.ap-southeast-1.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: [Errno 12] Timeout on http://packages.ap-southeast-1.amazonaws.com/2017.09/main/154a6dd467e2/x86_64/repodata/filelists.sqlite.bz2?instance_id=fail®ion=HTTPError: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 5 seconds')```
I removed it manually from the pip statements in the build script per this:
https://stackoverflow.com/questions/49835842/docker-no-such-option-use-wheel
It worked well for me, although my /outputs/venv.zip is 46M which is too large for me because I have other dependencies to add - but not your problem :-) ).
Hi,
are those instructions works on windows 7?
Thanks
Hi
Apologies in advance, I'm new to Docker.
I suspect this has nothing to do with your script, but I'm not able to run yum
with the latest amazonlinux
image. The same command works just fine if I run on the latest version of centos
.
$ docker run -it amazonlinux yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:
No module named yum
Please install a package which provides this module, or
verify that the module is installed correctly.
It's possible that the above module doesn't match the
current version of Python, which is:
2.7.12 (default, Sep 1 2016, 22:14:00)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)]
If you cannot solve this problem yourself, please go to
the yum faq at:
http://yum.baseurl.org/wiki/Faq
In fact, the python
version seems pretty crippled as well:
$ docker run -it amazonlinux python -c 'import os.path'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named path
I'm running docker version 17.03.1-ce-mac5, and amazonlinux with image ID 8ae6f52035b5
I get this massage when i write the command
$ docker run -v $(pwd):/outputs -it amazonlinux:2016.09
/bin/bash /outputs/build.sh
what can be the problem here ?
thanks
AWS Lambda now handle Python 3.6 and i want to use it, but do not sure if is simple to extend it to use Python 3.6 instead.
Any guide would be appreciated
Thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.