amazon-archives / aws-lambda-container-image-converter Goto Github PK

Hi.

I use it in windows 10.

$ img2lambda -i lambda-tesseract-ocr_tesseract:latest -r ap-northeast-1 -o 
./output
2019/10/16 10:24:00 Parsing the docker image docker-daemon:lambda-tesseract-ocr_tesseract:latest
2019/10/16 10:24:00 Error loading image from docker engine: error during connect: Get https://%2F%2F.%2Fpipe%2Fdocker_engine/v1.22/images/get?names=lambda-tesseract-ocr_tesseract%3Alatest: dial tcp: lookup //./pipe/docker_engine: no such host

Docker TLS check is enabled

And the environment variable sets the value.

$ echo %DOCKER_HOST% 
tcp://localhost:2375

How can I verify the validity of the tcp://localhost:2375 ?

thanks.

I work on several projects that leverage different AWS profiles and I notice that when building a lambda runtime using this tool, the output/layers.json file produced references the account id of my default profile. I have no way to specify the desired profile name and so the layers are built in the wrong AWS account.

To work-around this, I have to shift around my ~/.aws/credentials file so that the account I want to use is [default], then shift it back after using img2lambda.

It would be helpful if img2lambda accepted a --profile argument similar to awscli.

The image library used supports reading OCI images, with the format "oci:path:tag"

For extracting each layer's blob, OCI images can be either tar or tar.gz, based on the layer's media type.

Hi.

I returned to my child and started example.

An error occurred when deploying serverless.

Serverless Error ---------------------------------------

  No php-example-hello.zip file found in the package path you provided.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              10.13.0
     Framework Version:         1.54.0
     Plugin Version:            3.1.2
     SDK Version:               2.1.2
     Components Core Version:   1.1.1
     Components CLI Version:    1.2.3

Steps

$ git clone https://github.com/awslabs/aws-lambda-container-image-converter

$ cd aws-lambda-container-image-converter/example

$ docker build -t lambda-php .
[Succeeded]

$ img2lambda -i lambda-php:latest -r ap-northeast-1 -o ./output
[Succeeded, and 2 layers created]

$ cd deploy

$ sls deploy
[above error]

Is php-example-hello.zip created automatically? Do I have to create it myself?
What should I put in if I need to create it myself?

Thanks.

Hi I run your commands but encounter this issue

./bin/local/img2lambda -i zaobao_drush:latest
2019/05/29 18:09:49 Parsing the docker image docker-daemon:zaobao_drush:latest
2019/05/29 18:10:12 Image docker-daemon:zaobao_drush:latest has 21 layers
2019/05/29 18:10:12 Did not create a Lambda layer file from image layer sha256:7bff100f35cb359a368537bb07829b055fe8e0b1cb01085a3a628ae9c187c7b8 (no relevant files found)
2019/05/29 18:10:12 Did not create a Lambda layer file from image layer sha256:b57f3f21236a88541bddd22ccc18a7e54b4eeb05756bc4cb295e8bd555a0fd81 (no relevant files found)
2019/05/29 18:10:12 Did not create a Lambda layer file from image layer sha256:30461eee5472d34544daa3acf0f0f1ec1a0c396406d676a06947010d8a86d5fb (no relevant files found)
2019/05/29 18:10:12 Did not create a Lambda layer file from image layer sha256:70c6d1eb014488ec8355c7878c0ead807da686e8173e0ebac0fdcf49e05e770d (no relevant files found)
2019/05/29 18:10:12 Did not create a Lambda layer file from image layer sha256:c4bf75cee0e1f53ff4dfe80da8173f8f3e88019c38866c386f53b0745d86dc5c (no relevant files found)
2019/05/29 18:10:12 Did not create a Lambda layer file from image layer sha256:c2feeff076f9a0996b6a6cf528a11a4dcb180240f45b2abdde02ed784616d6a9 (no relevant files found)
2019/05/29 18:10:12 Did not create a Lambda layer file from image layer sha256:a058b24dac4735e049423c9f493e1a7b4a2f14094fc3819458911786a9cb7881 (no relevant files found)
2019/05/29 18:10:12 Did not create a Lambda layer file from image layer sha256:d3c3132289ac03aec3b8beeed8bd85d15d3adf5fa046dce2a5c7e746e69299ae (no relevant files found)
2019/05/29 18:10:12 Did not create a Lambda layer file from image layer sha256:46388d31ffe7499e91d49eea7d7c27001fdde0625a5ac5560514a11c5fdfe2f4 (no relevant files found)
2019/05/29 18:10:13 Did not create a Lambda layer file from image layer sha256:869a18aad610c811f4483612e0a6a196c367df19dc34e5821804c3208beba688 (no relevant files found)
2019/05/29 18:10:13 Did not create a Lambda layer file from image layer sha256:824410e4d7f4c8df76f036877eb4067ece14337121d99c6d1a825c2a67806292 (no relevant files found)
2019/05/29 18:10:13 Did not create a Lambda layer file from image layer sha256:e18ced74f6ed82f8ed1372bc9b0a1e96227ba2d55a8b2e05e0a127f9b6d235c3 (no relevant files found)
2019/05/29 18:10:13 Did not create a Lambda layer file from image layer sha256:a17ad75d62ff5ee1d49fdc5664b6e2edb51aa3cf90f4fae83436f7953ce9e462 (no relevant files found)
2019/05/29 18:10:13 Did not create a Lambda layer file from image layer sha256:63e7383fa0979e6afc462ea3aa1545293309eff6f8b865f95d582c8fb4074078 (no relevant files found)
2019/05/29 18:10:13 Did not create a Lambda layer file from image layer sha256:e894024d6e00d09b6477e19ee80142f0dfda930cebf43cd4b156c5160a3aa08f (no relevant files found)
2019/05/29 18:10:13 Did not create a Lambda layer file from image layer sha256:5bef7d78aec0e3cbd2616deb4f6637b98674271c7ef77851878c216f09068395 (no relevant files found)
2019/05/29 18:10:13 Did not create a Lambda layer file from image layer sha256:e43d9404cc8887e9e921b127e3634ef8a9db550f14c119975254d3e93958ae31 (no relevant files found)
2019/05/29 18:10:13 Did not create a Lambda layer file from image layer sha256:17589be8568fd02d32d35067288b63d6a34d0f83f715630e6ae0316ab3363ed8 (no relevant files found)
2019/05/29 18:10:13 Did not create a Lambda layer file from image layer sha256:05a588806cee1bad47322cd80af99b332a98356d38b26b1b65060533fcf813df (no relevant files found)
2019/05/29 18:10:13 Did not create a Lambda layer file from image layer sha256:5e6f54d367375499a4662fc49911c7247e82e1ec571c201a648f22a8db6ce21e (no relevant files found)
2019/05/29 18:10:14 Did not create a Lambda layer file from image layer sha256:6aff14f8bff9b5a17c16c11ef08976904bf41d03339a834b1ad41750b5d256f5 (no relevant files found)
2019/05/29 18:10:14 Created 0 Lambda layer files for image docker-daemon:zaobao_drush:latest
2019/05/29 18:10:14 No compatible layers found in the image (likely nothing found in /opt)

I was wondering if the process failed due to too many layers. could you help? thanks

While running the tool under wsl (Windows Subsystem for Linux), it shows the following error -

Error loading image from docker engine: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I can run and build the docker images using the docker command.

Will you be able to change the local socket dependency ? docker info result is below

Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 11
Server Version: 18.09.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 96ec2177ae841256168fcf76954f7177af9446eb
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.125-linuxkit
Operating System: Docker for Windows
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.837GiB
Name: linuxkit-00155df88701
ID: 43BH:53TT:6MAF:XAIP:MHPN:KSQU:42NS:YEPJ:HWOL:SLZ2:MHJU:3FMH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 22
Goroutines: 47
System Time: 2019-01-24T06:18:10.6113087Z
EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

Currently, running the img2lambda tool multiple always updates the published layers to a new version number, even if the content hasn't change.

De-dupe Lambda layers before publishing them: compare local file's SHA256 to published layer versions with the same name

Hello,

Kudos for this excellent project. I have noticed that the example PHP Runtime that you are currently using for the example is not spec-compliant. It only implements two of the four interface methods. InitializationError and InvocationError methods are not implemented. Also, there is no Context object passed to the execution function.

Also, as per the AWS docs, the format for functions is filename.methodname which is also not followed in the used example.

I have implemented a custom PHP Runtime myself to take care of these issues and to be full spec compliant. I'd be issuing a PR for your example to use my runtime instead of the example one used.

Please share your thoughts.

Regards,
Ajitem

Dependabot couldn't find a Gopkg.toml for this project.

Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /img2lambda/Gopkg.toml.

If this isn't a Go project, or if it is a library, you may wish to disable updates for it in the .dependabot/config.yml file in this repo.

View the update logs.

Hi I'm using an assumed role profile as follows:

img2lambda -i built-image:latest --profile otheraccount

and it is uploading the layer to the account for my default profile rather than the otheraccount profile. Seems like this should be using the account id for otheraccount based on the docs so I'm thinking its a configuration issue but all other aws cli stuff works so I'm a bit stuck.

My aws configs look like this:

~/.aws/config

[default]
region = us-east-1
output = json

[profile otheraccount]
role_arn = arn:aws:iam::<other-account-id>:role/AccountAccessRole
source_profile = default

~/.aws/credentials

[default]
aws_access_key_id=<key>
aws_secret_access_key=<secret>

[otheraccount]
role_arn = arn:aws:iam::<other-account-id>:role/AccountAccessRole
source_profile = default
region = us-east-1

It would be amazing to be able to deploy the same code to AWS Lambda and AWS Batch with an initial queue size of zero for longer running jobs. It would compile to a lambda trigger that kicked off the batch job.

Happy to write a PR myself, but I need a little direction on the best workflow. Would it be as easy as calling the existing https://github.com/aws/aws-parallelcluster scripts?

Publish step always uses 'provided' runtime for the new layers. Support other runtimes -- for example, support images that are based off of lambci/lambda:nodejs8.10

under /opt we put libraries our lambda would require to run .
in the lambda envrionment , we'd had to overwrite the default LD_LIBRARY_PATH to search under the newly library path first : $LAMBDA_TASK_ROOT/lib:$LAMBDA_TASK_ROOT/lib64:$LAMBDA_RUNTIME_DIR:$LAMBDA_RUNTIME_DIR/lib:$LAMBDA_TASK_ROOT:/opt/lib:/lib64:/usr/lib64

when the lambda ran it failed with:

/var/task/main: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /opt/lib/libheif.so.1)

but both were included in our newley created layer with img2lambda:

$ unzip -l lambda-img/layer-1.zip
Archive: lambda-img/layer-1.zip
Length Date Time Name

13762736 01-25-2019 10:47 lib/libMagickCore-7.Q16HDRI.so.6
3155944 01-25-2019 10:47 lib/libMagickWand-7.Q16HDRI.so.6
10996664 01-25-2019 10:47 lib/libde265.so.0
6577608 01-25-2019 10:47 lib/libheif.so.1
178376 01-25-2019 10:47 lib64/ld-linux-x86-64.so.2
2066416 01-25-2019 10:47 lib64/libc.so.6
19208 01-25-2019 10:47 lib64/libdl.so.2
92736 01-25-2019 10:47 lib64/libgcc_s.so.1
187576 01-25-2019 10:47 lib64/libgomp.so.1
281096 01-25-2019 10:47 lib64/libjpeg.so.62
1465520 01-25-2019 10:47 lib64/libm.so.6
149272 01-25-2019 10:47 lib64/libpthread.so.0
1590808 01-25-2019 10:47 lib64/libstdc++.so.6

so is adding libraries under /opt supported ?

Dependabot couldn't find a Gopkg.toml for this project.

Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /img2lambda/Gopkg.toml.

If this isn't a Go project, or if it is a library, you may wish to disable updates for it in the .dependabot/config.yml file in this repo.

View the update logs.

Dependabot couldn't find a Gopkg.toml for this project.

Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /img2lambda/Gopkg.toml.

If this isn't a Go project, or if it is a library, you may wish to disable updates for it in the .dependabot/config.yml file in this repo.

View the update logs.

Support images directly from a Docker registry, without needing a Docker daemon locally.

The image library used supports reading directly from a Docker registry already ("docker://" transport vs "docker-daemon://" transport), but this will require handling the .tar.gz-format of Docker registry images. For example, here the blob needs to be gunzipped before untarred.

Hi all, thanks for this, very cool. Apologies if this is too newbie of a question but I'm trying to utilize the darwin binaries on my mac, preferring not to build from source. I downloaded and attempted to run via targeting from bash but darwin-amd64-img2lamba is not recognized

sensei:~ *******$ sudo /Downloads/darwin-amd64-img2lambda --version
Password:
sudo: /Users/elcapitano/Downloads/darwin-amd64-img2lambda: command not found
sensei: *******$

Should I be able to use the darwin-amd64-img2lambda on mac out of the box, anything I'm missing on how to utilize the binary?

Thanks!

Hey, I am trying to run your script on a container, but I get the permisison error

2019/11/28 12:49:46 error creating temp directory: error untarring file "/var/tmp/oci121211700": lchown /var/tmp/oci121211700/blobs: operation not permitted

Is there a simple way to solve this except for running the script as super user?

Examples: license, description, etc

Hey, I have a container where I build vips and then copy it to the /opt directory with

RUN cp -r /lib/*.so.* /opt/lib/
RUN cp -r /lib64/*.so.* /opt/lib/

Now when trying to run the script, it fails at the relevant image with the error

2019/11/28 13:25:16 could not read layer with tar nor tar.gz: walking libvips-cpp.so.42 in layer tar: lib/libvips-cpp.so.42: readlink: readlink lib/libvips-cpp.so.42: no such file or directory, opening next file in layer tar: archive/tar:
 invalid tar header                        

Do you know what the problem here is?

Docker build is useful for building function code too! Support a mode that extracts files from /var/task/src (for function deployment package) instead of /opt (used for layers).

I received an

RequestEntityTooLargeException: Request must be smaller than 69905067 bytes for the PublishLayerVersion operation
        status code: 413, request id: e1d98462-c8e8-42de-92ed-3b93928acc8e

error when trying to publish a layer which is bigger than 50mb. With Serverless, it's possible to upload layer which are larger, because it first uploads it to S3 and then uses the uploaded zip file as source for the layer.

Dependabot couldn't find a Gopkg.toml for this project.

Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /img2lambda/Gopkg.toml.

If this isn't a Go project, or if it is a library, you may wish to disable updates for it in the .dependabot/config.yml file in this repo.

View the update logs.

Windows 7 Issue

If you could provide a flag/option on the build to point the daemon at a different port, this would resolve the issue.

When DockerToolbox installs, it points the daemon to 2376 on Windows 7. Windows 10 (Docker desktop) provides an option to bind to port 2375.

Expected output

Lambda layer deployed to AWS

Actual output

2019/06/21 16:36:55 Parsing the image docker-daemon:af:latest 2019/06/21 16:36:55 Error loading image from docker engine: error during connect: Get https://%2F%2F.%2Fpipe%2Fdocker_engine/v1.22/images/get?names=af%3Alatest: dial tcp: lookup //./pipe/docker_engine: no such host

Dependabot couldn't find a Gopkg.toml for this project.

Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /img2lambda/Gopkg.toml.

If this isn't a Go project, or if it is a library, you may wish to disable updates for it in the .dependabot/config.yml file in this repo.

View the update logs.

Is it possible to have this workflow be a part of SAM / Cloudformation?

Currently I can't seem to make layers compatible with node 10.x

Thanks for this great tool. Could you please had python3.8 as a valid runtime ?