Git Product home page Git Product logo

Comments (14)

dknz2008 avatar dknz2008 commented on June 25, 2024 4

Could we also have this added to the roadmap and some indication of timeline
https://github.com/orgs/aws-observability/projects/4

from aws-otel-lambda.

dknz2008 avatar dknz2008 commented on June 25, 2024 2

+1, this would be very useful to have

from aws-otel-lambda.

willarmiros avatar willarmiros commented on June 25, 2024 1

Hi @smcoll,

I think having container images for the Lambda layers is a perfectly valid feature request, so I can mark this as such.

Have you referenced the build script for our managed layer to see how to use otel-instrument script? You can find it here: https://github.com/aws-observability/aws-otel-lambda/blob/main/python/build.sh

from aws-otel-lambda.

alolita avatar alolita commented on June 25, 2024 1

@dknz2008 we will evaluate but can't commit to a prioritization yet.

from aws-otel-lambda.

NathanielRN avatar NathanielRN commented on June 25, 2024

I feel like this similar issue I answered a few days ago is related, so just dropping the link here: aws-observability/aws-otel-community#37 (comment)

Regarding this in your code:

FIXME: /opt/otel-instrument is in /opt/python/otel-instrument?

The upstream opentelemetry-lambda repo actually moves the python/otel-instrument file to be at the otel-instrument path when it creates the zip containing the build so it _should_ be in /opt/otel-instrument`: https://github.com/open-telemetry/opentelemetry-lambda/blob/8e77f5b6a4dbc2665bc15cf829cafa700456b1e9/python/src/otel/Dockerfile#L17

We run this Docker Image in this aws-otel-lambda repo here:

aws-otel-lambda/python/build.sh

Lines 9 to 12 in 486911e

# Build sdk layer
cd ../opentelemetry-lambda/python/src || exit
./build.sh

One last complexity is that we rename it and replace it with our own "otel-instrument" because we want to add AWS X-Ray Python configuration values:

aws-otel-lambda/python/build.sh

Lines 29 to 31 in 486911e

# Use our AWS scripts instead which extend and call OTel Lambda scripts
mv otel-instrument otel-instrument-upstream-lambda
cp ../../../../python/scripts/otel-instrument .

If you just run ./python/build.sh from this aws-otel-lambda repo it would do all these steps, but if you are doing your own version of python/build.sh these are things to keep in mind to make sure your final file directory looks like the same like the zip file the original buid.sh generates 🙂

from aws-otel-lambda.

brett-bim avatar brett-bim commented on June 25, 2024

@smcoll wondering if you got this working with python lambdas deployed as containers. Currently struggling through the same issue.

from aws-otel-lambda.

smcoll avatar smcoll commented on June 25, 2024

Getting back around to this...

It does look like copying files from the published layer could do the trick (although that requires passing through AWS creds into the build), but i'm using Python 3.9 and there isn't a layer published for it yet: #222.

Next approach was to use the build scripts as-is in a container so i could use the layer artifacts with COPY FROM, but this requires docker-in-docker, or equivalent. Using Amazon Linux 2022: docker run --rm -it public.ecr.aws/amazonlinux/amazonlinux:2022 bash:

bash-5.1# cd /home
bash-5.1# yum -y install git wget tar zip unzip make docker pip
bash-5.1# wget https://go.dev/dl/go1.17.3.linux-amd64.tar.gz && rm -rf /usr/local/go && tar -C /usr/local -xzf go1.17.3.linux-amd64.tar.gz
bash-5.1# export PATH=$PATH:/usr/local/go/bin
bash-5.1# systemctl enable docker
bash-5.1# systemctl start docker
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
bash-5.1# 
bash-5.1# # at this point i realize i need docker-in-docker to do this... if that were configured, i'd do the following:
bash-5.1# git clone --recurse-submodules https://github.com/aws-observability/aws-otel-lambda.git
bash-5.1# cd aws-otel-lambda/python
bash-5.1# ./build.sh

In the end, i just want appropriate files to put in the /opt dir in my Lambda container, based on the python:3.9 image. So i investigated the build script to better understand what is happening here. Maybe i can use a few different docker stages each with a different base image and COPY FROM to the the result it need?

Is this explicitly building for 3.8, and if so, how can i adjust to build for 3.9?

Stepping through the build script, to validate my comprehension:

aws-otel-lambda/python/build.sh

Lines 3 to 7 in 33c85b3

# Build collector
pushd ../opentelemetry-lambda/collector || exit
make package
popd || exit

This section says it's building the collector, but it appears to me that it results in a no-op; i ran them inside a container (without || exit) after checking out this source code, and i see it overlays a dir from the upstream submodule and creates python/build/ and contents, but then after popd i don't see any remaining artifacts... are we looking for a collector binary here?

(Details) 
bash-4.2# git clone --recurse-submodules https://github.com/aws-observability/aws-otel-lambda.git
Cloning into 'aws-otel-lambda'...
remote: Enumerating objects: 1794, done.
remote: Counting objects: 100% (180/180), done.
remote: Compressing objects: 100% (107/107), done.
remote: Total 1794 (delta 75), reused 99 (delta 34), pack-reused 1614
Receiving objects: 100% (1794/1794), 1.75 MiB | 9.85 MiB/s, done.
Resolving deltas: 100% (908/908), done.
Submodule 'opentelemetry-lambda' (https://github.com/open-telemetry/opentelemetry-lambda.git) registered for path 'opentelemetry-lambda'
Cloning into '/var/task/aws-otel-lambda/opentelemetry-lambda'...
remote: Enumerating objects: 1922, done.        
remote: Counting objects: 100% (680/680), done.        
remote: Compressing objects: 100% (238/238), done.        
remote: Total 1922 (delta 495), reused 466 (delta 421), pack-reused 1242        
Receiving objects: 100% (1922/1922), 591.42 KiB | 5.28 MiB/s, done.
Resolving deltas: 100% (857/857), done.
Submodule path 'opentelemetry-lambda': checked out 'e13a8ce7328e6739ed510d66bc5461427fee8eb2'
bash-4.2# cd aws-otel-lambda/
bash-4.2# ls -al
total 100
drwxr-xr-x 12 root root  4096 Mar  1 21:20 .
drwxr-xr-x  1 root root  4096 Mar  1 21:20 ..
drwxr-xr-x  5 root root  4096 Mar  1 21:20 adot
-rw-r--r--  1 root root  7850 Mar  1 21:20 CHANGELOG.md
-rw-r--r--  1 root root   309 Mar  1 21:20 CODE_OF_CONDUCT.md
-rw-r--r--  1 root root  3160 Mar  1 21:20 CONTRIBUTING.md
drwxr-xr-x  4 root root  4096 Mar  1 21:20 dotnet
drwxr-xr-x  9 root root  4096 Mar  1 21:20 .git
-rw-r--r--  1 root root    85 Mar  1 21:20 .gitattributes
drwxr-xr-x  4 root root  4096 Mar  1 21:20 .github
-rw-r--r--  1 root root   127 Mar  1 21:20 .gitignore
-rw-r--r--  1 root root   130 Mar  1 21:20 .gitmodules
drwxr-xr-x  4 root root  4096 Mar  1 21:20 go
drwxr-xr-x  7 root root  4096 Mar  1 21:20 java
-rw-r--r--  1 root root 10142 Mar  1 21:20 LICENSE
drwxr-xr-x  6 root root  4096 Mar  1 21:20 nodejs
-rw-r--r--  1 root root    67 Mar  1 21:20 NOTICE
drwxr-xr-x 11 root root  4096 Mar  1 21:20 opentelemetry-lambda
-rwxr-xr-x  1 root root  1626 Mar  1 21:20 patch-upstream.sh
drwxr-xr-x  5 root root  4096 Mar  1 21:20 python
-rw-r--r--  1 root root  2467 Mar  1 21:20 README.md
drwxr-xr-x  3 root root  4096 Mar  1 21:20 sample-apps
bash-4.2# cd python/
bash-4.2# ls -al
total 24
drwxr-xr-x  5 root root 4096 Mar  1 21:20 .
drwxr-xr-x 12 root root 4096 Mar  1 21:20 ..
-rwxr-xr-x  1 root root  758 Mar  1 21:20 build.sh
drwxr-xr-x  3 root root 4096 Mar  1 21:20 integration-tests
drwxr-xr-x  3 root root 4096 Mar  1 21:20 sample-apps
drwxr-xr-x  2 root root 4096 Mar  1 21:20 scripts
bash-4.2# pushd ../opentelemetry-lambda/collector
/var/task/aws-otel-lambda/opentelemetry-lambda/collector /var/task/aws-otel-lambda/python
bash-4.2# ls -al 
total 268
drwxr-xr-x  4 root root   4096 Mar  1 21:20 .
drwxr-xr-x 11 root root   4096 Mar  1 21:20 ..
-rw-r--r--  1 root root   2736 Mar  1 21:20 collector.go
-rw-r--r--  1 root root    251 Mar  1 21:20 config.yaml
drwxr-xr-x  2 root root   4096 Mar  1 21:20 extension
-rw-r--r--  1 root root   4964 Mar  1 21:20 go.mod
-rw-r--r--  1 root root 219786 Mar  1 21:20 go.sum
drwxr-xr-x  2 root root   4096 Mar  1 21:20 lambdacomponents
-rw-r--r--  1 root root   2358 Mar  1 21:20 main.go
-rw-r--r--  1 root root   1624 Mar  1 21:20 Makefile
-rw-r--r--  1 root root   1042 Mar  1 21:20 print.go
-rw-r--r--  1 root root   3222 Mar  1 21:20 README.md
-rw-r--r--  1 root root      7 Mar  1 21:20 VERSION
bash-4.2# make package
rm -rf build/
Building otel collector extension
mkdir -p /var/task/aws-otel-lambda/opentelemetry-lambda/collector/build/extensions
GOOS=linux GO111MODULE=on CGO_ENABLED=0 installsuffix=cgo go build -trimpath -ldflags "-s -w -X main.GitHash=e13a8ce7328e6739ed510d66bc5461427fee8eb2 -X main.Version=v0.1.0 -X github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsxrayexporter.collectorDistribution=opentelemetry-collector-lambda" -o /var/task/aws-otel-lambda/opentelemetry-lambda/collector/build/extensions .
Package zip file for collector extension layer
mkdir -p /var/task/aws-otel-lambda/opentelemetry-lambda/collector/build/collector-config
cp config* /var/task/aws-otel-lambda/opentelemetry-lambda/collector/build/collector-config
cd /var/task/aws-otel-lambda/opentelemetry-lambda/collector/build && zip -r collector-extension.zip collector-config extensions
  adding: collector-config/ (stored 0%)
  adding: collector-config/config.yaml (deflated 54%)
  adding: extensions/ (stored 0%)
  adding: extensions/collector (deflated 65%)
bash-4.2# ls -al
total 272
drwxr-xr-x  5 root root   4096 Mar  1 21:20 .
drwxr-xr-x 11 root root   4096 Mar  1 21:20 ..
drwxr-xr-x  4 root root   4096 Mar  1 21:20 build
-rw-r--r--  1 root root   2736 Mar  1 21:20 collector.go
-rw-r--r--  1 root root    251 Mar  1 21:20 config.yaml
drwxr-xr-x  2 root root   4096 Mar  1 21:20 extension
-rw-r--r--  1 root root   4964 Mar  1 21:20 go.mod
-rw-r--r--  1 root root 219786 Mar  1 21:20 go.sum
drwxr-xr-x  2 root root   4096 Mar  1 21:20 lambdacomponents
-rw-r--r--  1 root root   2358 Mar  1 21:20 main.go
-rw-r--r--  1 root root   1624 Mar  1 21:20 Makefile
-rw-r--r--  1 root root   1042 Mar  1 21:20 print.go
-rw-r--r--  1 root root   3222 Mar  1 21:20 README.md
-rw-r--r--  1 root root      7 Mar  1 21:20 VERSION
bash-4.2# popd
/var/task/aws-otel-lambda/python
bash-4.2# ls -al
total 24
drwxr-xr-x  5 root root 4096 Mar  1 21:20 .
drwxr-xr-x 12 root root 4096 Mar  1 21:20 ..
-rwxr-xr-x  1 root root  758 Mar  1 21:20 build.sh
drwxr-xr-x  3 root root 4096 Mar  1 21:20 integration-tests
drwxr-xr-x  3 root root 4096 Mar  1 21:20 sample-apps
drwxr-xr-x  2 root root 4096 Mar  1 21:20 scripts
bash-4.2# cd ..
bash-4.2# ls -al
total 100
drwxr-xr-x 12 root root  4096 Mar  1 21:20 .
drwxr-xr-x  1 root root  4096 Mar  1 21:20 ..
drwxr-xr-x  5 root root  4096 Mar  1 21:20 adot
-rw-r--r--  1 root root  7850 Mar  1 21:20 CHANGELOG.md
-rw-r--r--  1 root root   309 Mar  1 21:20 CODE_OF_CONDUCT.md
-rw-r--r--  1 root root  3160 Mar  1 21:20 CONTRIBUTING.md
drwxr-xr-x  4 root root  4096 Mar  1 21:20 dotnet
drwxr-xr-x  9 root root  4096 Mar  1 21:20 .git
-rw-r--r--  1 root root    85 Mar  1 21:20 .gitattributes
drwxr-xr-x  4 root root  4096 Mar  1 21:20 .github
-rw-r--r--  1 root root   127 Mar  1 21:20 .gitignore
-rw-r--r--  1 root root   130 Mar  1 21:20 .gitmodules
drwxr-xr-x  4 root root  4096 Mar  1 21:20 go
drwxr-xr-x  7 root root  4096 Mar  1 21:20 java
-rw-r--r--  1 root root 10142 Mar  1 21:20 LICENSE
drwxr-xr-x  6 root root  4096 Mar  1 21:20 nodejs
-rw-r--r--  1 root root    67 Mar  1 21:20 NOTICE
drwxr-xr-x 11 root root  4096 Mar  1 21:20 opentelemetry-lambda
-rwxr-xr-x  1 root root  1626 Mar  1 21:20 patch-upstream.sh
drwxr-xr-x  5 root root  4096 Mar  1 21:20 python
-rw-r--r--  1 root root  2467 Mar  1 21:20 README.md
drwxr-xr-x  3 root root  4096 Mar  1 21:20 sample-apps
bash-4.2#

aws-otel-lambda/python/build.sh

Lines 9 to 12 in 33c85b3

# Build sdk layer
cd ../opentelemetry-lambda/python/src || exit
./build.sh

This looks like it builds the SDK layer, using Docker, with volume mapping to a local directory, then puts the resulting zip as build/layer.zip. Is the opentelemetry-lambda container published somewhere such that i could use ...

aws-otel-lambda/python/build.sh

Lines 14 to 16 in 33c85b3

pushd ../sample-apps || exit
./build.sh
popd || exit

What does it need from the sample apps to complete this build?

aws-otel-lambda/python/build.sh

Lines 18 to 22 in 33c85b3

# Go to build folder
cd ./build || exit
# Add AWS X-Ray dependencies
pip install opentelemetry-sdk-extension-aws -t python/

Now it installs opentelemetry-sdk-extension-aws into build/python.

aws-otel-lambda/python/build.sh

Lines 24 to 27 in 33c85b3

# Combine the layers
unzip -qo layer.zip
rm layer.zip
unzip -qo ../../../collector/build/collector-extension.zip

That seems to unzip the layers already created, and combine them.

aws-otel-lambda/python/build.sh

Lines 29 to 31 in 33c85b3

# Use our AWS scripts instead which extend and call OTel Lambda scripts
mv otel-instrument otel-instrument-upstream-lambda
cp ../../../../python/scripts/otel-instrument .

@NathanielRN explained this:

One last complexity is that we rename it and replace it with our own "otel-instrument" because we want to add AWS X-Ray Python configuration values

https://github.com/aws-observability/aws-otel-lambda/blob/main/python/build.sh#L33-L35

And this creates the final layer as an archive (which i won't need).

At what point is patch-upstream.sh invoked?

So after i get the build working (in an image), i just need to COPY FROM all the files in the opentelemetry-lambda/python/src/build dir into the /opt dir of my lambda image?

from aws-otel-lambda.

dknz2008 avatar dknz2008 commented on June 25, 2024

@alolita any update if possible?

from aws-otel-lambda.

tmokmss avatar tmokmss commented on June 25, 2024

I also want this even for Docker Node.js environment to install otel in fewer steps

from aws-otel-lambda.

jfloodnet avatar jfloodnet commented on June 25, 2024

This would great, obviously for all language/environments.

@alolita In the shorter term, it might be helpful to provide some guidance on how one could achieve this themselves.

from aws-otel-lambda.

github-actions avatar github-actions commented on June 25, 2024

This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled

from aws-otel-lambda.

github-actions avatar github-actions commented on June 25, 2024

This issue was closed because it has been marked as stale for 30 days with no activity.

from aws-otel-lambda.

smcoll avatar smcoll commented on June 25, 2024

Yes, if publishing images is not on the roadmap, at least a user guide would be great. Should i open a separate issue?

It's been a long time since i gave this a shot, but i shared my progress and requests for context at #195 (comment) - i was unable to get further at the time.

EDIT: oh, it looks like someone was able to figure it out - i can start from #209 (comment)

from aws-otel-lambda.

miguelaferreira avatar miguelaferreira commented on June 25, 2024

Having the layer packaged as a container would make building custom lambda runtimes much easier. Although there is a workaround (download and unzip the layer), that is rather cumbersome, while building a container image from scratch with the contents of the layers seems rather trivial. I would be happy to contribute a PR to build container images for the layer, if there is interest from the maintainers.

from aws-otel-lambda.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.