Comments (9)
Hey @jkowall ,
Currently we don't have a plan to add the third party exporter to the ADOT collector for Lambda component, but we'll be working on that to enable adding other collector components from downstream. There is some workaround you can try at this moment:
- Fork the
aws-otel-collector
repo and added your exporter in defaults.go and go.mod. Replace this line in go.mod with the intended module name and release it as a module similar to lambdacomponents. - Git clone this repo (
git clone --recurse-submodules https://github.com/aws-observability/aws-otel-lambda.git
) and replace the module in patch-upstream.sh with the module you released. Run./patch-upstream.sh
- Navigate to the SDK repo you're using and run
./build.sh
- Navigate to the SDK’s sample app repo and build it using terraform. For instance, if you're using Python, then:
cd integration-tests/python/aws-sdk
terraform init
terraform apply -auto-approve
You should be able to see the Lambda layer ARN logged out in terminal and apply that to your Lambda application to enable logzio exporter.
from aws-otel-lambda.
Thanks for the response @lupengamzn
What's the purpose of getting added to the AWS collector when you are going to remove the exporters from the distribution at will. That seems like a pointless exercise we went through to be added to this distribution. I am very disappointed in this process and the way AWS manages the open-source community. It's not the way it should be working.
from aws-otel-lambda.
Sorry to cause trouble. There are 2 reasons that we cannot add all of exporters in AWS managed OpenTelemetry layer.
The main reason is a technical trouble issued in Design 2.2 and 2.3. A full Collector would exhaust 150MB+ memory in user's Lambda sandbox, but one user only need 1 or 2 exporters. so, we create a lambdacomponents module to strip down the Collector in Lambda layer. That means there is no one OpenTelemetry layer would satisfy all of users, user has to custom the layer based on his demand by replacing lambdacomponents. That is how AWS managed lambda layer come as a downstream Repo.
The second reason is AWS managed Lambda layer is not under OpenTelemetry community, any component to be added in must have passed security check, for example, there is end-2-end verification for xray exporter and daily canary test to make sure the released layer is available in every AWS region.
Ideally, as a downstream repo of opentelemetry-lambda, AWS managed layer wants to support 3rd party exporter as more as possible. Honestly we hope power contributor can help us add more 3rd party exporter. The basic principles are: 1) not inflate the size of layer too much; 2) add e2e test for that exporter in CI/CD if possible.
So, AWS managed Lambda layer is just a downstream of opentelemetry-lambda, not under opentelemetry community directly. We will add more 3rd party exporters in near future if that passes the security review. Before that an alternative solution is user builds his custom layer on demand. I also cut an issue in opentelemetry-lambda that ask for providing an easy way to build custom layer. open-telemetry/opentelemetry-lambda#100
from aws-otel-lambda.
Thanks for all of the detail here. The process to use Teraform must be done by each of our customers for each Lambda account. This is not really feasible. We're going to just use EC2 which eliminates the point of the layers. It's really unfortunate for users who aren't using x-ray. I understand the constraints, but the design should have been pluggable in some easier way versus just including the x-ray exporter. This is showing a bias towards the AWS tracing solution and not with any partner solution.
When we added the exporter to the AWS otel distro we had to jump through a lot of hoops for reviews both security, support, testing, and much more. We kind of expected that work to be included in the AWS services, and when the customer was unable to use it in a lambda layer it was pretty disappointing. I can't figure out why we bothered to do that work if we don't get the advantages within AWS.
To me this highlights yet another reason that AWS doesn't get the way open source should be working, it's about a community of technologies and not the AWS solution to the problem. I'll bring this up with our sponsors as this is by no means the first time we've seen this issue.
from aws-otel-lambda.
@jkowall Thanks for bringing this up - I agree this situation is far from ideal. Really, if the customer's function memory limit was not eaten up by the larger collector size we definitely would just go with the full aws-otel-collector distro. On the bright side, I don't see why we wouldn't do that for any future managed solution based on ADOT that may pop up.
I was wondering what would be a good solution for logzio customers? Would publishing another version of each layer with the full aws-otel-collector, with the caveat of the memory usage, be ok? Given that it's sort of a reasonable expectation for customers that aws-otel-lambda
would have the featureset of aws-otel-collector
this seems like a good thing to do either way in hindsight.
from aws-otel-lambda.
That's good to hear, but it doesn't really give us a good solution right now. The current plan is to suggest they do not use the layer and instead use an EC2 collector to forward the data. It's not ideal, but it is what it is.
I'm not a Lambda expert as yourself, so I am not sure what the impact is for the larger memory usage or how that will affect the general functioning of the Lambda functions. Is there higher hosts, slower performance?
from aws-otel-lambda.
Refer #104
from aws-otel-lambda.
That's good to hear, but it doesn't really give us a good solution right now. The current plan is to suggest they do not use the layer and instead use an EC2 collector to forward the data. It's not ideal, but it is what it is.
I'm not a Lambda expert as yourself, so I am not sure what the impact is for the larger memory usage or how that will affect the general functioning of the Lambda functions. Is there higher hosts, slower performance?
AWS Lambda charge user by Lambda sandbox capacity and execution time. If a Lambda layer eat too many resources, customer pay for that. When the memory consumption hit the red line, user see longer cold start than before, even get failed to launch unless upgrading to higher capacity.
One proposal in OTel Lambda SIG meeting is opentelemetry-lambda just contain collector core components in opentelemetry-collector. 3rd party companies not following OTLP, such as AWS, maintain their downstream repos/lambda layers respectively for their customers by customize the Collector extension.
At the same time, opentelemetry-lambda provides an easy operated tool to build custom Lambda layer based on demand.
opentelemetry-lambda (publish layer with core components such as otlp exporters)
|__aws_obersvability(publish layer with additional aws exporters)
|_ xxx (publish layer with xxx exporters)
|_??? (publish layer with logzip exporter ?)
So based on this model, any one can maintain a downstream lambda layer distro, either by user himself or by a company for its customers.
The reality of this model is that Lambda is AWS service, AWS Lambda Distro naturally would get more customer expectations, is wrongly thought AWS managed Lambda layer should support all components in opentelemetry. But that is a misunderstanding :)
from aws-otel-lambda.
This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled
from aws-otel-lambda.
Related Issues (20)
- Error exporting traces from Python lambda HOT 2
- Logging exporter does not log metrics or traces HOT 4
- node.js lambda -> aws-otel-lambda -> ADOT collector doesn't work HOT 10
- First set of counter value ignored after lambda cold boot
- Auto-instrumentation Agent only in Lambda HOT 5
- nodejs spans not exported when upgrading aws otel lambda layer to 1.15.0 HOT 7
- How to configure NodeJs auto-instrumentation to use tracecontext instead of xray? HOT 4
- Error invoking Lambda function after adding latest layer (1-28-1) for java HOT 5
- Steps to build and download a private Python lambda layer as a local .zip or .tar file? HOT 1
- Trace is broken in Lambda Java17 Runtime HOT 2
- Traces are not flushed to the server before the lambda terminates HOT 6
- Unable to access Lambda layer for Python with x86_64 architecture HOT 5
- docs: document of supported python version is old HOT 1
- 53612: bind: address already in use HOT 3
- [nodejs lambda layer] no metrics is exported to cloudwatch logs HOT 2
- Decouple processor needs to be added HOT 9
- Does not work with custom metrics HOT 2
- Request to include the awskinesisexporter HOT 2
- awsemf exporter won't create any log group, nor generating any metrics HOT 4
- Public ARN of aws-otel-nodejs-amd64 v1.18.1 is too large HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-otel-lambda.