Comments (7)
We started tagging lambda versions from main
(https://github.com/elastic/elastic-serverless-forwarder/tags)
It is a temporary solution to pin to a specific version of the lambda flow picture in the SAR readme (https://github.com/elastic/elastic-serverless-forwarder/blob/main/docs/lambda-flow.png), please, consider it a strategy that might change once we go GA.
from elastic-serverless-forwarder.
My company is in the process of provisioning ESF via Terraform this weekend. We've found a few friction points that may be worth considering:
- ESF requires a bucket to read from for the
sar-application.yml
configuration file. Terraform deployment would be much simpler if these configurations could be specified as part of the Lambda environment and via AWS secrets manager. - Deploying ESF as a SAR via Terraform is not straightforward, nor is it clear how to pin our ESF deployment to a specific revision. We ended up building this repo and bundling a zip of EST at 7fff40d to avoid changes to master or the SAR application from breaking any live ESF instances we have within our AWS accounts.
Happy to chat more about this. My company is really interested in making deployment of ESF via Terraform an easy process. You can reach me at damien.wilson at humaninterest dot com.
from elastic-serverless-forwarder.
hello @damien , glad to hear you are using ESF.
- ESF requires a bucket to read from for the
sar-application.yml
configuration file. Terraform deployment would be much simpler if these configurations could be specified as part of the Lambda environment and via AWS secrets manager.
how does exactly your terraform definition look like?
is it a single step deployment or one the goes through an initial deployment and importing the state of the deployed lambda?
so far I've only experimented a little with the terraform support for ESF, and from what I can see the aws_serverlessapplicationrepository_cloudformation_stack
(https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/serverlessapplicationrepository_cloudformation_stack) resource doesn't allow the same level of customisation/automation as the lambda_function
one (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function)
I have to figure out how to solve this problem but I would expect a similar solution to what's currently documented for deployment through cloudformation:
- apply the SAR stack definition
- import the lambda state
- customise definition from the state
- apply the customised definition
this will probably requires some scripting, without being able to rely only on terraform apply
- Deploying ESF as a SAR via Terraform is not straightforward, nor is it clear how to pin our ESF deployment to a specific revision. We ended up building this repo and bundling a zip of EST at 7fff40d to avoid changes to master or the SAR application from breaking any live ESF instances we have within our AWS accounts.
for concerns pinning the terraform definition to a specific version you can rely on semantic_version
argument in the definition (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/serverlessapplicationrepository_cloudformation_stack#semantic_version)
currently releases are not tagged in the repo, and we still have to decide a strategy regarding this, since when we will extended support to serverless from different cloud providers, a single version fitting all of them will not be most likely be available.
from elastic-serverless-forwarder.
Hey @aspacca, thanks for the response!
how does exactly your terraform definition look like?
is it a single step deployment or one the goes through an initial deployment and importing the state of the deployed lambda?so far I've only experimented a little with the terraform support for ESF, and from what I can see the aws_serverlessapplicationrepository_cloudformation_stack (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/serverlessapplicationrepository_cloudformation_stack) resource doesn't allow the same level of customisation/automation as the lambda_function one (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function)
We created our own Terraform modules loosely based off of the Cloudformation template you mentioned in the docs, though we opted to try to deploy the lambda directly rather than provision it via SAR. It was a long busy weekend, so I'll need to get back to you on the specific reasons why we did that after consulting some notes, but we ran into some kind of issue wiring the SAR app with other infrastructure components when done through Terraform code that deploying the forwarder via a lambda_function
resolved for us.
Sadly, to deploy as a lambda_function
resource we needed to specify a function payload. I (debatably) got around by using .internal/aws/scripts/dist.sh#L31-L50 from commit 58d81b1 (master at the time).
The problem with this approach today is that I have no idea if this is the correct revision to build from to end up with a payload identical to the one used by the public SAR application.
is it a single step deployment or one the goes through an initial deployment and importing the state of the deployed lambda?
The benefit to this approach is that it is single step deploy and doesn't depend on multiple terraform apply
steps; we just need to know the appropriate lambda payload ahead of time. :)
currently releases are not tagged in the repo, and we still have to decide a strategy regarding this, since when we will extended support to serverless from different cloud providers, a single version fitting all of them will not be most likely be available.
Adding a commit SHA to the metadata of the SAR distribution or some kind of "published-$TIMESTAMP" style tag on deployed commits would help me out here if decisions around semver and release management are going to take more time to figure out.
Thanks for your time @aspacca! Let me know if there's any more info I can share that would be interesting to you. Based on your commentary I think we're addressing this problem in similar ways. Looking forward to hearing more of your thoughts!
from elastic-serverless-forwarder.
@aspacca Thank you for the response. The tagging has been useful to derive the appropriate version of ESF to deploy with our own Terraform code, though there are still some gaps here I suspect you and the fine folks working on this project address as you near GA (outlined in my previous responses).
Ideally a GA release will be deployable via Terraform in a single run without needing to resort to state manipulation. Doing this is possible, it just requires a bit more formalization of how peripheral dependencies and configurations of ESF are deployed (AWS IAM roles, definition and association of relevant SQS queues and events, etc).
Happy to help if you have new releases or changes you'd like feedback on.
Cheers!
from elastic-serverless-forwarder.
hello @damien the problem in defining AWS IAM roles in the template is that users must opt-in in SAR to access applications that define such roles.
I'm currently looking to have same Params for the ARNs of event sources and subscriptions, according to the type (sqs, kinesis, cloudwatch logs) and with a macro expand them directly in the template: I this case I would expect default AWS IAM roles for the Lambda will be added and therefore no explicit opt-in in required to access the application. I still have to test this is the case.
Having that it should be possible to use https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/serverlessapplicationrepository_cloudformation_stack to deploy the application with terraform
from elastic-serverless-forwarder.
hello @damien
I'm closing this issue as terraform deployment is now supported: please, see the instructions at https://github.com/elastic/elastic-serverless-forwarder/blob/lambda-v0.30.0/docs/README-AWS.md#terraform
from elastic-serverless-forwarder.
Related Issues (20)
- Provide support for Python 3.12
- [DOCS] Fix outdated link
- CloudWatch log retention for ESF own logs
- Investigate ingestion of files with same IDs HOT 2
- Create pipeline to push zip file with dependencies to an S3 bucket HOT 3
- Misconfigured ESF input results in a warning message, not events sent to replay queue HOT 3
- CMA-CGM ESF Issue HOT 1
- no input defined on cloudwatch-logs trigger HOT 1
- Enable shared outputs and multiple outputs per input
- Investigate number of API calls of ESF due to ec2:describe_regions usage HOT 3
- Issue with region name check in publishing script
- Support SSM (Systems Manager) parameters
- policy size limits - max 44 log groups you can add
- Deployment method using AWS Console doesn't work HOT 5
- ESF incorrectly sets region for VPC flow logs from multi-region S3 bucket
- es_dead_letter_index
- Sercrets manager secrets caching seems not working
- What do I do when I hit the size limits?
- error: Argument 1 to "wrapper" has incompatible type "ProtocolStorageType@__init__"; expected "ProtocolStorageType@json_collector" [arg-type] HOT 1
- dead letter index: store errors using the ECS field format HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elastic-serverless-forwarder.