Comments (10)
aws.sns
andaws.s3_storage_lens
is incorrect. Let's fix that.
see #82
What about
aws.lambda
? It's in our list in documentation. I don't see any index template for it. Can you clarify?
I missed it, same as the two above, fixed in #82 as well
Based on your comments looks like we can't apply the current auto-discovery mechanism on the following logs: aws.ec2_logs, aws.route53_public_logs, aws.route53_resolver_logs, aws.s3access
exactly
from elastic-serverless-forwarder.
cc: @aspacca
from elastic-serverless-forwarder.
Here's the list of index templates that we create when users install AWS assets via integrations. So these are the ones we should support from the automatic routing perspective.
from elastic-serverless-forwarder.
@ravikesarwani I will review the list of logs integrations and see what can we do for adding support in ESF for the missing ones.
Please consider that auto-discovery is based on the expected path of AWS logs shipped to an S3 bucket by default setup.
missing integrations might be already motivated by the fact that AWS doesn't provide a default and/or unique path that's auto-discoverable when shipping to an S3 bucket
from elastic-serverless-forwarder.
These are the integrations supported in the code:
- aws.cloudtrail
- aws.cloudwatch_logs
- aws.elb_logs
- aws.firewall_logs
- aws.lambda
- aws.sns
- aws.s3_storage_lens
- aws.vpcflow
- aws.waf
aws.s3_storage_lens
is clearly incorrect: while S3 storage lens data can be exported in CSV format to an S3 bucket with a unique path format that we can auto-discover, they are not logs, so it should be removed.
- aws.ec2_logs
- aws.route53_public_logs
- aws.route53_resolver_logs
- aws.s3access
for all this logs types AWS doesn't provide a unique/default path we can base auto-discovery on. In order to support them we should try to inspect the content of the event, solution that we decided to not pursue.
Please let me know if we should investigate if we will be able to support auto-discovery for these missing ingestions through event inspection.
from elastic-serverless-forwarder.
in case we will decide to adopt event inspection we could investigate as well to switch to this logic also for auto-discovery of the already supported integrations: for any integration where this could be possible we will enable auto-discovery feature for users with a not default ingestion path on the S3 bucket
I would rather reduce the scope of event inspection based auto-discovery to native AWS logs types anyway: while it would be possible to expand the support to a lot of different integrations (as long as their events have a unique and discoverable structure) the scope would increase a lot and I wonder if the ESF is the correct place where to handle this feature
from elastic-serverless-forwarder.
aws.sns
is incorrect as well: SNS provides ingestion of both metrics and failure reasons logs to an S3 bucket.
the only auto-discoverable one would be the metrics one, that we have an integration for.
failure reason logs are not discoverable through path expectation and we don't have an integration for it as well.
I will remove it
from elastic-serverless-forwarder.
last note: enabling auto-discovery through event inspection we will be able to support auto-discovery on inputs/triggers other than the S3 SQS event notification
from elastic-serverless-forwarder.
This issue is to make sure our current method of auto-discovery is accurate from code and document perspective.
So in summary:
aws.sns
andaws.s3_storage_lens
is incorrect. Let's fix that.- What about
aws.lambda
? It's in our list in documentation. I don't see any index template for it. Can you clarify? - Based on your comments looks like we can't apply the current auto-discovery mechanism on the following logs: aws.ec2_logs, aws.route53_public_logs, aws.route53_resolver_logs, aws.s3access
Let's cover event based auto-discovery as a separate issue. I can open one for investigation to research and document the findings so that we can move forward with that as a separate issue.
from elastic-serverless-forwarder.
from elastic-serverless-forwarder.
Related Issues (20)
- Incorrect Handling of Permissions Can Result in Data Loss
- Provide support for Python 3.12
- [DOCS] Fix outdated link
- CloudWatch log retention for ESF own logs
- Investigate ingestion of files with same IDs HOT 2
- Create pipeline to push zip file with dependencies to an S3 bucket HOT 3
- Misconfigured ESF input results in a warning message, not events sent to replay queue HOT 3
- CMA-CGM ESF Issue HOT 1
- no input defined on cloudwatch-logs trigger HOT 1
- Enable shared outputs and multiple outputs per input
- Investigate number of API calls of ESF due to ec2:describe_regions usage HOT 3
- Issue with region name check in publishing script
- Support SSM (Systems Manager) parameters
- policy size limits - max 44 log groups you can add
- Deployment method using AWS Console doesn't work HOT 5
- ESF incorrectly sets region for VPC flow logs from multi-region S3 bucket
- es_dead_letter_index
- Sercrets manager secrets caching seems not working
- What do I do when I hit the size limits?
- error: Argument 1 to "wrapper" has incompatible type "ProtocolStorageType@__init__"; expected "ProtocolStorageType@json_collector" [arg-type] HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elastic-serverless-forwarder.