SAR: Review code and documentation and fix as appropriate for automatic routing support for S3 SQS Event Notifications input about elastic-serverless-forwarder HOT 10 CLOSED

aws.sns and aws.s3_storage_lens is incorrect. Let's fix that.

see #82

What about aws.lambda? It's in our list in documentation. I don't see any index template for it. Can you clarify?

I missed it, same as the two above, fixed in #82 as well

Based on your comments looks like we can't apply the current auto-discovery mechanism on the following logs: aws.ec2_logs, aws.route53_public_logs, aws.route53_resolver_logs, aws.s3access

exactly

from elastic-serverless-forwarder.

cc: @aspacca

from elastic-serverless-forwarder.

Here's the list of index templates that we create when users install AWS assets via integrations. So these are the ones we should support from the automatic routing perspective.

from elastic-serverless-forwarder.

@ravikesarwani I will review the list of logs integrations and see what can we do for adding support in ESF for the missing ones.

Please consider that auto-discovery is based on the expected path of AWS logs shipped to an S3 bucket by default setup.

missing integrations might be already motivated by the fact that AWS doesn't provide a default and/or unique path that's auto-discoverable when shipping to an S3 bucket

from elastic-serverless-forwarder.

These are the integrations supported in the code:

• aws.cloudtrail
• aws.cloudwatch_logs
• aws.elb_logs
• aws.firewall_logs
• aws.lambda
• aws.sns
• aws.s3_storage_lens
• aws.vpcflow
• aws.waf

aws.s3_storage_lens is clearly incorrect: while S3 storage lens data can be exported in CSV format to an S3 bucket with a unique path format that we can auto-discover, they are not logs, so it should be removed.

• aws.ec2_logs
• aws.route53_public_logs
• aws.route53_resolver_logs
• aws.s3access

for all this logs types AWS doesn't provide a unique/default path we can base auto-discovery on. In order to support them we should try to inspect the content of the event, solution that we decided to not pursue.

Please let me know if we should investigate if we will be able to support auto-discovery for these missing ingestions through event inspection.

from elastic-serverless-forwarder.

in case we will decide to adopt event inspection we could investigate as well to switch to this logic also for auto-discovery of the already supported integrations: for any integration where this could be possible we will enable auto-discovery feature for users with a not default ingestion path on the S3 bucket

I would rather reduce the scope of event inspection based auto-discovery to native AWS logs types anyway: while it would be possible to expand the support to a lot of different integrations (as long as their events have a unique and discoverable structure) the scope would increase a lot and I wonder if the ESF is the correct place where to handle this feature

from elastic-serverless-forwarder.

aws.sns is incorrect as well: SNS provides ingestion of both metrics and failure reasons logs to an S3 bucket.
the only auto-discoverable one would be the metrics one, that we have an integration for.
failure reason logs are not discoverable through path expectation and we don't have an integration for it as well.
I will remove it

from elastic-serverless-forwarder.

last note: enabling auto-discovery through event inspection we will be able to support auto-discovery on inputs/triggers other than the S3 SQS event notification

from elastic-serverless-forwarder.

This issue is to make sure our current method of auto-discovery is accurate from code and document perspective.
So in summary:

• aws.sns and aws.s3_storage_lens is incorrect. Let's fix that.
• What about aws.lambda? It's in our list in documentation. I don't see any index template for it. Can you clarify?
• Based on your comments looks like we can't apply the current auto-discovery mechanism on the following logs: aws.ec2_logs, aws.route53_public_logs, aws.route53_resolver_logs, aws.s3access

Let's cover event based auto-discovery as a separate issue. I can open one for investigation to research and document the findings so that we can move forward with that as a separate issue.

from elastic-serverless-forwarder.

#83

from elastic-serverless-forwarder.