Comments (1)
We identified two kinds of failure that can happen during the execution of the Lambda:
- Errors before the ingestion phase started
- Errors during the ingestion phase
For errors at (1), we can safely make the Lambda return a failure: these errors are mostly due to misconfiguration, improper permission on the AWS resources, etc. Most importantly when an error occurs at this stage we don’t have any state yet about the events that are ingested, so there’s no consistency to keep and we can safely let the Lambda return a failure, with the underlying trigger ending up in a built-in retry mechanism.
For errors at (2) the situation is more tricky: we have now a state for N failed events out of total X events, if we fail the whole Lambda all the X events will be processed again. While the N failed ones could now succeed, the remaining X-N will now fail, since the datastreams are append-only and we would try to recreate already ingested documents (the ID of the document is deterministic).
The implemented solution to overcome this problem is to add an internal “replay queue”, similar to the internal continuing queue, where to store a single failed event per message that the users can optionally use to trigger the Lambda itself, or consume the messages in the way they prefer.
In the case that the replay queue is used to trigger the Lambda itself, a proper handler must be implemented.
In both cases, supporting a replay queue has the drawback of adding code to inspect every single failure in the Shippers (ie: output components in the ESF context) and handling the messaging to the replay queue.
While at the moment we support only Elasticsearch as output/shipper, the Lambda function is designed to easily support multiple outputs.
If this will be the case in the future, complexity of handling the messaging to the replay queue will grow linearly, since a single event could have different failure/success outcome for different outputs, and all of them have to be handled and sent to the replay queue separately.
Overall the concrete implementation for the Lambda in #60 minimised the complexity with enough abstraction to have a single replay "dispatcher" that's generic enough for every shipper, and a single replay handler that will contain the concrete logic for every single shipper
from elastic-serverless-forwarder.
Related Issues (20)
- Incorrect Handling of Permissions Can Result in Data Loss
- Provide support for Python 3.12
- [DOCS] Fix outdated link
- CloudWatch log retention for ESF own logs
- Investigate ingestion of files with same IDs HOT 2
- Create pipeline to push zip file with dependencies to an S3 bucket HOT 3
- Misconfigured ESF input results in a warning message, not events sent to replay queue HOT 3
- CMA-CGM ESF Issue HOT 1
- no input defined on cloudwatch-logs trigger HOT 1
- Enable shared outputs and multiple outputs per input
- Investigate number of API calls of ESF due to ec2:describe_regions usage HOT 3
- Issue with region name check in publishing script
- Support SSM (Systems Manager) parameters
- policy size limits - max 44 log groups you can add
- Deployment method using AWS Console doesn't work HOT 5
- ESF incorrectly sets region for VPC flow logs from multi-region S3 bucket
- es_dead_letter_index
- Sercrets manager secrets caching seems not working
- What do I do when I hit the size limits?
- error: Argument 1 to "wrapper" has incompatible type "ProtocolStorageType@__init__"; expected "ProtocolStorageType@json_collector" [arg-type] HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elastic-serverless-forwarder.