SAR: Gracefull handling of error scenarios about elastic-serverless-forwarder HOT 1 CLOSED

We identified two kinds of failure that can happen during the execution of the Lambda:

1. Errors before the ingestion phase started
2. Errors during the ingestion phase

For errors at (1), we can safely make the Lambda return a failure: these errors are mostly due to misconfiguration, improper permission on the AWS resources, etc. Most importantly when an error occurs at this stage we don’t have any state yet about the events that are ingested, so there’s no consistency to keep and we can safely let the Lambda return a failure, with the underlying trigger ending up in a built-in retry mechanism.

For errors at (2) the situation is more tricky: we have now a state for N failed events out of total X events, if we fail the whole Lambda all the X events will be processed again. While the N failed ones could now succeed, the remaining X-N will now fail, since the datastreams are append-only and we would try to recreate already ingested documents (the ID of the document is deterministic).

The implemented solution to overcome this problem is to add an internal “replay queue”, similar to the internal continuing queue, where to store a single failed event per message that the users can optionally use to trigger the Lambda itself, or consume the messages in the way they prefer.
In the case that the replay queue is used to trigger the Lambda itself, a proper handler must be implemented.

In both cases, supporting a replay queue has the drawback of adding code to inspect every single failure in the Shippers (ie: output components in the ESF context) and handling the messaging to the replay queue.

While at the moment we support only Elasticsearch as output/shipper, the Lambda function is designed to easily support multiple outputs.

If this will be the case in the future, complexity of handling the messaging to the replay queue will grow linearly, since a single event could have different failure/success outcome for different outputs, and all of them have to be handled and sent to the replay queue separately.

Overall the concrete implementation for the Lambda in #60 minimised the complexity with enough abstraction to have a single replay "dispatcher" that's generic enough for every shipper, and a single replay handler that will contain the concrete logic for every single shipper

from elastic-serverless-forwarder.