kipcole9 / attempt Goto Github PK

Scenario

When executing a function against an external service (HTTP service, database service - any service which may suffer failure) we have an objective to retry the function when the failure mode is retryable.

However we want to also protect the external service from excessive retries when the service itself is under load or suffering higher than normal failure rates. We also don't want retries to swamp the overall number of service requests.

Problem statement

We have two objectives:

1. To maximise request throughput with limited capacity
2. We do not want retries to swamp the capacity

How can we address the problem

Finagle's strategy is:

By default, the RetryBudget allows for about 20% of the total requests to be immediately (no backoff) retried on top of 10 retries per second in order to accommodate clients that have just started issuing requests or clients that have a low rate of requests per second.

Which says:

1. the initial request is made
2. in case of failure, up to n% of requests may be retried immediately
3. those requests not immediately retried are retried in association with a token bucket (ie retries are rate limited)

Currently in Attempt

1. All initial requests are rate limited through a token bucket.
2. All retries are rate limited through the leaky bucket
3. All requests - initial or retry - are queued with the same priority and hence its possible that as failure rates increase that retries swamp new requests

Some thoughts on the way forward

1. Rate limited to an external service still makes sense as an option but this is a separate (although related) issue to managing retries.

2. Problem statement is reasonable then we can approach a solution as

• Estimate the maximum throughput in requests_per_second using the a sliding window of performance based upon successful requests.

• Define the maximum percentage of total throughput that may be permitted for retries

The intent of this strategy is to:

• Adapt to the available throughput on the target system - automatically ramp up and down as performance and load varies

• Prevent retries from swamping the target system, especially when the target system is under load

Other relevant items not currently covered in Attempt, but maybe should be

1. Metrics. Since in many cases Attempt will be used in front of an external service its a reasonable place to capture performance and reliability metrics.

2. Required behaviour when a service is down, or flapping, may well be different. More like a fuse or circuit breaker?