racksec / banach Goto Github PK

https://github.com/sunng87/diehard/

Our internal project, shrieker, currently just subverts these visibility modifiers.

retry currently calls (f) unconditionally, and strategies must synchronously return either a wait time or an exception. This is not good enough.

Here's an example case: f is a fn that gets some data from an API. That API uses a short-term auth token that you acquire with a long-term API key. Sometimes, the auth token expires and you must re-authenticate. Retry logic should be able to do this for you transparently; but that means potentially going off and doing something else (acquire a new API key), an operation which requires a deferred.

Because retry just calls (f), there's no way to communicate any new information to f other than some shared mutable state. Shared mutable state can be OK (e.g. caching the auth token), but it'd be even nicer if we can just modulate how f gets called.

Suggestion:

• strategies are fns that take a map with :failures and :args, returning a maybe-deferred map of the same shape; md/recur recurs with this. Keep in mind that md/recur doesn't understand how to deal with deferreds, only md/loop's body -- so you probably have to use let-flow here.
• signature for retry becomes [f strat] => [strat f & args], f is called as (apply f args)

Alternatively, we can stick to the current signature, but still have strategies return deferreds. This would mean exponentially returns a deferred that waits, instead of a number.

... really what I want is Effect, where strategies return effects and f is itself an effect, but such is life.

We've written this is cloudpassage-lib and then repeated it in shrieker -- we should just write it once in banach.

Right now, we have deterministic exponential backoff. This means users of banach potentially are exposed to stampeding herd problems.

We could do this in exponentially, or make it composable.

A composable API would look like [jitter-params strategy] -> strategy. If we use with-clock with some state, we can figure out what delays another strategy imposed and instead apply jitter and then run them with a real clock (unless if course it's composable turtles all the way down, life is a simulation, and all attempts at computer are intrinsically futile). This is not super great, but we didn't make up that global mutable state that is the reactoclock.

The benefit of a composable API is that it's reusable. The default, especially for helper fns, should absolutely include jitter.

Use cases include:

• When I see a failure that the server is hosed, I want to give up now so I can tell them to restart their broken microservice ASAP
• When I see an authentication error I want to reauth instead and forget that error; it shouldn't be considered a failure of the thing I was really trying to do.

This doesn't require changing the retry logic, but just adding a strategy that knows how to dispatch to other strategies. That means strategies can do this right now, it's just more work.

There should probably also be a dispatcher that knows how to dispatch on the last failure since that's the most common case.

exponentially is currently initial^attempts; it should probably be initial * base ^ attempts with base=2. Also, we should consider having jitter as an option to prevent stampeding herds. jitter can be linear on the result or on the attempts I guess (dunno how bad floats are for expt)

A lot of strategies take a deferred, only to let-flow/chain it and then in many cases either return a synchronous value or synchronously throw an exception.