99designs / cmdstalk Goto Github PK

In handover we have some tasks that have various failure scenarios; sometimes it would be most appropriate to bury a task, other times it could be re-queued, maybe with a delay.

Currently cmdstalk is quite opinionated as to how to handle job failures, this one size fits all policy makes for a fine default, but it would be nice if cmdstalk supported some opt-in behaviour to allow the job command a little more control of the release strategy.

Exit codes are already meaningful, in that a non-zero code will cause cmdstalk to act based on it's opinion to either bury or release the task with a delay. With that in mind I propose allow that default behaviour to be extended, via additional args to cmdstalk, along the lines of:

cmdstalk -on-exit=255:BURY -on-exit=254:BACKOFF -on-exit=253:RELEASE ...

This would override the default cmdstalk exit code handling, which is still a valid use-case for when it is not possible to control the exit codes of the job command, with a more targeted approach that allows the job command input into what happens next.

Possible release / verbs would look something like (BURY, DELETE, RELEASE, BACKOFF) with backoff representing an (exponential) backoff strategy similar to cmdstalk's current default mode of operation.

When a job overruns it's TTR, beanstalkd will increment the job's timeout stat and put it back on the work queue for another worker to reserve.

In an effort to prevent pathological jobs from dog-piling all available workers, cmdstalk will bury a task it reserves that has timeouts greater than 1. This means that once a task is buried because of a timeout, it will always re-bury instantly each time it is kicked: the job becomes un-runnable.

Using just the buried, kicked and timeout counters, there does not appear to be a way to differentiate between "kicks due buries due to timeouts" in the way that would allow cmdstalk to bury a job the next time it is reserved after a timeout.

The beanstalkd protocol docs make mention of a one second grace period at the end of a reserve time - would it be possible to use this grace period to bury a timed out job in the "same run" as the timeout occurred?

One drawback to our current docker setup is that services like mysql and beanstalk are not isolated between 99designs services (contests / swiftly etc).

This issue doesn't manifest for mysql, as (most) applications are isolated from each other at a database level. Unfortunately it does manifest for beanstalk, as cmdstalk, when run with -all will happily listen to queues from all applications.

While this problem is better addressed at a docker level, one feature that would alleviate this in the short term is wildcard tube matching.

Something like the following could listen to all present and future tubes starting with "contests_":
cmdstalk -cmd="/path/to/your/worker --your=flags --here" -tubes="contests_*"

Happy to pair with someone on it, for the go experience.