Batches wrapping unique_for jobs completing before jobs have run about sidekiq HOT 5 CLOSED

I've been doing more local testing and seeing what happens if we just have perform_async rather than perform_in so that the run order is more roughly the same as the enqueue order, and I have not been able to reproduce a situation where the callback is called before any instance of each of the nested jobs has run. So I suspect that the trick is to not combine unique_for and batches and perform_in and things will be more safe.

from sidekiq.

All of this is As Designed. Uniqueness and batches often don't play well together.

from sidekiq.

Mike - this is kind of the conclusion that I came to as well. How easy would it be within the current architecture to make it so that if you add a job to a batch which is declared as unique that it would raise an exception or log a warning? I'm happy to do my best to avoid doing this, but if someone changed the behaviour of a job without realising that other people were using it in a batch then this might cause hard-to-spot errors - it would be nice if we got a nice clear warning that this was happening?

from sidekiq.

Uniqueness is a sharp knife, that’s why the batch wiki page warns the reader about it, but logging a warning would be useful the first time but annoying every other time.

from sidekiq.

I am toying with recording an expiring key in REDIS each time a job completes that is unique by it's input params. Then when a new job starts up it can check to see whether the same job has completed which started after the current job was enqueued and if so then just returning as a noop. That way we can intermix batches and this-form-of-uniqueness without a problem.

from sidekiq.