Locking queues about fairy HOT 7 OPEN

Hey, we are probably going to start working on this soon as its becoming a problem for us. Would you be ok with the different locking mechanism?

from fairy.

Sorry! I missed the issue for 3 months! Does your project still need a workaround? Please let me know.

from fairy.

The locking mechanism is promising I think. But the blocking behavior is by design and there's a reschedule method to restart the blocked tasks. Does that decision suite your needs?

from fairy.

We found that we had problems with the queues getting stuck, as I mentioned. We haven't been able to track down how these got into an inconsistent state, where the FAIRY:QUEUE:* lists has elements in them but no fairy worker was processing them. Using a locking mechanism would be a partial fix for this since it would be more robust to stuck queues.

Yes, we managed to get around it by calling reschedule regularly when we need to, but it's still a workaround rather than a solution to the underlying problem.

We're also seeing trouble with the workers getting stuck regularly for some reason. We still haven't figured out why, but after a while, and at random intervals, the 5 workers that we have running have all stopped processing anything. A restart of our worker process sets everything going again, but it's not clear what the problem is here. We will keep you updated as we find out more.

from fairy.

Are there any FAIRY:PROCESSING:* keys when workers are stuck? That may because workers haven't callback for some reason. Could you please post a keys * result when the problem happens?

pub/sub rather than polling is definitely the right way to improve. That'll be done in the next version. But I'm afraid a timeout locking is not always better. Sometimes tasks show vast differences in demanding of processing time.

Thanks.

from fairy.

I don't think so - I think the only keys set were the FAIRY:QUEUE:* keys. It's been a while since we investigated this though since we've been running with the above workaround for a while now. I'll try to get you more debugging data next time we look into this.

Thanks a lot.

from fairy.

@baoshan @jpallen have you ever implemented the lock?

https://github.com/mike-marcacci/node-redlock This might help?

from fairy.