baoshan / fairy Goto Github PK

We've been using Fairy in production for a while now, and one of the problems that we keep running into is that we end up with a bunch of FAIRY:QUEUE:* keys in our Redis database that aren't being processed by a worker. Due to the way that Fairy locks the queues, if they contain something, these queues never get processed again and are effectively blocked. I don't know why these queues are getting stuck in the first place, but it could be to do with them not being cleared on a restart, due to a crash, or possibly a Fairy bug (wherever it comes from, I would definitely consider this behaviour a bug though).

I think Fairy could be improved by a more sophisticated and dedicated locking scheme. The Redis documentation comes with a thorough description of how to use SETNX to create a locking mechanism with a timeout. Each queue would have its own lock that would be acquired by a worker when processing the queue. This would then be released when the queue was empty again. This would replace the test of whether the queue already contains entries in the _poll method. Since the lock has a timeout (which could be set by the user depending on the expected length of tasks), even if the queue did get stuck as before, the lock would eventually expire and the queue could be processed again without manual intervention.

TypeError: Cannot read property '1' of null
at /root/cme/node_modules/fairy/lib/fairy.js:461:36
at /root/cme/node_modules/fairy/node_modules/redis/index.js:1077:13
at try_callback (/root/cme/node_modules/fairy/node_modules/redis/index.js:532:9)
at RedisClient.return_reply (/root/cme/node_modules/fairy/node_modules/redis/index.js:614:13)
at ReplyParser. (/root/cme/node_modules/fairy/node_modules/redis/index.js:266:14)
at ReplyParser.EventEmitter.emit (events.js:95:17)
at ReplyParser.send_reply (/root/cme/node_modules/fairy/node_modules/redis/lib/parser/javascript.js:300:10)
at ReplyParser.execute (/root/cme/node_modules/fairy/node_modules/redis/lib/parser/javascript.js:211:22)
at RedisClient.on_data (/root/cme/node_modules/fairy/node_modules/redis/index.js:488:27)
at Socket. (/root/cme/node_modules/fairy/node_modules/redis/index.js:82:14)
at Socket.EventEmitter.emit (events.js:95:17)
at Socket. (stream_readable.js:746:14)
at Socket.EventEmitter.emit (events.js:92:17)
at emitReadable (_stream_readable.js:408:10)
at emitReadable (_stream_readable.js:404:5)
at readableAddChunk (_stream_readable.js:165:9)

I need to build a distributed worker system where a group's tasks are worked in serial fashion. This module looks to be exactly what I need; however, I am running into a problem where it looks like the task is being passed to the worker multiple times. Is this a known issue and if I were to spend the time to code an example, is this code active so that the issue can be fixed quickly?

Thanks,
-Dan

queue.enqueue(["abc", "def"])

The arguments should be flexible enough.

Node normally exits with a non-zero error code when it crashes due to an exception, but the Fairy clean up code calls process.exit(), which then exists with a zero status code. I'm not sure what the appropriate exit code should be, (Node.js seems to use 8 in my preliminary testing) but it definitely shouldn't be zero.

Have you considered using the Redis pub/sub features instead of polling every 5ms? It would help with CPU load quite a bit! The implementation I have in mind is to get rid of the following line at the end of _poll:

setTimeout @_poll, @polling_interval

The polling could then be reactivated by subscribing to a channel like FAIRY:NEW_JOBS:<queue-name> which would be published to whenever queue.enqueue() is called. This way the whole thing would sit idle until there was actually something to do. I believe that the current implementation would then clean out the whole SOURCE queue without needing to poll again, but I could be wrong about this.

Is there a reason you chose polling? Sometimes things that seem like a good idea from the outside are actually avoided for good reasons, so I apologise if that's the case. If you don't have any objections I will likely spend some time converting fairy over to pub/sub rather than polling when I get the time.

(I'm working with @henryoswald on a project that uses fairy, hence the interest from us.)