Application server concurrency about go-workers HOT 4 CLOSED

Hey all,

Sorry for the delay. To give you some background, this project started as a port of Sidekiq to golang. The internals work almost identically. The fetcher in go-workers uses the "reliable_fetch" style as described in Sidekiq's documentation: https://github.com/mperham/sidekiq/wiki/Reliability.

To summarize:

When an instance fetches a message from redis, it uses redis's rpoplpush command to atomically remove the message from the queue and place it in an inprogress queue. This helps to make sure the message isn't lost if say, your go-workers process dies, etc.

The inprogress queue key is generated based on the unique process ID you pass into workers.Configure (shown in the README).

That operation is atomic (assuming the Redis instance doesn't die before persisting the change to disk), and any other process fetching messages will not retrieve the same message.

Once the job finishes, go-workers removes the message from the inprogress queue and fetches another message.

When an instance starts, it first looks at the inprogress queue to see if there are any messages which weren't completed (due to a crash, etc). If there are messages in the inprogress queue, it'll being processing them.

IF you are running multiple instances and passing the same process ID into workers.Configure, then each would view messages that are inprogress from another instance as messages it should process again.

If you aren't, then there are no known issues regarding duplicate job processing (in fact, most of the protections for reliable, unique enqueuing/dequeuing is left up to the redis instance itself!).

If you're still seeing weird behavior and can narrow it down to a test case I can reproduce, then I'd be happy to take a look and help debug!

Thanks,
John

from go-workers.

I suspect the issue arises from here:
https://github.com/jrallison/go-workers/blob/master/worker.go#L32-L34

if w.process(message) {
    w.manager.confirm <- message
}

If this is a blocking message then the job would remain in the queue until the work is done. Am I reading that correctly?

from go-workers.

IF you are running multiple instances and passing the same process ID into workers.Configure, then each would view messages that are inprogress from another instance as messages it should process again.

thanks @jrallison, I'll bet that's exactly what it is. I had the process id hard coded. Will try it out with dynamic ID based on worker number.

from go-workers.

as @maccman just yelled from across the room "the problem was between keyboard and chair"

from go-workers.