Message receiving stops if done() is not invoked about sqs-consumer HOT 9 CLOSED

I'm at least on the side of the inquiry here that says the "desired behavior" should be documented clearly. It felt like a bug when I didn't understand why it would stop suddenly. @robinjmurphy the explanation makes sense, as you have described. I tried out the clever patch by @kavuri and it brings up a host of other questions and potential issues which you probably don't want to support. That said, it doesn't seem unreasonable to disable this behavior, perhaps as an unsupported mode. But it would be great to add your explanation to the README. Thank you.

from sqs-consumer.

I think this is the behaviour I'd expect. When you don't call done in your handleMessage function, there are no more callbacks left to be called and the Node process will exit.

This library is designed specifically for task queues, where messages are processed and then deleted. If you need to inspect a queue or interact with it in another way, you might want to use the AWS library directly.

from sqs-consumer.

done() is overloaded with two pieces of functionality:

1. It deletes the currently processed message
2. It re-polls the queue with long poll

Is there a way to dis-associate this? The reason being that during the processing of a message (which happens in an async loop), I would like to leave the message in queue, but meanwhile, if there are new messages in queue, I would like to process them. This does not seem to be possible. Is there a way to not overload this functionality?

from sqs-consumer.

That's the intended behaviour. We want to couple those two actions together so that polling doesn't restart until the current message/batch of messages has been processed.

You can ask for more than one message at once using the batchSize option but polling won't restart until you've finished processing all of the messages in the current batch.

This is to stop each Node process from taking on too many messages at once. It also provides a predictable workload for each consumer. We fix the batch size and scale horizontally to cope with the number of messages on a queue.

from sqs-consumer.

The processing of a message in my usecase can go anywhere between few secs to minutes. I have modified the code to invoke consumer._poll() in the handleMessage() and deleteMessage(), like

  async.series([
    function handleMessage(done) {
      consumer.handleMessage(message, done);
      consumer._poll(); //SK
    },
    function deleteMessage(done) {
      consumer._deleteMessage(message, done);
      consumer._poll(); //SK
    }
  ], function (err) {

Can you suggest if this is an okay change to make?

from sqs-consumer.

It's not a change I'd like to introduce. We actually used to do this (re-polling immediately regardless of whether a message has been processed) but we changed it in favour of the current behaviour to make the message handling more controlled.

In particular, if the service is struggling to process messages, it shouldn't continue to consume them from the queue at the same rate. The current implementation provides some back pressure on the queue when processing slows down for some reason.

If you want a different polling behaviour, I'd suggest using the AWS client directly.

I'm going to close this issue off as the current re-polling behaviour is a feature of our library.

from sqs-consumer.

And a follow on to my comment, I was puzzled that the code would exit in this condition without firing an event. Unless I missed the event, it appeared to just exit.

from sqs-consumer.

@kavuri maybe let's fork the changes and some configuration to try out how this might look with an option to not have this as the default behavior?

from sqs-consumer.

@truedat101 I have moved on from this and am not pursuing this anymore. Having a configuration to take this patch might be involved but adding this patch to try if it works or not should not be difficult for you

from sqs-consumer.