ServiceBusExceptions thrown corresponding to missing messages about azure-event-hubs-java HOT 9 CLOSED

http://stackoverflow.com/questions/42563043/azure-event-hub-servicebusexception-causing-skipped-messages

from azure-event-hubs-java.

This looks a lot like issue 132 in the dotnet version Azure/azure-event-hubs-dotnet#132 . We are in the process of porting the fix to Java EPH.

from azure-event-hubs-java.

Unfortunately this is not the same as the dotnet issue 132. Continuing to investigate.

from azure-event-hubs-java.

Some questions:

1. The exceptions shown above were reported to the onError method of the customer’s IEventProcessor implementation, right? If not, where were they caught?
2. Are you using checkpoints? If so, when do you create checkpoints? If not, what initial offset provider do you set up in EventProcessorOptions? Exceptions like these will cause the existing IEventProcessor instance for that partition to shut down, and then it will be restarted, possibly on another EventProcessorHost instance if there are multiple. There’s a potential to skip messages during such a shutdown and restart if the starting point for the new receiver is not carefully chosen.

Thanks!

from azure-event-hubs-java.

from azure-event-hubs-java.

@JamesBirdsall answers to your questions:

1. You're correct, these exceptions were reported to the onError method of our IEventProcessor - I had missed them in the noise of the log.
2. We are using checkpoints. We call context.checkpoint() in the last line of our onEvents method. We are also, as one of the first lines of our onEvents method, calling context.setOffsetAndSequenceNumber() with details of the latest event provided in onEvents.

I see now that setOffsetAndSequenceNumber is deprecated. What is the recommended way to checkpoint to avoid the data loss you describe?

from azure-event-hubs-java.

Sreeram has identified the origin of this issue and his fix is pull request 78, linked above. My initial suspicion of a checkpointing problem was not correct.

The checkpointing recommendation is pretty straightforward: don't checkpoint until you're absolutely sure that you have either processed all events up to and including the checkpointed offset, or that you don't care if some haven't been. Depending on the nature of the processing, implementing that recommendation correctly can be simple or complicated, such as if the processing is async, or worse yet if it appears to be synchronous but is actually doing async under the covers, like a lazy or batched write/commit.

As far as checkpointing with a specific offset, the recommended way is PartitionContext.checkpoint(EventData), which checkpoints the offset and sequence number of the event given as the argument.

from azure-event-hubs-java.

@lyonsmg - this bug was fixed as part of our release http://mvnrepository.com/artifact/com.microsoft.azure/azure-eventhubs/0.12.0
We truly appreciate you for filing the issue and the patience with which you lead this bug to a resolution. Thanks a lot!
Sreeram

from azure-event-hubs-java.

Hi @SreeramGarlapati ,

I am still facing this issue in azure-eventhubs (version 0.13.0). Could you please let me know if I need to make any changes in code. I have 4 partitions and 2 of my partitions stopped suddenly, so I was not receiving events sent from device.

My code is same as the tutorial in this link

Error message I receive is:
2017-08-09 23:46:03 - IN: CH[1] : Detach{handle=0, closed=true, error=Error{condition=com.microsoft:container-close, description='The message container is being closed (20074). TrackingId:30a8f397-7843-41a5-aae3-ac2bbe25a0d6_B5, SystemTracker:NoSystemTracker, Timestamp:8/9/2017 11:46:03 PM', info=null}}
2017-08-09 23:46:03 - linkName[76e5b7_e0fb_G5_1502316395138], ErrorCondition[com.microsoft:container-close, The message container is being closed (20074). TrackingId:30a8f397-7843-41a5-aae3-ac2bbe25a0d6_B5, SystemTracker:NoSystemTracker, Timestamp:8/9/2017 11:46:03 PM]
2017-08-09 23:46:03 - linkName[76e5b7_e0fb_G5_1502316395138]

from azure-event-hubs-java.