initial offset provider ignored about azure-event-hubs-java HOT 11 CLOSED

From @JamesBirdsall on September 9, 2016 22:2

What version are you using? The ability to start at a timestamp was introduced in release 0.7.8.

Note that the initial offset provider is ignored in favor of a checkpoint for a partition. If you at some time checkpointed the partitions at the start of the stream, that would explain what you're seeing.

from azure-event-hubs-java.

From @PeterKelecom on September 10, 2016 6:22

Using version 0.8.1.

Fresh consumergroup so there shouldn't be any checkpoints..

from azure-event-hubs-java.

From @JamesBirdsall on September 12, 2016 22:10

This is a head-scratcher: your scenario, including an identical lambda, is the first test I wrote after adding this feature, and it still passes for me.
I double-checked the code and there isn't much to go wrong: the Instant that your lambda returns is passed down through the layers unmodified, until it is converted to a number of milliseconds, which is inserted in a filter expression that is passed to the service. If that part were broken I'd expect an exception back from the service.
Is there anything unusual about the clock or the time zone on your machine?
The code uses the standard java.util.logging Logger, and the logger name is available at EventProcessorHost.EVENTPROCESSORHOST_TRACE. If you turn on logging with level FINE, you should see "Calling user-provided initial offset provider" and "Initial timestamp provided: xxxxxxx". If you see "Retrieved starting offset xxxx//yyyy" then it has found a checkpoint instead.

from azure-event-hubs-java.

From @PeterKelecom on September 13, 2016 8:10

Ok, turns out it does find a checkpoint with offset -1.

Did I mess up when I created the consumer group?

from azure-event-hubs-java.

From @JamesBirdsall on September 13, 2016 17:54

No, consumer groups are a general Event Hub concept and checkpoints are particular to Event Processor Host. The only way to create a checkpoint is for user code in the event processor to call one of the PartitionContext.checkpoint() methods.
Right now there is no exposed way to delete a checkpoint, but if you create another consumer group, or use the EventProcessorHost constructor which takes a storage container name (to avoid using the existing storage container where the undesired checkpoints are), then you should avoid this problem.

from azure-event-hubs-java.

From @PeterKelecom on September 13, 2016 18:15

this happens when I create a new consumergroup, without checkpointing, for every consumergroup I create.

from azure-event-hubs-java.

From @JamesBirdsall on September 14, 2016 1:0

That was a valuable clue, thanks. It led me to discover that the default storage container name is being created from the event hub name only, which explains why those checkpoints are following you across consumer groups. I have opened issue 261 to fix the default container name to include the consumer group name.
Can you try using an EventProcessorHost constructor that lets you supply a storage container name? Ideally the container would not exist already, meaning no blobs in it, meaning no stray checkpoints, and then you should get the behavior you're expecting.

from azure-event-hubs-java.

From @PeterKelecom on September 14, 2016 8:26

that did it, thanks!

from azure-event-hubs-java.

From @PeterKelecom on September 14, 2016 9:8

Upon further testing, this only works the first time, when the container doesn't exist. After that, the container is created, contains a folder for the consumergroup and in that folder has checkpoint data, even though the processor never checkpoints. On restart of the processor, it starts processing messages from the start of the queue.

from azure-event-hubs-java.

From @JamesBirdsall on September 15, 2016 0:51

Never mind issue 261, that turned out to be me being confused. When I changed my test case to do the stop and restart as you describe, I was able to reproduce your issue, and I found the code which was creating the unwanted checkpoints. I have created issue 262 and am testing a fix now.

from azure-event-hubs-java.

The issue was fixed in version 0.8.2.

from azure-event-hubs-java.