Kafka plugin appears to be broken in Kestra 0.17.1 about kestra HOT 3 OPEN

@fhussonnois maybe you can give some advise/intell on consumer group?

from kestra.

Hi @aku, the group.id is required for Trigger and RealtimeTrigger. The group id must be unique for each trigger. Otherwise, if two triggers share the same group id, they will also share the same topic subscription. Messages relating to the consumed topic will then be split between these triggers (which is not a desirable behavior most of the time).

In addition, a Trigger or Consume task will, by default, only react to newly produced messages, if it's the first time the group is used (i.e., no existing position for the consumer). You can change this by setting the property property.auto.offset.reset=earliest to consume from the beginning of a topic.

from kestra.

Hi @aku, the group.id is required for Trigger and RealtimeTrigger. The group id must be unique for each trigger. Otherwise, if two triggers share the same group id, they will also share the same topic subscription. Messages relating to the consumed topic will then be split between these triggers (which is not a desirable behavior most of the time).

Hi Florian. You are totally right. The thing is that I used unique group and there were new messages available (group had consumer lag of several thousands messages).

In addition, a Trigger or Consume task will, by default, only react to newly produced messages

I tied to explicitly specify auto.offset.reset Also, I've tried to manually reset topic offset via kafka CLI tools.

from kestra.