Secor does not handle topic deletion gracefully about secor HOT 4 CLOSED

Would restart secor help to remove the deleted topic?

If not, would specify some regex expression on topicFilter work to exclude that topic?

Kafka also have a concept of Blacklist (we only used its Whitelist), you can extend secor to use the Kafka's Blacklist to exclude topics (and send us the PR).

from secor.

Hi henry,
Just tried restarting secor; doesn't seem to have an effect.

Filtering out the topic probably works, but it seems like a bandaid fix, and requires explicit input. I wonder how much work it would be to have secor fix its consumer group offsets whenever it encounters this sort of situation.

Perhaps TopicFilter (or something else) can be used to filter out topics that no longer exist?

from secor.

This seems to me more like a kafka bug (the high level consumers is still
consuming from a deleted topic), if you put more work in secor to handle
deleted topic, this sounds kludge to me.

Kafka topic deletion is usually not a common event in a production system,
that's why I suggest using topicFilter (or add support for Blacklist) for
those deleted topics, this only needs to happen once in a long while.

On Thu, Aug 27, 2015 at 4:14 PM, fabeschan [email protected] wrote:

Hi henry,
Just tried restarting secor; doesn't seem to have an effect.

Filtering out the topic probably works, but it seems like a bandaid fix,
and requires explicit input. I wonder how much work it would be to have
secor fix its consumer group offsets whenever it encounters this sort of
situation.

—
Reply to this email directly or view it on GitHub
#126 (comment).

from secor.

fabeschan, I am closing this one. Let me know whether you want to play with topic filters.

from secor.