Possible Memory Leak with Kafka source about vector HOT 9 CLOSED

The proposed fix is now merged into the master branch. Please confirm if this resolves the memory leak for you.

Yes, It is effective

from vector.

Hi @biuboombiuboom ,

There's not a lot of detail here. What leads you to believe there is a memory leak? If you are feeling motivated you could try running Vector under valgrind and providing the profile here.

from vector.

We're actually seeing a similar issue running the latest version. With a kafka source and a clickhouse sink. A sustained flow of messages seems to inevitably result in OOM. You can see the behaviour in these charts. We have very bursty workloads. Each workload burst results in sustained CPU use (makes sense) as well as continued memory growth. If the work stops - the memory goes back down. But if the work continues... it will OOM.

from vector.

ack_stream has too many FinalizerEntry not consumed.

Try sync store offsets,and disable BatchNotifier.

Or move ack_stream.next() before messages.next() & with biased.

from vector.

I'm not clear what you mean by the following:

Try sync store offsets,and disable BatchNotifier.

The second suggestion is spot on and definitely worth a try. The ack_stream should always be checked before consuming new events, because eventually it will run dry to allow progress for new data.

from vector.

I'm not clear what you mean by the following:

Try sync store offsets,and disable BatchNotifier.

The second suggestion is spot on and definitely worth a try. The ack_stream should always be checked before consuming new events, because eventually it will run dry to allow progress for new data.

Sorry ,I didn't express it clearly, please disregard the first suggestion

from vector.

I have pushed up a possible fix for this issue. If you are able to build Vector from source, check out PR #20467 and try the resulting binary.

from vector.

The proposed fix is now merged into the master branch. Please confirm if this resolves the memory leak for you.

from vector.

Thanks for confirming! And for suggesting the original fix. I'll close this issue out.

from vector.