options to limit max memory uaage about franz-go HOT 5 CLOSED

As a consumer or as a producer?

As a producer, you can limit MaxBufferedMessages, and then application side, limit the size of what you're producing, so that you know there's an upper bound of messages*record_size.

As a consumer, you can limit AllowedConcurrentFetches, and then limit FetchMaxBytes, so that you know at most n_concurrent_fetches*fetch_max_bytes will be buffered internally (minus the one caveat that if a single record exceeds FetchMaxBytes, kafka will return it).

from franz-go.

(I may change AllowedConcurrentFetches to MaxConcurrentFetches for the 1.0 release--allowed is odd naming considering the rest of the API)

from franz-go.

One last note: inflation from decompression cannot really be controlled. If a small fetch inflates to 1G when decompressed, the client can't know that going into the decompression. It's best to limit this on the producer side by setting BatchMaxBytes, which is the maximum uncompressed size when producing before compression is involved.

from franz-go.

no i'm not use compression =) yes question about consumer

from franz-go.

all fine thanks

from franz-go.