Comments (2)
There are two potential culprits,
- On any new group, Kafka internally forces a 3s wait, check the config value
group.initial.rebalance.delay.ms
for more information - If you are restarting your consumer using the same group, then your new consumer gets a new member ID in the existing group. Kafka triggers a join group, and then only after the previous consumer "dies" (misses the session timeout) does Kafka allow the rebalance to continue. In your above code, I don't see any room for a ctrl+c to be handled / it doesn't look like there's a proper
LeaveGroup
. Thedefer cl.Close
internally does aLeaveGroup
, but your for loop is infinite, so the close is never properly leaving the group.
Given the timeout in the logs is 38s, I suspect it's (2).
Let me know if you think it's something else.
from franz-go.
I add a logger, it spent a lot of time on read JoinGroup
time=2024-05-23T10:45:17.230+07:00 level=DEBUG msg="read FindCoordinator v4" broker=1 bytes_read=57 read_wait=38.879µs time_to_read=4.379832ms err=<nil>
time=2024-05-23T10:45:17.231+07:00 level=DEBUG msg="opening connection to broker" addr=localhost:9093 broker=1
time=2024-05-23T10:45:17.231+07:00 level=DEBUG msg="connection opened to broker" addr=localhost:9093 broker=1
time=2024-05-23T10:45:17.231+07:00 level=DEBUG msg="connection initialized successfully" addr=localhost:9093 broker=1
time=2024-05-23T10:45:17.231+07:00 level=DEBUG msg="wrote JoinGroup v9" broker=1 bytes_written=137 write_wait=587.209µs time_to_write=13.092µs err=<nil>
time=2024-05-23T10:45:17.240+07:00 level=DEBUG msg="read JoinGroup v9" broker=1 bytes_read=66 read_wait=38.105µs time_to_read=8.385049ms err=<nil>
time=2024-05-23T10:45:17.240+07:00 level=INFO msg="join returned MemberIDRequired, rejoining with response's MemberID" group=my-group-identifier member_id=kgo-2d3e0e05-dd52-4cb6-bb09-fcc2969a1d7a
time=2024-05-23T10:45:17.240+07:00 level=DEBUG msg="wrote JoinGroup v9" broker=1 bytes_written=177 write_wait=26.809µs time_to_write=10.857µs err=<nil>
time=2024-05-23T10:45:52.167+07:00 level=DEBUG msg="read JoinGroup v9" broker=1 bytes_read=193 read_wait=30.731µs time_to_read=34.926236626s err=<nil>
time=2024-05-23T10:45:52.167+07:00 level=INFO msg="joined, balancing group" group=my-group-identifier member_id=kgo-2d3e0e05-dd52-4cb6-bb09-fcc2969a1d7a instance_id=<nil> generation=2 balance_protocol=sticky leader=true
time=2024-05-23T10:45:52.167+07:00 level=INFO msg="balancing group as leader"
time=2024-05-23T10:45:52.167+07:00 level=INFO msg="balance group member" id=kgo-2d3e0e05-dd52-4cb6-bb09-fcc2969a1d7a interests="interested topics: [foo], previously owned: "
time=2024-05-23T10:45:52.167+07:00 level=INFO msg=balanced plan=kgo-2d3e0e05-dd52-4cb6-bb09-fcc2969a1d7a{foo[0]}
time=2024-05-23T10:45:52.167+07:00 level=INFO msg=syncing group=my-group-identifier protocol_type=consumer protocol=sticky
from franz-go.
Related Issues (20)
- RequestTimeoutOverhead appears to override Context cancellation HOT 5
- Broker Throttling
- Retry fetched records on handle error HOT 1
- github.com/twmb/franz-go/pkg/sr#Client.AllSchemas 404 not found HOT 3
- Deadlock in Produce() / TryProduce() when kgo.MaxBufferedBytes() is configured HOT 2
- Unable to specify LZ4 compression level HOT 3
- Error on kadm/UserSCRAMs calls with redpanda HOT 3
- Question: Is it safe to poll after receiving ErrGroupSession on poll when a broker node restarts in the cluster? HOT 1
- Data Race: concurrent access to recBatch.canFailFromLoadErrs during retry errors HOT 4
- Is there a way to test kgo.Opts returned from wrapper function? HOT 1
- fetch using topic id HOT 3
- How to delete offset when message got consumed? HOT 4
- Imbalanced Partition Assignment to Consumers Per Topic HOT 1
- Want to delete the consumer group after processing records from kafka HOT 1
- Connecting franz-go with a confluent kafka cloud cluster HOT 2
- Decoding GroupMetadataValueMember fails HOT 1
- Cache metadata more HOT 1
- Question regarding manual commit example HOT 1
- Update GroupMetadataKey / GroupMetadataValue
- Bump epoch if the log_start_offset advanced and broker returned unknown_producer_id
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from franz-go.