Comments (4)
If I'm reading this right, you'd like to build / destroy state in OnAssigned
and OnRevoked
, but building / destroying may return an error?
I think I'd try retrying on error within OnAssigned
and OnRevoked
, and if you reach a critical error, LeaveGroup
. If either of your hooks take too long, the client will be forced from the group by Kafka as it is.
I'm not sure I fully understand the goal here, though?
from franz-go.
Yes you perfectly understood it :) I will try to clarify my use case a bit more precisely.
The goal might be for example a transactional counter, where you have a stateful compacted topic (containing the total count per partition) and stetaless topic (containing each event to count per partition).
On assignment, we bootstrap the state by reading the compacted topic to get the current total per partition (by kafka key) and then continue reading from the events topic (kafka streams-ish).
On a large number of counters, we would prefer using disk for storing states and this can cause errors (even consumption of the compacted topic can cause errors). Maybe, on such errors, it is arguable that we can consider this an unrecoverable error and to LeaveGroup
or to panic
after a number of retries.
from franz-go.
I think I can see a few potential ways of doing things.
If OnAssigned takes too long, then the client is booted from the group from Kafka. When you finally finish OnAssigned, you will immediately proceed to OnRevoked, and then once that finishes, the client will retry joining the group. So, if you are fine with potentially causing rebalances when OnAssign/OnRevoke takes too long, that's one option.
If you have an unrecoverable error in either OnAssigned or OnRevoked, then leaving the group is the best bet, because it doesn't make sense to rejoin the group.
If you want to stop progress in OnAssigned/OnRevoked if the member is booted (so, you're taking too long and then the client is kicked from the group), that's not currently possible. The context passed to the callbacks is the context for the group member itself, not for the session it is in. I can look into potentially changing this down the line if that would help.
But, if you have an error in the callback, I can't think of a solid thing to do when the callback returns if it was able to return an error. The only two options for the client itself are to leave the group or to continue within the group, and you can ensure that behavior manually by either calling LeaveGroup or not?
from franz-go.
Alright, thank you very much for your propositions. I'll give it a shot.
from franz-go.
Related Issues (20)
- Retry fetched records on handle error HOT 1
- github.com/twmb/franz-go/pkg/sr#Client.AllSchemas 404 not found HOT 3
- Deadlock in Produce() / TryProduce() when kgo.MaxBufferedBytes() is configured HOT 2
- Unable to specify LZ4 compression level HOT 3
- Error on kadm/UserSCRAMs calls with redpanda HOT 3
- Question: Is it safe to poll after receiving ErrGroupSession on poll when a broker node restarts in the cluster? HOT 1
- Data Race: concurrent access to recBatch.canFailFromLoadErrs during retry errors HOT 4
- Is there a way to test kgo.Opts returned from wrapper function? HOT 1
- fetch using topic id HOT 3
- How to delete offset when message got consumed? HOT 4
- Imbalanced Partition Assignment to Consumers Per Topic HOT 1
- Want to delete the consumer group after processing records from kafka HOT 1
- Connecting franz-go with a confluent kafka cloud cluster HOT 2
- Decoding GroupMetadataValueMember fails HOT 1
- Cache metadata more HOT 1
- Question regarding manual commit example HOT 1
- Update GroupMetadataKey / GroupMetadataValue
- Bump epoch if the log_start_offset advanced and broker returned unknown_producer_id
- kgo.LiveProduceConnection option for low-latency settings HOT 4
- BrokerResponse too large HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from franz-go.