Comments (4)
Absolutely happy to get feedback!
(1) and (2), an examples folder has long been on my (undocumented) todo list. I'll get to these.
(3), right I'll add documentation on NewClient
. The client itself is basically just an initialized struct on return; nothing has actually happened yet. But you're right, the update metadata loop goroutine has been spun up, so soon enough, brokers will be discovered.
(4), this sounds like maybe request and response hooks would be beneficial here? Optional functions that are called before requests are issued, and before responses are parsed?
from franz-go.
Thanks for your replies.
Regarding (4): I don't think I need hooks, but as of now I can only get broker handles that do not expose any metadata (not even the broker id is public). I think if you store the metadata after connecting to the broker in the Broker
struct and make this accessible that's good enough. Otherwise/additionally maybe an OnBrokerConnected()
hook which contains the metadata?
I'm looking forward to what's coming 🚀
Edit: BrokerMetadata seems to contain the following data, which is what I'm looking for:
type MetadataResponseBroker struct {
// NodeID is the node ID of a Kafka broker.
NodeID int32
// Host is the hostname of a Kafka broker.
Host string
// Port is the port of a Kafka broker.
Port int32
// Rack is the rack this Kafka broker is in.
Rack *string // v1+
}
from franz-go.
re(4) still, it may be better to talk in the discord about this, but I'm curious: do you want just the information in the metadata response, or do you want information about connected brokers and whatnot? If it's the former, would issuing your own MetadataRequest be sufficient?
If the latter, I think it's worth talking more about what the goal is so that I can write an API that may be more generally useful / expandable in the future. As it currently is, brokers aren't connected to until a request is just about to be written, and a Broker
struct itself is a very thin abstraction to direct requests to a specific broker.
I was thinking about the hooks a bit more, one complicating factor is that the sink and source files use their own custom internal produceRequest
and fetchRequest
types. I've thought that hooks would be useful for having metrics around latencies for requests & responses, but those hooks would just need to know the request key that was issued (not the full request, and this is actually the reason I avoided having a ConnTimeoutFn(func(kmsg.Request) time.Duration)
option). So, a general hook is out.
Another option is to change or supplement the Dialer
option,
either 1) change it to func Dialer(fn func(context.Context, MetadataResponseBroker) (net.Conn, error)) Opt
or 2) supplement it with func MetadataDialer(..the same..) Opt
This would allow a direct hook into which broker is actually dialed when, and allow you to hook into the net.Conn to know when the broker connection is lost.
But anyway, most importantly is, what is the goal with this information?
from franz-go.
re(4) still, it may be better to talk in the discord about this, but I'm curious: do you want just the information in the metadata response, or do you want information about connected brokers and whatnot?
What I usually want to achieve is to connect to a Kafka cluster and afterwards print something that indicates "everything worked, I'm connected to at least one broker and discovered these 6 other brokers". Often users have troubles connecting to their cluster because of missing certificates, connectivity, wrong credentails etc. so that's really important for me.
I also want to await this point, so that I can then proceed with the rest of my application (e. g. starting a HTTP server afterwards / switching my probe endpoints to "ok" etc).
from franz-go.
Related Issues (20)
- Retry fetched records on handle error HOT 1
- github.com/twmb/franz-go/pkg/sr#Client.AllSchemas 404 not found HOT 3
- Deadlock in Produce() / TryProduce() when kgo.MaxBufferedBytes() is configured HOT 2
- Unable to specify LZ4 compression level HOT 3
- Error on kadm/UserSCRAMs calls with redpanda HOT 3
- Question: Is it safe to poll after receiving ErrGroupSession on poll when a broker node restarts in the cluster? HOT 1
- Data Race: concurrent access to recBatch.canFailFromLoadErrs during retry errors HOT 4
- Is there a way to test kgo.Opts returned from wrapper function? HOT 1
- fetch using topic id HOT 3
- How to delete offset when message got consumed? HOT 4
- Imbalanced Partition Assignment to Consumers Per Topic HOT 1
- Want to delete the consumer group after processing records from kafka HOT 1
- Connecting franz-go with a confluent kafka cloud cluster HOT 2
- Decoding GroupMetadataValueMember fails HOT 1
- Cache metadata more HOT 1
- Question regarding manual commit example HOT 1
- Update GroupMetadataKey / GroupMetadataValue
- Bump epoch if the log_start_offset advanced and broker returned unknown_producer_id
- kgo.LiveProduceConnection option for low-latency settings HOT 4
- BrokerResponse too large HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from franz-go.