Comments (17)
I'm confused as to what you are proposing. what would you propose as default values? maybe that would help me understand your thinking here.
from kafka.
Ah sorry, so each of those would be a hash pointing to the same sorts of config items as now. Not to plug our own stuff, but something like this as an example. Mostly, just to consolidate the configuration parameters into a separate, isolated namespace, which lets you do things like render the config file without a template, etc.
The default values of the parameters would remain the same, it's just reorganizing how they're namespaced. Sorry for the confusion!
from kafka.
ah got it -- so you are trying to get consumer and producer library configs? I've only ever used this cookbook to setup brokers.
from kafka.
Oh, right. In that case, might be worth just having node[:kafka][:config]
.
from kafka.
Heya,
The attributes that are currently being used for rendering the main configuration file attempts to follow some kind of naming convention, mapping to the actual configuration keys used by Kafka, i.e. attributes that are in the socket
"namespace" all start with socket.
in the actual configuration file. Unfortunately the Kafka team have been quite inconsistent when naming configuration options, and they tend to add/remove configuration options between "patch" releases (which is a pain to attempt to maintain...), though I don't think they follow semantic versioning to any degree when it comes to configuration at least.
I've been playing with the idea of removing the current attempt at grouping options and just go with a completely flat namespace, and with this in mind I can see a point in grouping everything under a config
namespace, even though I haven't seen it used in many other cookbooks (it's been a while though). But really one could explicitly pass all configuration options using variables
on the template
resource.
What I'd really like is a solution that makes it "easy" to manage the rendering of configuration files between different releases, for example one "renderer" for 0.8.0, one for 0.8.1, etc, etc. That would also imply an easy way to manage configuration options between releases, and I'm not sure how to go about that. While one would always like to be using the latest and "bestest" version of software, it's not always possible to due a number of reasons that vary between companies and cultures.
In short I'd like to make something smart when it comes to configuration file rendering, but I'm not sure how to approach the issue. But I don't think putting all configuration options under a config
namespace is going to make it much better, albeit it might make it a bit easier to test (maybe).
I'd like to ponder this a bit and try out some things. If you have suggestions or opinions on how you'd like things to work (preferably with concrete examples) that'd be much appreciated! Sorry about the length of this comment, but I've had this on my mind back and forth a while.
from kafka.
By the way, I should probably mention that I've yet to actually deploy a Kafka cluster using this cookbook :). Should've done it to our staging environment this week, but looks like it might have to wait until next week.
from kafka.
That's one of the nice things about the flat pattern, is you can specify just the defaults, and then avoid further levels of namespacing. In that case, you could replace periods with, e.g., underscores and implement logic in the rendering method to replace them when it runs. So like,
node[:kafka][:config] = {
log_dirs: ['/mnt/kafka']
}
node[:kafka][:config].collect do |k,v|
"#{k.gsub('_', '.')}=#{v}"
end
Or what not.
The config versioning thing is interesting. Usually my feeling on that is to specify as little as possible, since Kafka provides defaults for non-mandatory parameters. In that case, if the mandatory parameters change, you're still in trouble, but otherwise the user of this cookbook is expected to know the changes they're bringing onboard. So if queued.max.requests
changed to queued.maximum.requests
, presumably I've upgraded my Kafka version, which shifts the onus on me to change that attribute, since this cookbook didn't name it specifically, but provided an easy/forwards-compatible way to insert or delete parameters.
from kafka.
Yeah, that's definitely a big win with having everything scoped under a config
namespace, though one has to do some special logic when it comes to Arrays and whatnot, but that's definitely doable (it's Ruby after all :)).
Initial feels about this is that it would make it a lot more maintainable when it comes to different configuration options between releases as it would put the burden on the user of the cookbook rather than the maintainer (as it should be, in my opinion). Then there could be specialized unit tests for different versions, etc, etc.
From the start I defined all the current defaults (according to the Kafka documentation), but later switched to just explicitly name all the configuration options in attributes/default.rb
with either a nil
value or an empty Array (where Arrays are appropriate).
from kafka.
I've fleshed out a first stab at dynamically generating the main configuration file in #47, and I must say that it looks pretty promising.
Everything (i.e. configuration attributes) is currently scoped under a broker
namespace, as config
was already used for the filename of the main configuration file (might change that though). What do you guys think?
from kafka.
So, I just merged #47 and now all configuration attributes should be defined under the broker
namespace. And I just realized that I should probably update the README to reflect these changes, or at least document it in the default
attributes file. Oh well.
from kafka.
@mthssdrbrg, thanks a ton!
from kafka.
No probs. I'm gonna hold off on releasing a new version for a little bit though, got some other bits and pieces that I want to incorporate before the next release.
from kafka.
That's okay, we use git refs ;)
from kafka.
Cool. If you find anything weird while testing it out let me know / open a pull request ;). There might've been some breaking changes that I haven't considered yet, so any feedback is welcome.
from kafka.
Did you guys end up doing anything for configuring the consumer.properties file? I can't seem to find anything in the cookbook for that. I presume right now its just the default from the package install?
I have deployed a 3 node kafka / 3node zookeeper with the cookbook (though I haven't passed data thru it yet :-) Hopefully tonite!
from kafka.
No, as the intention of this cookbook is to set up a Kafka broker and not for client applications. That being said, you could use the Configuration
in a similar way to what's done here, but that might just be a bit overkill :)
from kafka.
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
from kafka.
Related Issues (20)
- How is broker.id generated? HOT 6
- Kafka 1.0 download link does not work HOT 2
- TLS Support HOT 3
- no implicit conversion of nil into String HOT 2
- Kafka 1.0.1 HOT 3
- Kafka restarts immediately twice (v2.2.2) HOT 5
- Sous-chefs would like to adopt your cookbook. HOT 5
- Java 10 compatibility HOT 3
- Changelog mentions haproxy instead of kafka HOT 1
- Runit support not working HOT 1
- Dangerfile should use failure instead of fail
- Remove .rubocop.yml with Dangerfile HOT 1
- Update Changelog HOT 1
- Run latest cookstyle
- Update builds to be parallel
- Kafka Service Stop Clean Shutdown
- Stop the lock[bot] spam! HOT 1
- Files being generated without newlines HOT 4
- Double starts while upgrading an existing broker with new setup.
- Dependency Dashboard
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka.