Comments (7)
Typesafe config looks like a very nice written library.
For broker config, I'd be ok to use it to populate the ServiceConfiguration. I'd like to maintain to have a single point to have to update when adding a new config variable. Currently, ServiceConfiguration
is a pojo, and you just need to add a new variable with default value and example in conf/broker.conf
.
Alternatively, what about having ServiceConfiguration
with only getters and adding a ServiceConfigurationBuilder
?
from pulsar.
Pushed the branch to remote but no pr, just so you can have a look. The most interesting points:
ServiceConfiguration
wraps and instance of Typesafe Config: https://github.com/radekg/pulsar/blob/7f58a69a2b51c1b3dfca7e03438e9cd1ec5a3afe/pulsar-broker-common/src/main/java/com/yahoo/pulsar/broker/ServiceConfiguration.java- pulsar-broker-common/src/main/java/com/yahoo/pulsar/broker/ServiceConfigurationLoader.java is gone, not required
- config takes the following form: https://github.com/radekg/pulsar/blob/7f58a69a2b51c1b3dfca7e03438e9cd1ec5a3afe/pulsar-broker-common/src/test/resources/broker.conf
- config init: https://github.com/yahoo/pulsar/compare/master...radekg:immutable-config?expand=1#diff-b9da9463a373b0b19d516884fb70c189R33
- an example of setting up configuration for a test: https://github.com/yahoo/pulsar/compare/master...radekg:immutable-config?expand=1#diff-0a160247a547f35234439402e845f463R83 (it can be set up from the resource, this is what the final PR would be)
- overriding config from a base test class: https://github.com/yahoo/pulsar/compare/master...radekg:immutable-config?expand=1#diff-00841fa6f00fe82dab598394fc501144R107
This isn't a finished PR, tests need much more work. Also, the validation of minValue
and maxValue
isn't implemented but would be rather straightforward to implement.
To comply with what Typesafe Config needs, we would have to add reference.conf
file to the resources
.
Have a look, let me know what you think.
from pulsar.
For config file, would it be possible to maintain the same java properties format? That would keep it consistent with the other config files (client, bookkeeper, ..). It seems the only change is right now the pulsar {....}
enclosing namespace.
I think that having to specify the config var name (either a string or a constant), makes a it bit more complicated to configure programmatically, since you have to lookup for the list of allowed variables, while with setters it's easy to discover all the options.
from pulsar.
But other than the 2 notes above, it seems nice to combine multiple config sources
from pulsar.
@merlimat I'd suggest using the hocon format across all configuration. The reason is that if someone builds software where Pulsar is only one of the components using Typesafe Config, they can single hocon file for all their components. We can keep the properties look and feel but, at least, I'd suggest prefixing all settings with pulsar.
to give them a namespace.
As for having to know the property names, yes, this is slightly inconvenient but, I think, you are looking from the perspective of unit tests. Preferably, all configuration variants would come as a separate .conf
file. Combined with a reference.conf
, everything is covered. However, if that is not enough, we can always implement a builder which would construct the necessary properties. The base is what it is, how config properties are constructed, we can change that to a friendlier looking version.
Shall I come up with a full PR?
from pulsar.
I would rather not break the config file compatibility unless is really necessary. Also, the other concern is that bookkeeper.conf
and zookeeper.conf
will still be in Java properties format and that cannot be easily changed.
from pulsar.
This seems to have gone quiet. Closing. Please reopen if you want this work to continue.
from pulsar.
Related Issues (20)
- [Bug] Broker became irresponsive due to too many open files error HOT 2
- [Doc] Document the removal of compaction
- [Bug] Major compaction is not recovered automatically after the disk is writable again
- [Bug] `status.html` can't access using 3.3.0 image
- [Bug] Ledger can not recover with Digest Mismatch Error HOT 5
- [Bug] Pulsar Functions Runtime doesn't properly enable direct byte buffer access for Netty on Java 17+
- [Bug] Pulsar broker CPU stratification problem HOT 5
- [Bug] [broker] broker log a full thread dump when a deadlock is detected in healthcheck every time
- [Doc][Improve] Backlog increase during subscription replication
- [Doc] add golang in transaction support list
- [improve]Perform health checks on the endpoints passed in by serviceUrl
- [Bug] Unable to initialize Stream metadata
- Jetty Upgrade: 12.x.x or latest HOT 6
- [Bug][broker] cursor will read in dead loop when do tailing-read with enableTransaction
- [Bug] Pulsar Functions ignores compressionType and crypto config for producers created with Context produce/newOutputMessage methods
- [Bug] Update partitions call is failing when topic level replication is disabled HOT 1
- As a websocket consumer I need to set InitialSubscriptionPosition to earliest HOT 1
- [Bug][broker] BrokerId npe when broker restart HOT 1
- [Bug] Unexpected Package Manager Behavior in Pulsar 3.3.0 Standalone Mode HOT 1
- [Bug] Dead lock error in Pulsar 3.0.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pulsar.