jdiazcano / cfg4k Goto Github PK

Minor issue with cfg4k 0.9.0: always get this WARN level message from PropertyConfigLoader at startup, using Oracle JDK 1.8:

Key (java.vendor.url.bug=Oracle Corporation) has overridden another value used as root

Exception in thread "TimeReloadStrategy" java.lang.RuntimeException: No state found: INIT EOF
at com.beust.klaxon.StateMachine.next(Parser.kt:74)
at com.beust.klaxon.Parser.parse(Parser.kt:211)
at com.jdiazcano.konfig.loaders.JsonConfigLoader.loadProperties(JsonConfigLoader.kt:42)
at com.jdiazcano.konfig.loaders.JsonConfigLoader.reload(JsonConfigLoader.kt:37)
at com.jdiazcano.konfig.providers.DefaultConfigProvider.reload(DefaultConfigProvider.kt:137)
at com.jdiazcano.konfig.loaders.TimedReloadStrategy$register$$inlined$timer$1.run(Timer.kt:149)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

This occurs "randomly" when running test, it might be impacted by the way tests are designed (they create the file and then they destroy it)

The Hocon config mapper does not handle nulls.

https://github.com/jdiazcano/cfg4k/blob/master/cfg4k-hocon/src/main/kotlin/com/jdiazcano/cfg4k/hocon/HoconConfigMapper.kt#L19

According to the docs, value.unwrapped() can return null (which it does when the hocon config is something like value = null). This should get passed through to support setting null when the config interface defines a nullable type.

Describe the bug
Since upgrading to com.jdiazcano.cfg4k:cfg4k-core:0.9.41 i am getting the following crash

Exception in thread "main" java.lang.IllegalArgumentException: Trying to get a key from a primitive
at com.jdiazcano.cfg4k.loaders.DefaultConfigLoader.get(DefaultConfigLoader.kt:27)
at com.jdiazcano.cfg4k.providers.DefaultConfigProvider.get(DefaultConfigProvider.kt:51)
at com.jdiazcano.cfg4k.providers.CachedConfigProvider.get(CachedConfigProvider.kt:37)
at com.abc.config.ApplicationConfig.getProperty(ApplicationConfig.kt:24)
at com.abc.repository.util.DataSource.(DataSource.kt:26)

To Reproduce
In order for me to reproduce this behavior I need:

1. Version of Cfg4k - com.jdiazcano.cfg4k:cfg4k-core:0.9.41
2. ConfigProvider - CachedConfigProvider
3. ConfigLoader - EnvironmentConfigLoader
4. ConfigSource - Environment Variables
5. Configuration files unless the source is a String - Environment Variables
6. Any interfaces/dataclasses used for binding/getting - nope

As example, it is easy to get an easy prototype with all the 5 points in the list:
I have the following configs for a DB.

        val url = Property("db.conn", "")
        val driverName = Property("db.driver.name", "org.postgresql.Driver")
        val userName = Property("db.username", "")
        val password = Property("db.password", "")
        val connectionTimeout = Property("db.conn.timeout.sec", TimeUnit.SECONDS.toMillis(30))
        val idleTimeout = Property("db.conn.idle.timeout.sec", TimeUnit.MINUTES.toMillis(10))
        val maxLifetime = Property("db.conn.max.lifetime.sec", TimeUnit.MINUTES.toMillis(30))
        val minimumIdle = Property("db.conn.minimum.idle", 4)
        val maximumPoolSize = Property("db.conn.maximum.pool.size", 32)
        val connectionTestQuery = Property("db.conn.test.query", "SELECT 1")


data class Property<T : Any>(
    val key: String,
    val default: T
)
I use cfg4k like the following:
 override fun <T : Any> getProperty(prop: Property<T>): T {

        return configProvider.get(prop.key, prop.default.javaClass, prop.default)
    }

It seems like when i am searching for
val connectionTimeout = Property("db.conn.timeout.sec", TimeUnit.SECONDS.toMillis(30))
This value does not exist in my Environment Variables, and should return the default.
it may be getting confused with
val url = Property("db.conn", "")
Which does exist in the Environment Variables.
Since it finds
ConfigObject(value=jdbc:postgresql://127.0.0.1:1234/abc)
as root and then fails.

This works on 0.9.3

Before the key in fun DefaultConfigLoader.get() was different
key = DB_CONN_TIMEOUT_SEC - 0.9.3
key = DB.CONN.TIMEOUT.SEC - 0.9.41
Before it wasnt split and now it is.

Expected behavior
Since val connectionTimeout = Property("db.conn.timeout.sec", TimeUnit.SECONDS.toMillis(30))
does not exist it should return the default value.

Additional context
Let me know if i am not using this Library correctly.

Dependencies should be inside the gradle folder and importing them is the correct way to use them so a cleanup must be done there.

Hi,
Love the concept of this library and that it appears to be actively maintained.

I'm trying to use it in a project, and running into an issue where it seems that it relies on interfaces having a $DefaultImpls which as far as I can tell will not be the case in the basic use cases listed in e.g. your examples?

I end up with a ClassNotFoundException on Config$DefaultImpls in https://github.com/jdiazcano/cfg4k/blob/master/cfg4k-core/src/main/kotlin/com/jdiazcano/cfg4k/binders/BindingInvocationHandler.kt#L63 when using the following code:
Config.kt:

enum class Environment {
    DEVELOPMENT,
    STAGING,
    PRODUCTION;

    fun slug() = name.toLowerCase()
}

interface Config {
    val database: DatabaseConfig
    val server: ServerConfig
}

interface DatabaseConfig {
    val url: String
    val username: String
    val password: String
}

interface ServerConfig {
    val port: Int
    val host: String
    val environment: Environment
}

App.kt (redacted):

        val sysLoader = SystemPropertyConfigLoader()
        val envLoader = EnvironmentConfigLoader()
        val baseProvider = OverrideConfigProvider(
            DefaultConfigProvider(sysLoader),
            DefaultConfigProvider(envLoader)
        )
        val baseConfig = baseProvider.bind<Config>()

        val propsDefaultLoader = PropertyConfigLoader(ClasspathConfigSource("/app.properties"))
        val propsEnvLoader = PropertyConfigLoader(ClasspathConfigSource("/app.${baseConfig.server.environment}.properties"))
        val provider = OverrideConfigProvider(
            DefaultConfigProvider(sysLoader),
            DefaultConfigProvider(envLoader),
            DefaultConfigProvider(propsDefaultLoader),
            DefaultConfigProvider(propsEnvLoader)
        )
        val config = provider.bind<Config>()

        print(config.server.host)
        print(config.database.url)

Am I missing something obvious? If I make a stubbed default function on the objects I can get past the first error (missing $DefaultImpls) but it doesn't get me much further as it still seems to expect Kotlin to auto-stub all of these, which isn't happening for me.

Thanks in advance for your help!

Support binding environment variables to configuration (12-factor app compatible).

Spring Boot does a particularly good job at this, by mapping properties defined internally to corresponding idiomatic environment variable syntax e.g. foo.bar.baz -> FOO_BAR_BAZ. They also support some flexibility in the mapping (relaxed binding).

Cfg4j on the other hand does a poor job of this, unless its improved since the last time I looked at cfg4j.

Hi,
EnvironmentConfigLoader and SystemPropertyConfigLoader loaders are returning empty string for any key. So they are not usable in case of OverrideConfigProvider, when first provider is one of above.

There should be general performance benchmarks with/without caching. With/without binding.

Also Bytebuddy vs java proxy performance should be tested

A new config provider should be created where you can just pass in a var args of config loaders and the provider will be able to select the setting of the first one.

I'm not sure if that's really performant but should be fine:

1- Try to grab from the first loader
1.1- If it's non empty then return it
1.2- else try to grab the setting from the next loader

After all the changes the coverage was down by 1%. It should be a good oportunity to invest some time to put it to 90% and bump the version.

Also #32 must be included.

BinTray deprecated, this library should move to Maven Central.

There are libraries with known CVEs in the published version of cfg4k -- mainly the s3 dependency brings in older versions of jackson and httpclient, and the jgit dependency also brings in an older version of httpclient.

This is fixed on my branch here: https://github.com/rocketraman/cfg4k/tree/library-updates-security.

The HOCON specification (https://github.com/lightbend/config/blob/master/HOCON.md#substitutions) supports referencing properties like this:

a: foo
b: ${a}bar

which should result in b equal to foobar.

However, when attempting to use resolution like this via cfg4k, I get the following stack trace:

Exception in thread "main" com.typesafe.config.ConfigException$NotResolved: need to Config#resolve(), see the API docs for Config#resolve(); substitution not resolved: ConfigReference(${a})
	at com.typesafe.config.impl.ConfigReference.notResolved(ConfigReference.java:33)
	at com.typesafe.config.impl.ConfigReference.unwrapped(ConfigReference.java:45)
	at com.jdiazcano.cfg4k.hocon.HoconConfigMapperKt.parseObject(HoconConfigMapper.kt:16)
	at com.jdiazcano.cfg4k.hocon.HoconConfigMapperKt.parseObject(HoconConfigMapper.kt:15)
	at com.jdiazcano.cfg4k.hocon.HoconConfigMapperKt.toConfig(HoconConfigMapper.kt:8)
	at com.jdiazcano.cfg4k.hocon.HoconConfigLoader.<init>(HoconConfigLoader.kt:35)
	at com.jdiazcano.cfg4k.hocon.HoconConfigLoaderKt.HoconConfigLoader(HoconConfigLoader.kt:22)
	at com.jdiazcano.cfg4k.hocon.HoconConfigLoaderKt.HoconConfigLoader$default(HoconConfigLoader.kt:20)

Right now there are two issues with Bytebuddy:

1. It will not reload
2. It does not accept nested binding so if you have an Interface that references another Interface it will just throw an exception

When using the ByteBuddy binder, every method invocation to a config object causes a new class to be created and loaded into memory. Instead, the binder should create the class only once, with the methods in the class reading the provider every time they are executed.

This is easy to reproduce. Create any config binding with ByteBuddy. Read a method on the config in a tight loop. Start the JVM with -verbose:class to see the problem.

A new config loader must be created to fetch files from S3.

Right now the readme is under construction and it should look nice!

https://github.com/raphw/byte-buddy

Hello!
Firstly I used an OverrideConfigProvider combining two proxy config providers based on both an environment config loader and a property config loader (from the properties file) and everything worked well.
Then I decided to refactor the code to be able to load configs from the environment with an opportunity to use a secondary properties source if the config is not consistent after the 'first' load.
I began with validating the loaded config structure but failed because I thought only about the reflection API possibilities. (Would appreciate a piece of advice if a one can perform the consistency validation somehow). So I managed to change the solution to a worse one - I just return 'environment provider' instead of mixed one in case of a file reading failure.

And here comes the main question. So if I try to load a config using ONLY environment variables there must be contained each 'intermediate node'.
So if I have the following picture in my environment ('properties' is a prefix):

PROPERTIES_GROUPONE_KEYONE -> "VALUEONE"

I just get an Exception: com.jdiazcano.cfg4k.utils.SettingNotFound: Setting 'properties.groupone' not found. on an attempt to get the value.
But if I add:

PROPERTIES_GROUPONE -> ""
PROPERTIES_GROUPONE_KEYONE -> "VALUEONE"
PROPERTIES_GROUPONE_KEYTWO -> "VALUETWO"

everything gets working. Is this behavior incorrect? I do not need to include in my properties file each 'intermediate node' though. But I do in case of working with the environment.

Thank you!

Basically using Spek1 has been pretty hard since the start so finally let's get rid of it once and for all.

The substitutor would be Kotlintest as it seems to be pretty good.

With the rewrite of all the tests, no code should change, and everything would be a port but also rethinking how they are organized as right now they are definitely not sorted. Also right now there is the code coverage that didn't really seem useful and with that it'll be possible to migrate to .kts and also to a beter kts version of the buildSrc with a git submodule (it should be possible)

Resolve environment variables with some kind of syntax

Is it possible to take a property with a Map<String, String>/Properties? I've tried creating parsers etc, but I couldn't get it to use my parser, and it seems the BindingInvocationHandler just throws SettingNotFound when trying to use any of the maps methods.

I even tried just creating a property that was a ConfigObject type and turn it into a map myself, but it threw a SettingNotFound: Setting 'asObject' not found.

While I know most of my config and its types beforehand and can add them to the interface, I'd like to be able to supply dynamic/unknown pairs of config through. Is this possible?

The FileChangeReloadStrategy has no exception handler. This means that if an exception is thrown when the underlying config is updated -- for example, if the config is updated with invalid syntax -- the reload thread dies and no more config reloads are handled until the ConfigProvider is constructed again.

Instead, I think I would log a warning in this situation, but continue to monitor the file for more changes. However, I also note that there is no logging in the lib now, so an alternate approach may be to trigger another listener on provider in this case e.g. addReloadErrorListener?

1. BigDecimal
2. BigInteger
3. DateTime
4. Date

Maybe some other, any ideas?

Before @detouched commit it was not possible to fetch a single file from a basic authorization source. Now after his commit it is now possible. Related, he had the idea as the cfg4k-git has JGit as dependency you don't always want to introduce new dependencies so having some kind of easy authorization when dealing with git files it would be nice.

For now two utility ConfigSources should be created:

• BitbucketConfigSource
• GithubConfigSource

With this approach it is now possible to use Git as ConfigSource without having to depend on JGit which is awesome!

Create another module for YAML config files

Settings should be also able to be configured from the system settings:

-Ddatabase.url=test.database.com

Description

Right now if you merge two ConfigLoaders there is no way to keep the reload. An empty reload will take its place.

One thing that could be done is that the reload is a function in the constructor or something that can also be merged/called one after the other, this needs a little bit of design!

Acceptance criteria

• Document the fix in this issue
• Code changes and tests

Move the plugin version number to the gradle/versions file.

Maybe in some way it would be nice to have a jdbc config loader to be able to use it with all the databases.

This has the inconvenience of nested properties.

a.b could be property B from table A but... what about a.b.c ?

Another thing is that it could have a properties table (defined when creating the loader) and the key would be a.b or a.b.c

Reloader when S3 file is updated. This can be done by using SNS notification from S3 to SNS and getting that update in the library.

HOCON seems to be a nice parser for json that can even be flattened into simple strings so would be nice to have a separated module for it

https://github.com/typesafehub/config#other-apis-wrappers-ports-and-utilities

When resolving hocon variables, this works:

something {
  foo = "whatever"
  bar = ${something.foo}
}

but this does not:

something {
  foo = "whatever"
  bar = "${something.foo}"
}

The FileChangeReloadStrategy watcher uses an incorrect test to check if the event corresponds to the configuration file.

The javadocs for WatchEvent.context state that the path returned is relative to the parent on which the watch is registered.

However, the current code just naively does fileName == file. The following code seems to be a pretty foolproof way of doing the correct test:

parent.resolve(fileName).toRealPath() == file.toRealPath()

Class not found!

I would be interested to load configs from Vault
Would that be best implemented as a source or a provider?

Right now if there is no setting it will just return an empty string but this was done to have a fast prototype but now something must be done.

The best option is to throw an exception if there is no setting and adding a default value.

Probably changing the method to have a default null value/Optional.absent() value and throw the exception when there is no value.

Write all the things in the javadocs. Right now there is none and is sad :(

HOCON substitutions work well within a single file. However, when the substituted value needs to be loaded from another higher priority source e.g. another HOCON config file, system prop, or environment, the substitution fails.

At least for a HOCON-only solution, one can inject the HoconConfigLoader with the pre-resolved substitutions:

HoconConfigLoader {
      ConfigFactory
        .parseResources("application-local.conf")
        .withFallback(ConfigFactory.parseResources("application.conf"))
    })

However, when combining this with environment and/or system prop overloads, that solution is not sufficient as the substitutions are applied by typesafe config before the data is ever seen by cfg4k.

One can override the final key value, but not the individual substitution.

Example:

application-local.conf

environment {
  prefix = "dev-"
}

application.conf

environment {
  prefix = ""
}
foo = ${environment.prefix}bar

with configuration something like:

OverrideConfigProvider(
    ProxyConfigProvider(EnvironmentConfigLoader()),
    ProxyConfigProvider(SystemPropertyConfigLoader()),
    ProxyConfigProvider(HoconConfigLoader("application-local.conf"))
    ProxyConfigProvider(HoconConfigLoader("application.conf"))
)

My ideal scenario would be that the following would be true:

1. No system or env properties, set, then

foo = dev-bar

2. No application-local.conf present, or is empty:

foo = bar

3. With System property or equivalent env var -Denvironment.prefix=test-:

foo = test-bar

The general idea here being that one can specify some defaults for substitutions, but override them with earlier config providers in the same way that one can specify defaults for entire config values but override them with earlier config providers.

When reloading, the list of configuration changes should be listed so whomever is reloading can act to that. (Even if it shouldn't be needed because it shouldn't be used as static fields but with the binding)

Any thoughts about moving to a multiplatform implementation

Describe the bug
In some environments (Kubernetes), a config file can actually be a chain of symlinks. When the ConfigMap backing the chain of symlinks changes, Kubernetes doesn't update the real file itself, but instead just changes the symlinks to point to a different file entirely.

In Kubernetes, this layout may look like this:

configfile -> ..data/configfile
 ..data -> ..2019_09_20_05_25_13.1
..2019_09_20_05_25_13.1/configfile
..2019_09_20_05_25_13.2/configfile

and when the backing ConfigMap changes, K8S creates a new timestamped directory containing the updated config file, points the ..data symlink to it, and then deletes the old timestamped directory.

To Reproduce
Create the directory structure shown above. Modify the ..data symlink to point to the .2 timestamped directory, containing a different version of configfile:

ln -sfn ..2019_09_20_05_25_13.2 ..data

The config should reload, but it does not.

Expected behavior
The watcher needs to follow the chain of symlinks up to the real file, and check if any of these symlinks have been changed in order to trigger a reload.

Additional context
I will submit a pull request for this.

A new module should be created in order to be able to grab properties/json files from Git, any public/private repository even with ssh key!

Resolve other variables with some kind of syntax so you can just create a property based on another like test=a, ${test}hello -> ahello

The ConfigProviderTest has no test for nested.a.