avaje / avaje-config Goto Github PK

I like how it looks.

We did dabble with this when adam asked for specialized yml parsing, but I think this is a good idea to allow for more config file types.

This is added as a form of convention over configuration.

The convention adopted is that in Kubernetes environment variables can be exposed:

• POD_NAME
• POD_NAMESPACE
• POD_VERSION
• POD_IP

We use these to set system properties - the most useful are appName (derived from POD_NAME) and appEnvironment (copied from POD_NAMESPACE).

If there is a POD_NAME env var then we can use that to set appName and appInstanceId.
If there is a POD_NAMESPACE env var then we set that as appEnvironment
If POD_IP then we set appIp
If POD_VERSION then set appVersion

When test resources where detected (application-test.yaml, application-test.properties) then the command line arguments were not being read / checked.

This prevents the case where we run something in src/test (usually a test) and provide properties configuration via command line arguments. This is a pretty rare scenario, in fact I don't think anyone has hit this scenario yet but I think the existing behaviour is confusing.

Changing to: Always going to read command line arg properties.

I want to be able to load env specific props based on profile. Say I have an application properties file like this

some.common.prop=true
load.properties=src/main/resources/application-${profile:local}.properties

In the current state, files are only supported, so when I upload my jar to my server it doesn't pull the props because there is no such folder.

I'd like something like this to work as well as standard path based config

some.common.prop=true
load.properties=application-${profile:local}.properties

Caused by: java.lang.IllegalArgumentException: Expecting only yaml or properties file but got [ort]
at io.avaje.config.load.Loader.loadFileWithExtensionCheck(Loader.java:230)
at io.avaje.config.load.Loader.loadViaPaths(Loader.java:192)
at io.avaje.config.load.Loader.loadViaCommandLine(Loader.java:140)
at io.avaje.config.load.Loader.loadViaCommandLineArgs(Loader.java:126)
at io.avaje.config.load.Loader.loadLocalFiles(Loader.java:119)
at io.avaje.config.load.Loader.load(Loader.java:83)
at io.avaje.config.CoreConfiguration.load(CoreConfiguration.java:25)
at io.avaje.config.Config.(Config.java:18)

This is because running a test in Eclipse runs the following command...

[org.eclipse.jdt.internal.junit.runner.RemoteTestRunner, -version, 3, -port, 53186, -testLoaderClass, org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader, -loaderpluginname, org.eclipse.jdt.junit4.runtime, -test, debrepo.teamcity.ebean.server.migration.MainDbMigrationTest:testGenerateMigrationFiles]

What our config framework provides is a simple event system. When logging is done you can drain the queue of events usually replaying them to logging our System.err.

The reason for doing this is because sometimes we want the configuration to load completely BEFORE any logging is used. This allows the configuration to put things into System Properties which is used to configure the actual logging.

This allows for 1 extra level of indirection to control the System.Logger implementation when we can't configure it via System.LoggerFinder.

When there is no value for myapp.doesNotExist3 then expect:

  @Test
  public void get_default_repeated_expect_returnDefaultValue() {
    assertThat(Config.get("myapp.doesNotExist3", null)).isNull();
    assertThat(Config.get("myapp.doesNotExist3", "other")).isEqualTo("other");
    assertThat(Config.get("myapp.doesNotExist3", "foo")).isEqualTo("foo");
    assertThat(Config.get("myapp.doesNotExist3", "junk")).isEqualTo("junk");
  }

Hi,

when trying to evaluate avaje i could not help but notice, that there is a hard dependency on avaje-inject, albeit the require static statement in the modules-info.

A MRE can be found here:
https://github.com/NoonRightsWarriorBehindHovering/mre-avaje-config-1

The issue is expected as far as i can see, given that the official documentation says this:
A module M declares that it 'uses p.S' or 'provides p.S with ...' but package p is neither in module M nor exported to M by any module that M "reads".
Source: https://docs.oracle.com/javase/9/docs/api/java/lang/module/package-summary.html

As such without avaje-inject being read by any module explicitely (requires static does not "read", because it it optional), the above fails.

A simple workaround would be to require it in the consuming application.
This might not be feasible, when inject is not being used.
Alternatively the provides statement could (naively) be moved somewhere else (e. g. a new jar or avaje-inject).

In any case the intended outcome should be documented.

Sorry for opening two bugs already in quick succession. :)

For a variety of reasons I often need to do my own initialization.

Avaje-config uses a plain static single and not the static service locator pattern like logging facades and their frameworks do.

This is problematic if you want to guarantee something always happens before or after avaje-config has loaded.

Where this usually rears its head is unit testing. It can be difficult and error prone to control the initialization of Service Locators. It's also a problem if you want to plug in a different implementation for testing or perhaps a threadlocal version for some sort of context config.

The simple solution is to make an interface like ConfigProvider or ConfigurationProvider (I'm still confused what the difference is between the two) that gets loaded with the ServiceLoader and if there are no implementations then a default one is loaded.

Consequently I recommend making an AbstractConfiguration that makes it easy for others to make their own custom Configuration implementations (again not sure if Config or Configuration is the better choice here).

This allows us to plugin custom sources for configuration (like rest endpoint, redis, vaults etc).

Hi,

Could you please add something like micronaut's @Value / @Property (see micronaut's document)?

Refer 599836a

Failing test case

statusPageUrl=https://${eureka.instance.hostname}:${server.port}/status

java.lang.StringIndexOutOfBoundsException: begin 17, end 15, length 51

	at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3720)
	at java.base/java.lang.String.substring(String.java:1909)
	at io.avaje.config.load.CoreExpressionEval$EvalBuffer.parseForDefault(CoreExpressionEval.java:164)
	at io.avaje.config.load.CoreExpressionEval$EvalBuffer.evalNext(CoreExpressionEval.java:209)
	at io.avaje.config.load.CoreExpressionEval$EvalBuffer.process(CoreExpressionEval.java:214)
	at io.avaje.config.load.CoreExpressionEval.eval(CoreExpressionEval.java:102)
	at io.avaje.config.load.CoreExpressionEval.eval(CoreExpressionEval.java:62)
	at io.avaje.config.load.LoadContext.evalAll(LoadContext.java:129)
	at io.avaje.config.load.Loader.eval(Loader.java:238)
	at io.avaje.config.load.LoaderTest.loadProperties(LoaderTest.java:72)

Currently, to have profiles we do.

load.properties=application-${profile:local}.properties

and set the command line args.env variables.

This is pretty good, but doesn't support multiple profiles.

I propose we add native support for profiles such that if profile=dev,docker the files application-dev.properties and application-docker.properties will be loaded

  /**
   * Run eval of the given properties modifying the values if changed.
   */
  void evalModify(Properties properties);

Example use:

  @Test
  void evalModify() {
    final Properties properties = basicProperties();
    properties.setProperty("someA", "before-${foo.bar}-after");
    properties.setProperty("yeahNah", "before-${no-eval-for-this}-after");

    String beforeYeahNahValue = properties.getProperty("yeahNah");

    final CoreConfiguration config = new CoreConfiguration(new Properties());
    config.evalModify(properties);

    String someAValue = properties.getProperty("someA");
    assertThat(someAValue).isEqualTo("before-42-after");

    String afterYeahNahValue = properties.getProperty("yeahNah");
    assertThat(beforeYeahNahValue).isSameAs(afterYeahNahValue);
  }

stuff like this doesn't work

database.schema=foobar
database.host=localhost
database.portPrefix=${global.portPrefix}
database.port=${global.portPrefix}5432
database.url=jdbc\:postgresql\://${database.host}\:${database.port}/${database.schema}
database.username=${database.schema}
database.password=${database.schema} 

Funnily enough, it works if you do it programmatically via setProperty. I guess because CoreMap uses a HashMap iterating over it doesn't guarantee that the keys are read and interpolated in the correct order

Hey Rob,

Massive fan and like all your libraries.

I have been making a code generator (aka annotation processors) version of Spring's Petclinic to highlight various annotation processor libraries such as yours.

I wanted to use our own in-house config framework but I don't have the time to make it open source and accidentally stumbled on yours which has a similar initialization and design as mine. Mine has a general URI SPI mechanism, more filtering options, and the Config part is separated from the Key Value loading. There is also an annotation processor to map config to objects. It is probably massively over designed :).

Anyway here are some missing features I miss.

Feel free to let me know if anything is out of scope or not worthwhile or you want separate bugs filed for each.

Maven Plugin

One of the major features I'm missing is our config framework has a maven plugin very
much analogous to the codehaus maven properties plugin.

<plugin>
 <groupId>com.snaphop</groupId>
 <artifactId>snaphop-config-maven-plugin</artifactId>
 <executions>
   <execution>
     <phase>initialize</phase>
     <goals>
       <goal>read-config</goal>
     </goals>
     <configuration>
       <urls>
         <url>file:///${project.basedir}/src/main/resources/db/database.properties</url>
       </urls>
       <!-- finds all properties with prefix and removes it -->
       <propertiesKeyPrefix>database.</propertiesKeyPrefix>
       <!-- prefix the found properties -->
       <pomKeyPrefix>my_project.database.</pomKeyPrefix>
     </configuration>
   </execution>
 </executions>
</plugin>
<!-- now use the properties for other plugins like jOOQ or Flyway -->

I think I can trivially port our maven plugin and we could add it as a module to avaje-config (well not yet as avaje-config is not multimodule but that would be easy to fix). If you are interested I could do a PR.

More configurable initialization

On the one hand I mostly like the current opinionated behavior for simplicity on the other hand it is not exactly the canonical behavior I want. For example I prefer ~/.config/application instead of ~/.localdev. Besides the config source SPI I think a worthwhile add on would be an SPI to provide your own initialization.

Annotation processor for config objects

Another thing our config framework has is a general purpose flat mapping of key values to objects.

@ConfigBean
public interface MyClientConfig {
  public String username();
  public int port();
}

What happens is an implementation is generated that essentially will map method calls like username to Map<String,String>#get or Config#getString. I take @Nullable into account.

Anyway this is again I think something I could port to use your library.

Able to load env like files AND generate them

One of the annoying things is not everything can read java like config. Particularly shell scripts.

One of the things we have is something that will generate shell environment config from your property files config. That is initialize the framework and export out an ".env" file. This requires deciding how you want to name mangle or convert property names to environment variable names.

Side note

As a tangent I will say on the dynamic properties front which we have as well we never use anymore because reload whole app is fast these days. I wonder do you use that feature?

if (Config.enabled("feature.cleanup")) {
  ...
}

Which is just the same as:

if (Config.getBool("feature.cleanup")) {
  ...
}

I decided to file this as a feature/bug instead of continuing in the long SPI thread.

In my experience you rarely need to "react" to a single property event (property being name value pair).

Let us say I have some sort of external system with credentials and location information. Let us use a datasource ignoring the complexity reloading of database pool as an example.

I have my properties like "database.user" and "database.password" and many more. I stress many properties.

Now I load all of these guys up into some object in one event change.

How do I do the above without reacting to every single property change?

I think the event system needs to be more batching or transactional.

What I recommend is removing all the setProperties functions and potentially the onChangeXXX or deprecating them.

Here is an abbreviated version of our in house one with some name changes:

    void onEvent(Consumer<? super Event> consumer);

    // captures a snapshot of the current config to prepare for edit
    EventBuilder eventBuilder();

    void publish(Event event);

    public interface EventBuilder {

        // This snapshot is mutable and copy of the current config
        Map<String, ConfigEntry> snapshot();

        EventBuilder description(String description);

        // whether or not to take the changes of the snapshot
        EventBuilder update(boolean update);

        Event build();

    }

So instead of calling setProperty("foo", "bar") you do something like this:

eb = config.eventBuilder();
eb.snapshot().put("foo", "bar");
eb.snapshot().put("another", "one");
config.publish(eb.build());

I have simplified and removed some ergonomics as well as combined the "change request" with the downstream event but hopefully you get the idea.

Now downstream you could add information about which properties have changed but I have found that just re pulling the config you need good enough.

config.onEvent( e -> { if (e.update) service.reload(config); });

In the old system I would get an event for each property change potentially causing lots of problems particularly if the most common case is some file reload of lots of properties.

BTW to do an event system correctly otherwise bizarre shit happens is to use some sort of concurrent queue (blocking or not depending on implementation). While concurrent hashmap saves you from concurrent modifications it doesn't mean it guarantees consistency.

Anyway if you are interested I can do a PR on how I would implement it.

In some container file systems we can not use last modified timestamp for detection of config file changes. This change additionally uses the file length, which will only be reliable if the actual file length changes of course.

Note that in the CI build we observe this behavior.