Command-line arguments. Environment variables. Configuration files. Java properties. Defaults of all kinds. Almost every program requires some configuration which is usually spread around multiple sources. Keeping track of all of them, mapping ones to others, making sure they are present and correct, passing them around is a constant cause of headache and inconvenient code design.

Configuration libraries, which there are plenty of, promise to solve the configuration problem, and they partly do. However, they are usually too simple, and provide only the mapping of configuration sources to Clojure data, leaving out the verification part. Omniconf tries to deliver the complete solution, even at the cost of increased complexity.

Omniconf is developed with the following principles in mind:

1. Configuration should be defined in a single place, but populated from many. A single form should specify which configuration options the application expects. But the options themselves can come from files, CLI, environment variables, etc.
2. Configuration can be accessed from anywhere in the application. We don’t want to carry that port number five layers through the callstack, intermixing dynamic vars, optional and keyword arguments, and different defaults at each layer. We treat the configuration as an immutable constant that is set before your main code is executed.
3. All configuration sources must be unified. It shouldn’t matter where the option is set from — it is uniformly initialized, verified, and accessed as regular Clojure data.
4. Maximum verification. We don’t want to see FileNotFound stacktraces in the middle of your program run. The whole configuration should be checked early and automatically before the main application is executed. If there are any problems with it, a helpful message should be presented to the user.

Add this line to your list of dependencies:

1. You start by defining a set of supported options. cfg/define takes a map of options to their different parameters. The following small example shows the syntax:

   (require '[omniconf.core :as cfg])
   (cfg/define
     {:hostname {:description "where service is deployed"
                 :type :string
                 :required true}
      :port     {:description "HTTP port"
                 :type :number
                 :default 8080}})
       

   The full list of supported parameters is described here.

2. Populate the configuration from available sources:

   (cfg/populate-from-cmd args) ;; args is a command-line arguments list
   (when-let [conf (cfg/get :conf)]
     (cfg/populate-from-file conf))
   (cfg/populate-from-properties)
   (cfg/populate-from-env)
       

   The order in which to tap the sources is up to you. Perhaps you want to make environment variables overwrite command-line args, or give the highest priority to the config file. In the above example we get the path to the configuration file as --conf CMD argument. For more information, see this.

3. Call verify. It marks the boundary in your system, after which the whole configuration is guaranteed to be complete and correct.

   (cfg/verify :quit-on-error true)
       

   If there is something wrong with the configuration, verify will throw a proper exception. This behavior is appropriate in REPL because you can inspect the error and act upon it. But in the case of a standalone launch you would rather see the exact error rather than the whole stacktrace. So, if :quit-on-error true is passed, the app will exit 1 if verification fails.

   If everything is alright, verify will pretty-print the whole configuration map into the standard output. It is convenient because it gives you one final chance to look at your config values and make sure they are good. :silent true can be passed to verify to prevent it from printing the map.

4. Use get to extract arbitrary value from the configuration.

   (cfg/get :hostname)
       

   For nested values you can pass an address of the value, either as a vector, or like varargs:

   (cfg/get :database :ip)
   (cfg/get [:database :ip])
       

   set allows you to change a value. It is definitely not recommended to be used in production code, but may be convenient during development:

   (cfg/set :database :port 3306)
   (cfg/set [:database :port] 3306)
       

Sample programs that use Omniconf: example-lein and example-boot. There is not much difference in using Omniconf with these build tools, but Boot requires a little hack to achieve parity with Leinigen.

Configuration scheme is a map of option names to maps of their parameters. Option name is a keyword that denotes how the option is retrieved inside the program, and how it maps to configuration sources. Naming rules are the following:

For command-line arguments:

:some-option    =>    --some-option

For environment variables:

:some-option    =>    SOME_OPTION

For Java properties:

:some-option    =>    some-option   (java -Dsome-option=... if set from command line)

Each option can have the following parameters:

• :description — string that describes this option. This description will be used to generate a help message for the program.
• :type — currently the following types are supported: :string, :keyword, :number, :boolean, :edn, :file, :directory. Setting a type automatically defines how to parse a value for this option from a string, and also verifies that the resulting value has the correct Clojure type.

  Boolean types have some special treatment. When setting them from the command line, one can omit the value completely.

  (cfg/define {:foo {:type :boolean}, :bar {:type :boolean}})
  ...
  $ my-app --foo --bar    # Confmap is {:foo true, :baz true}
      

  A string parser for booleans treats strings “0” and “false” as false, anything else as true.

• :parser — a single-arg function that converts a string value (given in command-line option or environment variable) into a Clojure value. This option can be used instead of :type if you need a custom option type.
• :default — the option will be initialized with this value. The default value must be specified as a Clojure datatype, not as a string yet to be parsed.
• :required — if true, the value for this option must be provided, otherwise verify will fail. The value of this parameter can also be a nullary function: if the function returns true then the option value must be provided. It is convenient if the necessity of an option depends on the values of some other options. Example:

  (cfg/define {:storage   {:one-of [:file :s3]}
               :s3-bucket {:requried #(= (cfg/get :storage) :s3)}})
      

• :one-of — a sequence of values that an option is allowed to take. If the value isn’t present in the :one-of list, verify will fail. :one-of automatically implies :required true unless you add nil as a permitted value.
• :verifier — a function of [option-name value] that should throw an exception if the value is not correct. Verifier is only executed if the value is not nil, so it doesn’t imply :required true. Predefined verifiers:
  • cfg/verify-file-exists
  • cfg/verify-directory-non-empty — checks if the value is a directory, and if it is non-empty.
• :delayed-transform — a function of option value that will be called not immediately, but the first time when the option is accessed in the code. Transform will be applied only once, and after that the option will store the transformed value. Usefulness of this feature is yet in question. You can mimic it by using a custom parser that wraps the value in a delay, the only difference that you will also have to call force on it every time.
• :nested — a map that has the same structure as the top-level configuration scheme. Nested options have the same rights as top-level ones: they can have parsers, verifiers, defaults, etc. Example:

  (cfg/define
    {:statsd {:nested {:host {:type :string
                              :required true
                              :description "IP address of the StatsD server"}
                       :port {:type :number
                              :default 8125}}}})
      

  CLI and ENV arguments have special transformation rules for nested options — dot as a separator for CLI arguments and Java properties, and double underscore for ENV.

  [:statsd :host]    =>    --statsd.host   (cmdline args)
  [:statsd :host]    =>    -Dstatsd.host   (properties)
  [:statsd :host]    =>    STATSD__HOST    (env variables)
      

  In the program you can use cfg/get to fetch a concrete value, or a whole map at any level:

  (cfg/get :statsd :port) ;=> 8125
  (cfg/get :statsd) ;=> {:host "127.0.0.1", :port 8125}
      

• :secret — if true, the value of this option won’t be printed out by cfg/verify. You will see <SECRET> instead. Useful for passwords, API keys and such.

There are a few. First of all, Omniconf is much more complex and intertwined than, say, Environ. This might put off some developers, although we suspect they are re-implementing half of Omniconf functionality on top of Environ anyway (like we did before).

Omniconf is not suited for dynamic configuration. If you need options to be changed during runtime, values coming from some external dynamic sources, you are better off using a proper solution for that, e.g. Zookeeper together with some wrapper library.

Omniconf configuration map is a global mutable singleton. It is OK if you use Omniconf like we suggest to — populate the values before any application code is executed, and then never change them again — but there might be usecases where this approach does not fit.

Omniconf is an application-level tool. You most likely don’t want to make your library depend on it, forcing the library users to configure through Omniconf too.

In the end we distribute and deploy our applications as uberjars. As a standalone JAR our program doesn’t have access to Leiningen or Boot. Hence, it is better not to offload anything to plugins to avoid spawning differences between development and production time.

:help option gets a special treatment in Omniconf. It can have :help-name and :help-description parameters that will be used when printing the help message. If populate-from-cmd encounters --help on the arguments list, it prints the help message and quits.

with-options works as let for configuration values, i.e. it takes a binding list of symbols that should have the same names as options’ keyword names. Only top-level options are supported, destructuring of nested values is not possible right now.

(cfg/with-options [username password]
  ;; Binds (cfg/get :username) to username, and (cfg/get :password) to password.
  ...)

It proves very useful to run cfg/verify as a part of the build step. If you provide all the options during that step as you do when running the program, then you will be able catch the misconfiguration errors before the app is deployed.

To do this properly you have to provide another entry point into your program that only runs the config definition, population and verification. Look into example projects for inspiration.

EDN files are another source of configuration that Omniconf can use. They must contain a map of options to their values, which will be merged into the config when populate-from-file is called. The values should already have the format the option requires (number, keyword); but you can also use strings so that parser will be called on them.

It is somewhat tricky to tell Omniconf where to look for a configuration file at runtime. One of the solutions is to specify the configuration file in one of the command-line arguments. So you have to populate-from-cmd first, and then to populate from config file if it has been provided. However, this way the configuration file will have the priority over CLI arguments which is not always desirable. As a workaround, you can call populate-from-cmd again, but only if your CLI args are idempotent (i.e. they don’t contain ^:concat).

Sometimes you don’t want to completely overwrite an EDN value, but append to it. For this case two special operations, — ^:concat and ^:merge — can be attached to a map or a list when setting them from any source. Example:

(cfg/define {:emails {:type :edn
                      :default ["[email protected]" "[email protected]"]}
             :roles  {:type :edn
                      :default {"[email protected]" :admin
                                "[email protected]" :admin}}})
...
$ my-app --emails '^:concat ["[email protected]"]' --roles '^:merge {"[email protected]" :user}'

By default, Omniconf prints errors and final configuration map to standard output. But if you have many servers, it may not be very convenient to connect to each to see if all of them are correctly configured. Perhaps you have a Logstash forwarder running on the instance, or some other centralized logging solution. So, you can call cfg/set-logging-fn to make Omniconf use it instead of println. For Timbre 4.3.1 it will be something like this:

(require '[taoensso.timbre :as log])
(cfg/set-logging-fn (fn [& args]
                      (log/-log! log/*config* :info "omniconf.core"
                                 nil nil :p nil (delay (vec args)) nil)))

Note that this will only work if you are able to initialize logging without any data from Omniconf. This is a chicken-and-egg problem that doesn’t have a proper solution, as it is very case-specific.

© Copyright 2016 Grammarly, Inc.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.