Git Product home page Git Product logo

confita's Introduction

Build Status GoDoc Go Report Card

Confita is a library that loads configuration from multiple backends and stores it in a struct.

Supported backends

Install

go get -u github.com/heetch/confita

Usage

Confita scans a struct for config tags and calls all the backends one after another until the key is found. The value is then converted into the type of the field.

Struct layout

Go primitives are supported:

type Config struct {
  Host        string        `config:"host"`
  Port        uint32        `config:"port"`
  Timeout     time.Duration `config:"timeout"`
}

By default, all fields are optional. With the required option, if a key is not found then Confita will return an error.

type Config struct {
  Addr        string        `config:"addr,required"`
  Timeout     time.Duration `config:"timeout"`
}

Nested structs are supported too:

type Config struct {
  Host        string        `config:"host"`
  Port        uint32        `config:"port"`
  Timeout time.Duration     `config:"timeout"`
  Database struct {
    URI string              `config:"database-uri,required"`
  }
}

If a field is a slice, Confita will automatically split the config value by commas and fill the slice with each sub value.

type Config struct {
  Endpoints []string `config:"endpoints"`
}

As a special case, if the field tag is "-", the field is always omitted. This is useful if you want to populate this field on your own.

type Config struct {
  // Field is ignored by this package.
  Field float64 `config:"-"`

  // Confita scans any structure recursively, the "-" value prevents that.
  Client http.Client `config:"-"`
}

Loading configuration

Creating a loader:

loader := confita.NewLoader()

By default, a Confita loader loads all the keys from the environment. A loader can take other configured backends as parameters.

loader := confita.NewLoader(
  env.NewBackend(),
  file.NewBackend("/path/to/config.json"),
  file.NewBackend("/path/to/config.yaml"),
  flags.NewBackend(),
  etcd.NewBackend(etcdClientv3),
  consul.NewBackend(consulClient),
  vault.NewBackend(vaultClient),
)

Loading configuration:

err := loader.Load(context.Background(), &cfg)

Since loading configuration can take time when used with multiple remote backends, context can be used for timeout and cancelation:

ctx, cancel := context.WithTimeout(context.Background(), 5 * time.Second)
defer cancel()
err := loader.Load(ctx, &cfg)

Default values

If a key is not found, Confita won't change the respective struct field. With that in mind, default values can simply be implemented by filling the structure before passing it to Confita.

type Config struct {
  Host        string        `config:"host"`
  Port        uint32        `config:"port"`
  Timeout     time.Duration `config:"timeout"`
  Password    string        `config:"password,required"`
}

// default values
cfg := Config{
  Host: "127.0.0.1",
  Port: "5656",
  Timeout: 5 * time.Second,
}

err := confita.NewLoader().Load(context.Background(), &cfg)

Backend option

By default, Confita queries each backend one after another until a key is found. However, in order to avoid some useless processing the backend option can be specified to describe in which backend this key is expected to be found. This is especially useful when the location of the key is known beforehand.

type Config struct {
  Host        string        `config:"host,backend=env"`
  Port        uint32        `config:"port,required,backend=etcd"`
  Timeout     time.Duration `config:"timeout"`
}

Command line flags

The flags backend allows to load individual configuration keys from the command line. The default values are extracted from the struct fields values.

A short option is also supported.

To update usage message on the command line, provide a description to the given field.

type Config struct {
  Host        string        `config:"host,short=h"`
  Port        uint32        `config:"port,short=p"`
  Timeout     time.Duration `config:"timeout,description=timeout (in seconds) for failure"`
}
./bin -h

Usage of ./bin:
  -host string
       (default "127.0.0.1")
  -h string
       (default "127.0.0.1")
  -port int
       (default 5656)
  -p int
       (default 5656)
  -timeout duration
       timeout (in seconds) for failure (default 10s)

License

The library is released under the MIT license. See LICENSE file.

confita's People

Contributors

asdine avatar conradkurth avatar jhchabran avatar philippgille avatar rogpeppe avatar sidh avatar sixstone-qq avatar skateinmars avatar steamrolla avatar tealeg avatar wojteninho avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

confita's Issues

default value as fallback

Hi , I was looking for a config loading tool and this is exactly the one.
After several apps, we found that config file or other source is not always reliable.
In the case failed to retrievce config files, we have to manually set a default value for crucial fields.
Is it possible to have a default value written in tag?

Very apppreciate for it.

keys are inconsistent with different backends

When using the env backend, then the only keys considered will be those with config: field tags. However, if you're using a file backend, then the entire struct is unmarshaled from the source, and all fields will be considered, even those without a config tag.

So it's possible there are some fields which are configurable only with the file backend and not with the env backend.

I think that probably that any keys that don't specify the config: tag should be ignored so backends are more consistent.

file.Backend should not fail if the file is not found

Currently if you use a file backend and the file is not found, the loader fails with

failed to open file at path "conf/xxx.toml": open conf/xxx.toml: The system cannot find the file specified.

It should either not fail by default, or there should be a way to specify that a file is optional.

Reusing nested config structs

Hi! I am using confita configs as API for internal libraries in my projects. I've tried to use config structs multiple times in single project setup. And I found a problem for my usage. Field tags can't repeat in one config:

type HTTPConfig struct {
	Addr string `config:"addr"`
	Port int `config:"port"`
}

type AppConfig struct {
	MetricsHTTPServer HTTPConfig
	LoggingHTTPServer HTTPConfig
}

There are not possibilities for defining different options for different servers. Also if we'll try to use flags as backend, we'll get panic (below):

panic: ./cmd/example/example flag redefined: addr

goroutine 1 [running]:
flag.(*FlagSet).Var(0xc0001371a0, 0x551e910, 0xc000614440, 0x4f288db, 0xc, 0x0, 0x0)
	/-S/contrib/go/_std/src/flag/flag.go:871 +0x485
flag.(*FlagSet).StringVar(...)
	/-S/contrib/go/_std/src/flag/flag.go:760
flag.(*FlagSet).String(0xc0001371a0, 0x4f288db, 0xc, 0x0, 0x0, 0x0, 0x0, 0xc000614430)
	/-S/contrib/go/_std/src/flag/flag.go:773 +0xa5
flag.String(...)
	/-S/contrib/go/_std/src/flag/flag.go:780
github.com/heetch/confita/backend/flags.(*Backend).LoadStruct(0x5d76df8, 0x552dea8, 0xc0005d2340, 0xc00025f080, 0x0, 0x0)
	/-S/vendor/github.com/heetch/confita/backend/flags/flags.go:73 +0x548
github.com/heetch/confita.(*Loader).resolve(0xc00025f050, 0x552dea8, 0xc0005d2340, 0xc00025f080, 0x5d30240, 0x199)
	/-S/vendor/github.com/heetch/confita/config.go:171 +0x788
github.com/heetch/confita.(*Loader).Load(0xc00025f050, 0x552dea8, 0xc0005d2340, 0x4f63ac0, 0x5d30240, 0x49, 0xd)
	/-S/vendor/github.com/heetch/confita/config.go:58 +0x20a

It may be solved using something like prefixes. Example usage is following:

type HTTPConfig struct {
	Addr string `config:"addr"`
	Port int `config:"port"`
}

type AppConfig struct {
	MetricsHTTPServer HTTPConfig `config:"prefix=metrics"`
	LoggingHTTPServer HTTPConfig `config:"prefix=logging"`
}

Or more idiomatic if it needs. I think this feature will be useful not only me))

Set path to config file via arguments

Hello! Sometimes I need to do something like this: ./my_app --config configs/stable.yaml --port 7777. It's quite the same as a file backend but with a dynamic path to the file. In this case, the port will be overwritten with the provided value.
I haven't find any simple or nice solution for this.

I think that it will be nice to have a sort of DynamicFile backend with options like key, short, required and so on.

Support "description" tag

Help usage generation would be improved if struct members supported a "description" tag (or a sub-tag of "config"), e.g.

type Configuration struct {
	Port          int    `config:"port" description:"listening port number"`
}

or

type Configuration struct {
	Port          int    `config:"port,description=listening port number"`
}

Can't load config into struct with json tags

I noticed a regression in behaviour. Here is code for reproducing the issue.

package confitajson

import (
	"context"
	"log"
	"testing"

	"github.com/heetch/confita"
	"github.com/heetch/confita/backend/file"
)

func TestLoad(t *testing.T) {
	b := file.NewBackend("config.json")

	var c struct {
		Field string `json:"field"`
	}

	if err := confita.NewLoader(b).Load(context.Background(), &c); err != nil {
		log.Fatalf("failed to load config: %v", err)
	}

	if c.Field != "value" {
		log.Fatal("field value is not updated")
	}
}

Content of config.json file is following:

{
    "field": "value"
}

When running this test on the latest confita release, struct field is not updated. But on version v0.7.0 code works fine.

prime@bee ~/C/confitajson> go get github.com/heetch/[email protected]
go: downloading github.com/heetch/confita v0.7.0
prime@bee ~/C/confitajson> go test .
ok  	github.com/slon/confitajson	0.002s
prime@bee ~/C/confitajson> go get github.com/heetch/confita
go: github.com/heetch/confita upgrade => v0.9.1
prime@bee ~/C/confitajson> go test .
2020/05/27 20:02:34 field value is not updated
FAIL	github.com/slon/confitajson	0.002s
FAIL

I also noticed, that removing these 3 lines, seems to fix this issue for me. https://github.com/heetch/confita/blob/master/config.go#L183-L185

Is that change in behaviour intentional, or is it indeed a regression?

backend/flag: error when config is partial

With confita, it's usual to use Load on a struct which doesn't contain all the possible configuration information. For example, we might require information on just the configuration required by a single component.

Unfortunately this does not work well with the flags backend, because it's not possible to parse command line flags correctly without knowing all the possible flags.

This code demonstrates the issue: https://play.golang.org/p/rxws8K5Noxb

Impossible to get a value from env

This bug is introduced since the last release.

Here how to reproduce:

type Config struct {
	Host string `config:"host,required"`
}

func main() {
	var cfg Config
	l := confita.NewLoader(env.NewBackend(), flags.NewBackend())
	_ = l.Load(context.Background(), &cfg)
}

When trying to execute the code above I have:

$> HOST=localhost go run main.go
required key 'host' for field 'Host' not found
exit status 1

Improve documentation

IMO we should specify in the doc that when a key is found in a backend, we won't continue to look for it in the remaining backends.

So there is an importance on how the backends are sorted inside the slice.

What do you think ?

Add string support

Looking for backend currently implemented I was asking myself about the utility of a new one able to handle strings given as parameter.

For exemple if the configuration is fetched from a remote URL (that returns it as a JSON or YAML etc..) - it could be nice to be able to include the fetched result in the config.

As you already are handling Unmarshaler here : ( https://github.com/heetch/confita/blob/master/config.go#L161 )
it could, for exemple, take the following form :

loader := confita.NewLoader(
  data.NewBackend(json.NewDecoder(content)), 
)

or, if for any reason you want to keep the content :

loader := confita.NewLoader(
  data.NewBackend(content, "json"), 
)

( maybe data is not the most relevant name tbh ๐Ÿ™ˆ )

for the rest it's pretty similar with what you already are doing in the file backend (just replacing the file operations part)

Custom tag

Make the struct tag config customizable per loader

Structured configuration support (path/dots)

Does confita support structured configuration? It would be useful to make configurations more clean/organized/structure, and for configurations of growing/large projects.

I have read the README.md, but there's no example for structure config,... And, I didn't find any TOML/YAML/JSON examples of structured configurations.

I would like to populate the following structure:

type Config struct {
	// notifications configuration
	Notifications struct {
		// turns on user notifications for changes/updates/creations
		Enabled bool
		// from email address of emails
		From string
		// BCC addresses of every notification sent
		BCC []string
	}

	// SMTP server/mail-sending configuration
	SMTP struct {
		// format server:port
		ServerAddr string

		// plain auth credentials
		Username, Password string

		// default email address to sent from
		DefaultFrom string
	}
}

For example, with TOML:

notifications.enabled = true
notifications.from = "[email protected]"
notifications.bcc = "[email protected]"

smtp.serveraddr = "smtp.gmail.com:587"
smtp.username = "[email protected]"
smtp.password = "secret"
smtp.defaultfrom = "[email protected]"

Or, with YAML:

notifications:
  - enabled: true
  - from: "[email protected]"
  - bcc:  "[email protected]"

smtp:
  - serveraddr: "smtp.gmail.com:587"
  - username: "[email protected]"
  - password: "secret"
  - defaultfrom: "[email protected]"

File backend

Example:

confita.NewLoader(
  file.NewBackend("somefile.json"),
  file.NewBackend("some-other-file.yaml"),
)

We could use the right parser based on the extension.

Support short/long flags

Most command-line tools support a short and a long form for flags, e.g. "-p" or "--port".
Currently confita only supports a single flag per member.

Suggestion: add a "short" sub-tag and handle it as a synonym of the default name. Also update help usage generation to reflect this.

type Configuration struct {
	Port          int    `config:"port,short=p"`
}

Specify the backend

In order to avoid some useless processing we can specify for a field in which backend we can find its value.

e.g.

type Config struct {
  Foo        string        `config:"foo",bcknd:"etcd"`
  Bar        string        `config:"bar",bcknd:"consul"`
}

Map backend

A simple map, could be used by library wrappers to prefill some configuration variables during runtime for example.

still maintained?

last commit/pr was brought in coming up on 4 months ago... is this still maintained?

Weird behavior when mixing backends

Some examples of the weird behavior when mixing flag and envvar:

โŒ bar default overwrite bar value from envvar

$ BAR=bar-envvar ./test-confita -foo=foo-flag
2019/09/29 10:24:37 cfg.Foo: 'foo-flag'
2019/09/29 10:24:37 cfg.Bar: 'bar-default'

โœ… foo value from envvar it's overwritten by flag as expected

$ FOO=foo-envvar ./test-confita -foo=foo-flag
2019/09/29 10:29:22 cfg.Foo: 'foo-flag'
2019/09/29 10:29:22 cfg.Bar: 'bar-default'

โŒ foo keeps the value from envvar ignoring the flag

$ BAR=bar-envvar FOO=foo-envvar ./test-confita -foo=foo-flag
2019/09/29 10:25:33 cfg.Foo: 'foo-envvar'
2019/09/29 10:25:33 cfg.Bar: 'bar-envvar'

go code:

package main

import (
	"context"
	"log"

	"github.com/heetch/confita"
	"github.com/heetch/confita/backend/env"
	"github.com/heetch/confita/backend/flags"
)

type Config struct {
	Bar string `config:"bar"`
	Foo string `config:"foo"`
}

func main() {
	cfg := Config{
		Bar: "bar-default",
	}

	loader := confita.NewLoader(
		env.NewBackend(),
		flags.NewBackend(),
	)

	err := loader.Load(context.Background(), &cfg)
	if err != nil {
		panic(err)
	}

	log.Printf("cfg.Foo: '%s'\n", cfg.Foo)
	log.Printf("cfg.Bar: '%s'\n", cfg.Bar)
}

Issue when trying to load a list of struct from a yaml file.

Given this type:

type migrationSettings struct {
	Source      string `config:"source,required"`
	Destination string `config:"destination,required"`
	Partitions  int    `config:"partitions,required"`
}

The following yaml:

- source: topic1
  destination: topic2
  partitions: 3

- source: topic1
  destination: topic2
  partitions: 3

- source: topic1
  destination: topic2
  partitions: 3

The snippet:

var s []migrationSettings

l := confita.NewLoader(file.NewBackend("config.yaml"))
l.Load(context.Background(), &s)

fmt.Println(s)

The output:

[]

parse yaml file failled, I have that key but confita said not found

hi, guys.
I met some trouble with use confita. Anybody can help me?Please.
I have a nest struct:

type Database struct {
	Which 	string  `config:"which"`
	DbDir 	string	`config:"db_dir"`
}

type Config struct {
	Database Database 	`config:"database,required"`
	AppName string 		`config:"app_name,required"`
}

and then,when i use goconvey to test, it failed.
here is my code:

func createTempFile(name, content string) (string, func()) {
	dir, err := ioutil.TempDir("", "testDir")
	So(err, ShouldBeNil)

	path := filepath.Join(dir, name)
	f, err := os.Create(path)
	So(err, ShouldBeNil)

	_, err = fmt.Fprint(f, content)
	So(err, ShouldBeNil)
	So(f.Close(), ShouldBeNil)

	return path, func() {
		So(os.RemoveAll(dir), ShouldBeNil)
	}
}

func TestLoadWithTempFile(t *testing.T) {
	Convey("test Load with temp file", t, func() {
		var cfg Config
		path, cleanUp := createTempFile("app.yaml", `
app_name: blog
database:
  which: sqlite3
  db_dir: /tmp/test.db
`)
		defer cleanUp()

		loader := confita.NewLoader(file.NewBackend(path))
		err := loader.Load(context.Background(), &cfg)

		So(err, ShouldBeNil)
		So(cfg.Database, ShouldNotBeNil)
		So(cfg.AppName, ShouldEqual, "blog")
		So(cfg.Database.Which, ShouldEqual, "sqlite3")
		So(cfg.Database.DbDir, ShouldEqual, "/tmp/test.db")
	})
}

and the test result:

....x.
Failures:

  * /home/lks/go/src/github.com/crazypandas/goBlog/config/config_test.go 
  Line 46:
  Expected: nil
  Actual:   'required key 'app_name' for field 'AppName' not found'


6 total assertions

Is there anything wrong?

Batched loading

Let f be the number of fields in a struct, and b be the number of backends, the worst case scenario makes Confita call the backends f * b times per structure.
If the backends are all remote ones (etcd, vault, etc.) that would mean f * b network calls.
We can improve that by taking advantage of the batch capabilities of some of the backends and do the cascading resolution in memory.

Environment variable specification

The XCU specification says that the environment variable names consist solely of upper-case letters, digits and the "_" (underscore).

Maybe, we should respect this spec and not handle the case if names contains "-" (dash).

Toml support

We could add support for TOML files in the file backend.

Example:

file.NewBackend("config.tml")

When value defined in multiple backend, last one wins

This was previously discussed in #7, but my tests are giving different results, If I define a variable in a file as an env var, the last backend on the list wins

Examples:
l := confita.NewLoader(file.NewBackend(path),env.NewBackend())
in this case the env backend overwrites the values in the file files

n := confita.NewLoader(env.NewBackend(),file.NewBackend(path))
same issue here, the file will overwrite any value previously set as env var

The docs say: "Confita scans a struct for config tags and calls all the backends one after another until the key is found" from my understanding once found it stops and continue looking for the next config key

There is a working example of the issue:

https://play.golang.org/p/c8l9NbRnJOT

Thanks to @rogpeppe for your go playground example on issue 61

Add the possibility to get a single value

Hey !

Using the library for a project I am looking for a way to be able to get a single value from the config and potentially load a different file following the returned value.

I think it should be really interesting to have a way to get this value without loading the whole config on a struct or having to create a struct with just one field to get it.

After taking a quick look at how you implemented the backends I think it should not be that hard to add a method like :

func (l *Loader) Get(name string) interface{}

or maybe something typed with a method for each ? (String, int ...)

WDYT ? I should take a look and try to work on it if this is something you are interested by.

Flag help generation doesn't handle []string type

I have a struct with a field which is a []string but when I call the help flag I have a wrong type and a wrong default value:

type Etcd struct {
    Endpoints []string `config:"etcd-endpoints"`
}

var etcd Etcd
etcd.Endpoints = []string{"127.0.0.1:2379"}

The help output:

$> bin -h
Usage of bin:
  -etcd-endpoints value
    	 (default etcd-endpoints)

run CI on Go 1.12 and 1.13 only

We don't need to support earlier Go versions, and it would be nice to simplify the travis configuration, so let's drop explicit support for versions before 1.12.

required tag not being honoured

I created a config struct like this

type rabbitMQ struct {
	USER     string `config:"user,required"`
	PASSWORD string `config:"password,required"`
	HOST     string `config:"host"`
	PORT     string `config:"port"`
}

type dbConfig struct {
	DIALECT  string `config:"dialect,required"`
	NAME     string `config:"name,required"`
	USER     string `config:"user,required"`
	PASSWORD string `config:"password,required"`
	HOST     string `config:"host"`
	PORT     string `config:"port"`
}

type Config struct {
	DATABASES   map[string]dbConfig `config:"databases,required"`
	RABBITMQ    rabbitMQ            `config:"rabbitmq,required"`
	ENVIRONMENT string              `config:"environment"`
}

confita seems to be ignoring the required tag for all but the password field in the rabbitMQ config
The file I'm reading from is a yaml file

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.