heetch / confita Goto Github PK

In order to avoid some useless processing we can specify for a field in which backend we can find its value.

e.g.

type Config struct {
  Foo        string        `config:"foo",bcknd:"etcd"`
  Bar        string        `config:"bar",bcknd:"consul"`
}

Hi! I am using confita configs as API for internal libraries in my projects. I've tried to use config structs multiple times in single project setup. And I found a problem for my usage. Field tags can't repeat in one config:

type HTTPConfig struct {
	Addr string `config:"addr"`
	Port int `config:"port"`
}

type AppConfig struct {
	MetricsHTTPServer HTTPConfig
	LoggingHTTPServer HTTPConfig
}

There are not possibilities for defining different options for different servers. Also if we'll try to use flags as backend, we'll get panic (below):

panic: ./cmd/example/example flag redefined: addr

goroutine 1 [running]:
flag.(*FlagSet).Var(0xc0001371a0, 0x551e910, 0xc000614440, 0x4f288db, 0xc, 0x0, 0x0)
	/-S/contrib/go/_std/src/flag/flag.go:871 +0x485
flag.(*FlagSet).StringVar(...)
	/-S/contrib/go/_std/src/flag/flag.go:760
flag.(*FlagSet).String(0xc0001371a0, 0x4f288db, 0xc, 0x0, 0x0, 0x0, 0x0, 0xc000614430)
	/-S/contrib/go/_std/src/flag/flag.go:773 +0xa5
flag.String(...)
	/-S/contrib/go/_std/src/flag/flag.go:780
github.com/heetch/confita/backend/flags.(*Backend).LoadStruct(0x5d76df8, 0x552dea8, 0xc0005d2340, 0xc00025f080, 0x0, 0x0)
	/-S/vendor/github.com/heetch/confita/backend/flags/flags.go:73 +0x548
github.com/heetch/confita.(*Loader).resolve(0xc00025f050, 0x552dea8, 0xc0005d2340, 0xc00025f080, 0x5d30240, 0x199)
	/-S/vendor/github.com/heetch/confita/config.go:171 +0x788
github.com/heetch/confita.(*Loader).Load(0xc00025f050, 0x552dea8, 0xc0005d2340, 0x4f63ac0, 0x5d30240, 0x49, 0xd)
	/-S/vendor/github.com/heetch/confita/config.go:58 +0x20a

It may be solved using something like prefixes. Example usage is following:

type HTTPConfig struct {
	Addr string `config:"addr"`
	Port int `config:"port"`
}

type AppConfig struct {
	MetricsHTTPServer HTTPConfig `config:"prefix=metrics"`
	LoggingHTTPServer HTTPConfig `config:"prefix=logging"`
}

Or more idiomatic if it needs. I think this feature will be useful not only me))

I created a config struct like this

type rabbitMQ struct {
	USER     string `config:"user,required"`
	PASSWORD string `config:"password,required"`
	HOST     string `config:"host"`
	PORT     string `config:"port"`
}

type dbConfig struct {
	DIALECT  string `config:"dialect,required"`
	NAME     string `config:"name,required"`
	USER     string `config:"user,required"`
	PASSWORD string `config:"password,required"`
	HOST     string `config:"host"`
	PORT     string `config:"port"`
}

type Config struct {
	DATABASES   map[string]dbConfig `config:"databases,required"`
	RABBITMQ    rabbitMQ            `config:"rabbitmq,required"`
	ENVIRONMENT string              `config:"environment"`
}

confita seems to be ignoring the required tag for all but the password field in the rabbitMQ config
The file I'm reading from is a yaml file

When using the env backend, then the only keys considered will be those with config: field tags. However, if you're using a file backend, then the entire struct is unmarshaled from the source, and all fields will be considered, even those without a config tag.

So it's possible there are some fields which are configurable only with the file backend and not with the env backend.

I think that probably that any keys that don't specify the config: tag should be ignored so backends are more consistent.

jwt-go has a security vulnerability (CVE-2020-26160). There is no patch available and users of jwt-go are advised to migrate to golang-jwt at version 3.2.1. 👀

Does confita support structured configuration? It would be useful to make configurations more clean/organized/structure, and for configurations of growing/large projects.

I have read the README.md, but there's no example for structure config,... And, I didn't find any TOML/YAML/JSON examples of structured configurations.

I would like to populate the following structure:

type Config struct {
	// notifications configuration
	Notifications struct {
		// turns on user notifications for changes/updates/creations
		Enabled bool
		// from email address of emails
		From string
		// BCC addresses of every notification sent
		BCC []string
	}

	// SMTP server/mail-sending configuration
	SMTP struct {
		// format server:port
		ServerAddr string

		// plain auth credentials
		Username, Password string

		// default email address to sent from
		DefaultFrom string
	}
}

For example, with TOML:

notifications.enabled = true
notifications.from = "[email protected]"
notifications.bcc = "[email protected]"

smtp.serveraddr = "smtp.gmail.com:587"
smtp.username = "[email protected]"
smtp.password = "secret"
smtp.defaultfrom = "[email protected]"

Or, with YAML:

notifications:
  - enabled: true
  - from: "[email protected]"
  - bcc:  "[email protected]"

smtp:
  - serveraddr: "smtp.gmail.com:587"
  - username: "[email protected]"
  - password: "secret"
  - defaultfrom: "[email protected]"

Hello! Sometimes I need to do something like this: ./my_app --config configs/stable.yaml --port 7777. It's quite the same as a file backend but with a dynamic path to the file. In this case, the port will be overwritten with the provided value.
I haven't find any simple or nice solution for this.

I think that it will be nice to have a sort of DynamicFile backend with options like key, short, required and so on.

Given this type:

type migrationSettings struct {
	Source      string `config:"source,required"`
	Destination string `config:"destination,required"`
	Partitions  int    `config:"partitions,required"`
}

The following yaml:

- source: topic1
  destination: topic2
  partitions: 3

- source: topic1
  destination: topic2
  partitions: 3

- source: topic1
  destination: topic2
  partitions: 3

The snippet:

var s []migrationSettings

l := confita.NewLoader(file.NewBackend("config.yaml"))
l.Load(context.Background(), &s)

fmt.Println(s)

The output:

[]

Currently, the TOML support that was added supports simple Key/Value configurations but doesn't handle some of TOML's more complex structures.

We should:

• Define test cases for all TOML features
• Ensure that the TOML backend can pass those cases.

The spec for TOML can be found here: https://github.com/toml-lang/toml

Some examples of the weird behavior when mixing flag and envvar:

❌ bar default overwrite bar value from envvar

$ BAR=bar-envvar ./test-confita -foo=foo-flag
2019/09/29 10:24:37 cfg.Foo: 'foo-flag'
2019/09/29 10:24:37 cfg.Bar: 'bar-default'

✅ foo value from envvar it's overwritten by flag as expected

$ FOO=foo-envvar ./test-confita -foo=foo-flag
2019/09/29 10:29:22 cfg.Foo: 'foo-flag'
2019/09/29 10:29:22 cfg.Bar: 'bar-default'

❌ foo keeps the value from envvar ignoring the flag

$ BAR=bar-envvar FOO=foo-envvar ./test-confita -foo=foo-flag
2019/09/29 10:25:33 cfg.Foo: 'foo-envvar'
2019/09/29 10:25:33 cfg.Bar: 'bar-envvar'

go code:

package main

import (
	"context"
	"log"

	"github.com/heetch/confita"
	"github.com/heetch/confita/backend/env"
	"github.com/heetch/confita/backend/flags"
)

type Config struct {
	Bar string `config:"bar"`
	Foo string `config:"foo"`
}

func main() {
	cfg := Config{
		Bar: "bar-default",
	}

	loader := confita.NewLoader(
		env.NewBackend(),
		flags.NewBackend(),
	)

	err := loader.Load(context.Background(), &cfg)
	if err != nil {
		panic(err)
	}

	log.Printf("cfg.Foo: '%s'\n", cfg.Foo)
	log.Printf("cfg.Bar: '%s'\n", cfg.Bar)
}

IMO we should specify in the doc that when a key is found in a backend, we won't continue to look for it in the remaining backends.

So there is an importance on how the backends are sorted inside the slice.

What do you think ?

We could add support for TOML files in the file backend.

Example:

file.NewBackend("config.tml")

Hi , I was looking for a config loading tool and this is exactly the one.
After several apps, we found that config file or other source is not always reliable.
In the case failed to retrievce config files, we have to manually set a default value for crucial fields.
Is it possible to have a default value written in tag?

Very apppreciate for it.

This was previously discussed in #7, but my tests are giving different results, If I define a variable in a file as an env var, the last backend on the list wins

Examples:
l := confita.NewLoader(file.NewBackend(path),env.NewBackend())
in this case the env backend overwrites the values in the file files

n := confita.NewLoader(env.NewBackend(),file.NewBackend(path))
same issue here, the file will overwrite any value previously set as env var

The docs say: "Confita scans a struct for config tags and calls all the backends one after another until the key is found" from my understanding once found it stops and continue looking for the next config key

There is a working example of the issue:

https://play.golang.org/p/c8l9NbRnJOT

Thanks to @rogpeppe for your go playground example on issue 61

With confita, it's usual to use Load on a struct which doesn't contain all the possible configuration information. For example, we might require information on just the configuration required by a single component.

Unfortunately this does not work well with the flags backend, because it's not possible to parse command line flags correctly without knowing all the possible flags.

This code demonstrates the issue: https://play.golang.org/p/rxws8K5Noxb

last commit/pr was brought in coming up on 4 months ago... is this still maintained?

Let f be the number of fields in a struct, and b be the number of backends, the worst case scenario makes Confita call the backends f * b times per structure.
If the backends are all remote ones (etcd, vault, etc.) that would mean f * b network calls.
We can improve that by taking advantage of the batch capabilities of some of the backends and do the cascading resolution in memory.

I have the exact same issue as #30 when upgrading from v0.8.0 to v0.9.0.

The new feature that allows flags to have descriptions precludes them having a comma in.
We should document that restriction.

Hey !

Using the library for a project I am looking for a way to be able to get a single value from the config and potentially load a different file following the returned value.

I think it should be really interesting to have a way to get this value without loading the whole config on a struct or having to create a struct with just one field to get it.

After taking a quick look at how you implemented the backends I think it should not be that hard to add a method like :

func (l *Loader) Get(name string) interface{}

or maybe something typed with a method for each ? (String, int ...)

WDYT ? I should take a look and try to work on it if this is something you are interested by.

Make the struct tag config customizable per loader

I have a struct with a field which is a []string but when I call the help flag I have a wrong type and a wrong default value:

type Etcd struct {
    Endpoints []string `config:"etcd-endpoints"`
}

var etcd Etcd
etcd.Endpoints = []string{"127.0.0.1:2379"}

The help output:

$> bin -h
Usage of bin:
  -etcd-endpoints value
    	 (default etcd-endpoints)

Example:

confita.NewLoader(
  file.NewBackend("somefile.json"),
  file.NewBackend("some-other-file.yaml"),
)

We could use the right parser based on the extension.

Add go1.11.x on the .travis.yml file.

Most command-line tools support a short and a long form for flags, e.g. "-p" or "--port".
Currently confita only supports a single flag per member.

Suggestion: add a "short" sub-tag and handle it as a synonym of the default name. Also update help usage generation to reflect this.

type Configuration struct {
	Port          int    `config:"port,short=p"`
}

A simple map, could be used by library wrappers to prefill some configuration variables during runtime for example.

Currently if you use a file backend and the file is not found, the loader fails with

failed to open file at path "conf/xxx.toml": open conf/xxx.toml: The system cannot find the file specified.

It should either not fail by default, or there should be a way to specify that a file is optional.

I noticed a regression in behaviour. Here is code for reproducing the issue.

package confitajson

import (
	"context"
	"log"
	"testing"

	"github.com/heetch/confita"
	"github.com/heetch/confita/backend/file"
)

func TestLoad(t *testing.T) {
	b := file.NewBackend("config.json")

	var c struct {
		Field string `json:"field"`
	}

	if err := confita.NewLoader(b).Load(context.Background(), &c); err != nil {
		log.Fatalf("failed to load config: %v", err)
	}

	if c.Field != "value" {
		log.Fatal("field value is not updated")
	}
}

Content of config.json file is following:

{
    "field": "value"
}

When running this test on the latest confita release, struct field is not updated. But on version v0.7.0 code works fine.

prime@bee ~/C/confitajson> go get github.com/heetch/[email protected]
go: downloading github.com/heetch/confita v0.7.0
prime@bee ~/C/confitajson> go test .
ok  	github.com/slon/confitajson	0.002s
prime@bee ~/C/confitajson> go get github.com/heetch/confita
go: github.com/heetch/confita upgrade => v0.9.1
prime@bee ~/C/confitajson> go test .
2020/05/27 20:02:34 field value is not updated
FAIL	github.com/slon/confitajson	0.002s
FAIL

I also noticed, that removing these 3 lines, seems to fix this issue for me. https://github.com/heetch/confita/blob/master/config.go#L183-L185

Is that change in behaviour intentional, or is it indeed a regression?

The XCU specification says that the environment variable names consist solely of upper-case letters, digits and the "_" (underscore).

Maybe, we should respect this spec and not handle the case if names contains "-" (dash).

This bug is introduced since the last release.

Here how to reproduce:

type Config struct {
	Host string `config:"host,required"`
}

func main() {
	var cfg Config
	l := confita.NewLoader(env.NewBackend(), flags.NewBackend())
	_ = l.Load(context.Background(), &cfg)
}

When trying to execute the code above I have:

$> HOST=localhost go run main.go
required key 'host' for field 'Host' not found
exit status 1

Help usage generation would be improved if struct members supported a "description" tag (or a sub-tag of "config"), e.g.

type Configuration struct {
	Port          int    `config:"port" description:"listening port number"`
}

or

type Configuration struct {
	Port          int    `config:"port,description=listening port number"`
}

We should have a backend that uses the flag package to load conf from the command line.

.properties file are a common configuration file format.
Support for this format could be added.

hi, guys.
I met some trouble with use confita. Anybody can help me?Please.
I have a nest struct:

type Database struct {
	Which 	string  `config:"which"`
	DbDir 	string	`config:"db_dir"`
}

type Config struct {
	Database Database 	`config:"database,required"`
	AppName string 		`config:"app_name,required"`
}

and then,when i use goconvey to test, it failed.
here is my code:

func createTempFile(name, content string) (string, func()) {
	dir, err := ioutil.TempDir("", "testDir")
	So(err, ShouldBeNil)

	path := filepath.Join(dir, name)
	f, err := os.Create(path)
	So(err, ShouldBeNil)

	_, err = fmt.Fprint(f, content)
	So(err, ShouldBeNil)
	So(f.Close(), ShouldBeNil)

	return path, func() {
		So(os.RemoveAll(dir), ShouldBeNil)
	}
}

func TestLoadWithTempFile(t *testing.T) {
	Convey("test Load with temp file", t, func() {
		var cfg Config
		path, cleanUp := createTempFile("app.yaml", `
app_name: blog
database:
  which: sqlite3
  db_dir: /tmp/test.db
`)
		defer cleanUp()

		loader := confita.NewLoader(file.NewBackend(path))
		err := loader.Load(context.Background(), &cfg)

		So(err, ShouldBeNil)
		So(cfg.Database, ShouldNotBeNil)
		So(cfg.AppName, ShouldEqual, "blog")
		So(cfg.Database.Which, ShouldEqual, "sqlite3")
		So(cfg.Database.DbDir, ShouldEqual, "/tmp/test.db")
	})
}

and the test result:

....x.
Failures:

  * /home/lks/go/src/github.com/crazypandas/goBlog/config/config_test.go 
  Line 46:
  Expected: nil
  Actual:   'required key 'app_name' for field 'AppName' not found'


6 total assertions

Is there anything wrong?

We don't need to support earlier Go versions, and it would be nice to simplify the travis configuration, so let's drop explicit support for versions before 1.12.

Looking for backend currently implemented I was asking myself about the utility of a new one able to handle strings given as parameter.

For exemple if the configuration is fetched from a remote URL (that returns it as a JSON or YAML etc..) - it could be nice to be able to include the fetched result in the config.

As you already are handling Unmarshaler here : ( https://github.com/heetch/confita/blob/master/config.go#L161 )
it could, for exemple, take the following form :

loader := confita.NewLoader(
  data.NewBackend(json.NewDecoder(content)), 
)

or, if for any reason you want to keep the content :

loader := confita.NewLoader(
  data.NewBackend(content, "json"), 
)

( maybe data is not the most relevant name tbh 🙈 )

for the rest it's pretty similar with what you already are doing in the file backend (just replacing the file operations part)