uber-go / config Goto Github PK
View Code? Open in Web Editor NEWConfiguration for Go applications
Home Page: https://godoc.org/go.uber.org/config
License: MIT License
Configuration for Go applications
Home Page: https://godoc.org/go.uber.org/config
License: MIT License
Hello, it would be nice if the error for when there is a missing variable expansion states that it is missing instead of "empty default". Example below.
go version - 1.21.2
uber-go/config version - 1.4.0
sidecar:
global-tags:
- "pod=$MY_POD_NAME"
- "namespace=$MY_NAMESPACE"
main: go
func main() {
cfgProvider, err := config.NewYAML(config.Source(strings.NewReader("sidecar: { log-level: INFO, global-tags: [pod=$POD, namespace=$NAMESPACE]}")), config.Expand(os.LookupEnv))
if err != nil {
zap.L().Fatal("could not read yaml config", zap.Error(err))
}
var cfg AppConfig
if err := cfgProvider.Get("sidecar").Populate(&cfg); err != nil {
zap.L().Fatal("could not read app config", zap.Error(err))
}
}
output:
go run main.go # no exported env vars
{"level":"fatal","message":"could not read yaml config","error":"couldn't expand environment: default is empty for \"POD\" (use \"\" for empty string)"}
It would be clearer if the error was - "missing expansion variable {VAR_NAME}" instead of "default is empty".
Currently, the error messages from merging are not very helpful:
returned a non-nil error: couldn't merge YAML sources: can't merge a scalar into a mapping
It'd be nice to have some information on what key failed.
Hello,
I'm currently upgrading my dependencies to start building a new projet and the latest version of config
lead to the use of 2 vulnerables dependencies.
His the project still maintened ?
At the moment it's func NewYAMLProviderFromFiles(files ...string) (Provider, error)
Couple of things come to mind:
FromFiles
typically implies os.File
. We should consider renaming to FromPaths
, or actually accept an os.File
[]string
or []os.Path
? It would be a lot more flexible if we changed the signature to something like (fils []string/os.File, ...Option)
or even ignore the Option
part all together than can be added later.We should check for correctness using gofmt
, govet
, and golint
as part of Travis builds. We already do this in the rest of the Fx-related OSS projects, so we should be able to just copy those bits of Makefile.
YAML allows for any type for a mapping key so long as the key is unique. An example would be integers as keys:
SF Giants:
roster:
38: Tyler Beede
50: Ty Blach
62: Ray Black
It should be possible to query the above configuration with YAML.Get
, eg baseball.Get("SF Giants.roster.50")
, but it is not as the Get key segments are always interpreted as strings.
We've seen several users be quite confused about the defaults section of environment variable expansion defined here.
The behavior around quote characters is inconsistent and this has caused user confusion.
${NOTSET:} - fails saying no default has been provided.
${NOTSET:""} - uses empty string and strips the quote characters.
${NOTSET:"value"} - returns "value" with the quotes, usually breaking the customer's usage. It's very natural for them to take a "" default and fill in a string inside the quotes.
${NOTSET:value} - returns value as the user probably intended.
Quotes should either always be stripped or never be stripped.
Option 1:
The most backwards compatible change is simply to allow the form ${NOTSET:} but this is more confusing because "" will still really mean empty string while "a" will mean quoted "a".
Option 2:
Strip outer quotes. Potentially a breaking change in the unlikely case people actually want literal quote characters in their defaults. Better long term outcome because behavior is totally consistent and all forms of defaults are supported. If you want quotes, you would escape them like ${NOTSET:"\"quoted\""}
Config populates values only for scalar types, functions and channel, but not for structs, right now it simply skips this check:
func TestHappyZapTextUnMarshallerParsing(t *testing.T) {
t.Parallel()
withYamlBytes([]byte(`level: debug`), func(provider Provider) {
lvl := zap.NewAtomicLevel()
err := provider.Get("level").Populate(&lvl)
assert.NoError(t, err)
assert.Equal(t, zap.DebugLevel, lvl.Level())
})
}
Output:
Error: Not equal: -1 (expected)
!= 0 (actual)
From uber-go/fx#594
GOOS=android GOARCH=arm CGO_ENABLED=0 go build
fails to compile:
./decoder.go:76: constant 9223372036854775807 overflows uintptr
./decoder.go:122: constant 18446744073709551615 overflows uintptr
I have been following this project when it's on https://github.com/uber-go/fx/config
.
Comparing to traditional way to deal with configurations from command line,file,environment or some key-value storage(consul/etcd). I think is an excellent idea to unify these with provider
.
So , I am expecting for completion for this.
But, I notice that there are big differences from the earlier versions, I am confused. Like.
I believe there are more inside-team disscussioins, that guiding this way.
Can some explain what is the design target for the future?
I've noticed that it has been quite a while since the last release, and some security vulnerabilities have been detected in some of the dependencies used in the repository ##125.
Are there any plans to release a new minor version?
See example test:
https://play.golang.org/p/c1LG6FyTBp2
Caller 1: Create and populate a map. Modify the map.
Caller 2: Calls Populate, sees values as modified by caller 1
=== RUN TestProviderMutability
--- FAIL: TestProviderMutability (0.00s)
Error Trace: mutable_test.go:36
Error: Not equal: map[string]interface {}{"app":map[interface {}]interface {}{"foo":"bar"}} (expected)
!= map[string]interface {}{"app":map[interface {}]interface {}{"foo":"baz"}} (actual)
Diff:
--- Expected
+++ Actual
@@ -2,3 +2,3 @@
(string) (len=3) "app": (map[interface {}]interface {}) (len=1) {
- (string) (len=3) "foo": (string) (len=3) "bar"
+ (string) (len=3) "foo": (string) (len=3) "baz"
}
FAIL
exit status 1
base.yaml
service:
name: abc
main.go
package main
import (
"go.uber.org/config"
"fmt"
)
func main() {
type ServiceConfig struct {
Name string
}
type cfg struct {
Name string `yaml:"service.name"`
Service ServiceConfig
}
// provider, err := config.NewYAML(config.File("base.yaml"))
provider, err := config.NewYAMLProviderFromFiles("base.yaml")
if err != nil {
panic(err) // handle error
}
var c cfg
if err := provider.Get("").Populate(&c); err != nil {
panic(err) // handle error
}
fmt.Printf("%+v\n", c)
}
Expected result:
{Name:abc {Service:{Name:abc}}
Actual result:
{Name: {Service:{Name:abc}}
Pinned down to 46dea54 after git bisect
func TestYAMLEmptyToInt(t *testing.T) {
t.Parallel()
provider := NewYAMLProviderFromBytes([]bytes(""))
c := provider.Get("not_there")
val, ok := c.TryAsInt()
assert.False(t, ok, "Invalid int succeeded TryAsInt")
assert.Equal(t, 0, val)
}
The default config provider conflicts badly with cobra:
https://github.com/uber-go/config/blob/master/config.go#L76
There's no way to easily turn off the command line provider as far as I know if you want to use the default fx fileset and env expansion logic.
This was breaking Catalyst's flags and as soon as I stopped using the command line provider Catalyst's flags came back.
Hi Uber Team,
I was just scanning my project and noticed that the https://nvd.nist.gov/vuln/detail/CVE-2020-14040 security issue was flagged. It's a transitive dependency from this config module. Might be worth bumping up the dependency versions.
We attempt to expand environment variable references in comments, which is irritating - constructing a provider fails if any of the referenced variables aren't available. For example, this YAML can't be loaded because $ZONE
isn't available:
# We use $DATACENTER because the more-correct $ZONE isn't available yet.
tag: ${DATACENTER:unknown}
Right now config.Value implements only fmt.Stringer
interface simply returning fmt.Sprint(v.Value())
which is fine for small values, but reading the entire config will be extremely hard.
Value should implement json.Marshaler
interface to be able to print config in a structured way. The corner case here is implementing marshaller for map[interface{}]interface{}
which is the usual returned type used by the yaml library we use, but it is not supported by the standard library.
package main
import (
"fmt"
"go.uber.org/config"
)
func main() {
p, err := config.NewYAMLProviderFromBytes([]byte(`foo: yes`))
if err != nil {
panic(err)
}
var foo string
if err := p.Get("foo").Populate(&foo); err != nil {
panic(err)
}
fmt.Println(foo)
}
This prints true
, not yes
as I'd expect.
Group provider iterates over all underlying providers and returns the first found value.
This is ok for scalars/arrays, but can lose extra keys for maps.
The following test fails, yet it should succeed since NopProvider by definition can not have values.
package config
import (
"testing"
"github.com/stretchr/testify/require"
)
func TestNopProviderHasValue(t *testing.T) {
p := NopProvider{}
val := p.Get("some.obviously.missing.value")
require.False(t, val.HasValue(), "nop provider can't have values")
}
Config API has been pretty stable and we should start the release cycle.
There have been a lot of commits for bug fixes and such, but all of that would fall nicely under ^1
semver.
I don't imagine we will have any changes between v1.0-rc1 and v1.0, but I don't feel 100% confident jumping straight to a full release that will lock the repo in for a long time.
Thoughts?
YAML requires that strings which contain characters that conflict with YAML syntax elements must be quoted in order to be parsed properly. For example, YAML does not allow unquoted strings to begin with the @
character so any string that begins with this character must be quoted. The expandTransformer
, however, does not currently check if a replacement value needs to be quoted and will produce invalid YAML if it's lookup function returns a value that must be quoted. This problem is reproduced in the following program:
package main
import (
"fmt"
"os"
"strings"
"go.uber.org/config"
)
func main() {
environment := map[string]string{"FOO": "@foo"}
lookup := func(key string) (string, bool) {
s, ok := environment[key]
return s, ok
}
yaml := strings.NewReader("key: ${FOO}")
p, err := config.NewYAML(config.Source(yaml), config.Expand(lookup))
if err != nil {
fmt.Printf("Error: Unable to construct YAML provider: %v.\n", err)
os.Exit(1)
}
fmt.Println(p.Get("key").Value())
}
The output of this program is:
Error: Unable to construct YAML provider: couldn't decode merged YAML: yaml: found character that cannot start any token.
The issue is that the YAML template
key: ${FOO}
is converted to the following YAML source
key: @foo
but in order for this source to be valid YAML it should be
key: "@foo"
Populate
does not return an error if the type of the container being decoded is wrong. It does fail if the primitive type being decoded does not match, but not if you try to decode say, a list, where an integer was given.
For example,
package main
import (
"fmt"
"go.uber.org/config"
)
func main() {
cfg := config.NewYAMLProviderFromBytes([]byte(`foo: ["a", "b", "c"]`))
var (
intMap map[int]int
intList []int
stringListList [][]string
)
if err := cfg.Get("foo").Populate(&intMap); err != nil {
fmt.Println("failed to load int map:", err)
} else {
fmt.Printf("got int map: %#v\n", intMap)
}
if err := cfg.Get("foo").Populate(&intList); err != nil {
fmt.Println("failed to load int list:", err)
} else {
fmt.Printf("got int list: %#v\n", intList)
}
if err := cfg.Get("foo").Populate(&stringListList); err != nil {
fmt.Println("failed to load string list list:", err)
} else {
fmt.Printf("got string list list: %#v\n", stringListList)
}
}
Output:
got int map: map[int]int(nil)
failed to load int list: for key "foo.0": strconv.ParseInt: parsing "a": invalid syntax
got string list list: [][]string{[]string(nil), []string(nil), []string(nil)}
As we're shooting towards the v1 release, the number of public methods on the Value type has been called into question.
type Value
// Keep:
func (cv Value) ChildKeys() []string
func (cv Value) HasValue() bool
func (cv Value) IsDefault() bool
func (cv Value) LastUpdated() time.Time
func (cv Value) Populate(target interface{}) error
func (cv Value) Get(key string) Value
func (cv Value) Source() string
func (cv Value) Value() interface{}
func (cv Value) WithDefault(value interface{}) Value
// Decision zone ("helpers"):
func (cv Value) AsBool() bool
func (cv Value) AsFloat() float64
func (cv Value) AsInt() int
func (cv Value) AsString() string
func (cv Value) String() string
func (cv Value) TryAsBool() (bool, bool)
func (cv Value) TryAsFloat() (float64, bool)
func (cv Value) TryAsInt() (int, bool)
func (cv Value) TryAsString() (string, bool)
Godoc: https://godoc.org/go.uber.org/config#Value
Most of these helpers look very similar. They rely heavily on the Populate()
functionality and try to mimic what the users would otherwise do should these methods not be available.
Here are a subset of two:
// TryAsBool attempts to return the configuration value as a bool
func (cv Value) TryAsBool() (bool, bool) {
var res bool
err := newValueProvider(cv.Value()).Get(Root).Populate(&res)
return res, err == nil
}
// TryAsFloat attempts to return the configuration value as a float
func (cv Value) TryAsFloat() (float64, bool) {
var res float64
err := newValueProvider(cv.Value()).Get(Root).Populate(&res)
return res, err == nil
}
Do nothing. As ever this is the first option.
The main problem for me is that we have a strange subset of types that we consider "worthy" of the own method. What about float32
, what about int16
? What about a []int
?
If we go down this road we'd need some sort of a heuristic to determine what warrants a conversion method.
Move these methods into functions in another package. func (cv Value) TryAsInt()
can become func TryAsInt(v Value)
.
For the sake of discussion lets name it package conv
.
This is a very attractive option, as we can have a more comprehensive coverage of available functions and types without diluting the API of Value struct
.
The question here is versioning. Ideally this would be a separate conversion library, or a package here in config that wouldn't be bound by the same semver rules (is that even a thing?)
Just flat out nuke the conversion methods, do not even provide a package conv
, as these methods are easy to re-create on a one-by-one basis in the few places that they are used.
I think this makes the API a less approachable, at least initially
Along the lines of Option B.
Outsource this to an external library, such as https://github.com/spf13/cast
The user experience (which we can try to slim down) would be:
cast.ToInt(cfg.Get("foo.bar").Value())
Open to other ideas as well.
Config used to be a package of the Fx framework and often sets itself up as such in the README. A pass should be taken to make sure config is viewed as a top-level library, rather than part of the framework.
I started such process by removing mentions of Fx, but there is a lot more work to do.
Provider group should report merge errors on construction, not when trying to retrieve a value via Get
method, because we need to define invalid Values then and show how users can work with them.
The difficulty here is to flatten maps, e.g. if we have base.yaml
:
series:
episodes:
s01e01: Pilot
and test.yaml
:
series.episodes:
- Pilot
The provider should fail to merge these files together because base.yaml
has series.episodes
key defined as a map and test.yaml
defines it as a slice. This test should pass:
func TestProviderGroup_ConstructorMergeError(t *testing.T) {
t.Parallel()
f, err := NewStaticProvider(map[string]interface{}{
"series": map[string]interface{}{
"episodes": map[string]string{"s01e01": "Pilot"},
}})
require.NoError(t, err, "Can't create the first provider")
s, err := NewStaticProvider(map[string]interface{}{
"series.episodes": []interface{}{"Pilot"},
})
require.NoError(t, err, "Can't create the second provider")
_, err := NewProviderGroup("test-group", f, s)
assert.Error(t, err)
}
glide.yaml has the wrong package name (still says that it's the top-level fx project), and it should be pinned to the v2
branch of github.com/go-validate/validate
.
Documenting this here - I will dig deeper into finding a solution.
func TestConstantDefaultTimestamp(t *testing.T) {
t.Parallel()
provider := NewProviderGroup(
"test",
NewYAMLProviderFromBytes(),
)
target := &root{}
v := provider.Get(Root)
assert.NoError(t, v.Populate(target))
ts := v.Timestamp
v = provider.Get(Root)
assert.Equal(t, ts, v.Timestamp)
}
Loosely related to #21. Looks like we sometimes do unnecessary allocations for maps and slices. We need to make sure to allocate after we got some values from provider.
// ChildKeys returns the child keys
// TODO(ai) what is this and do we need to keep it?
func (cv Value) ChildKeys() []string {
return nil
}
It currently always returns nil. In order to get all the child keys, there is a work-around using Populate
into a map
package main
import (
"fmt"
"go.uber.org/config"
)
func main() {
p := config.NewYAMLProviderFromBytes([]byte("foo: bar"))
m := map[string]interface{}{}
p.Get("").Populate(&m)
for k := range m {
fmt.Printf("key %q\n", k)
}
}
Need to enable code coverage and I'd like to try out codecov.io to see how it compares with coveralls
func TestInterpolatedBool(t *testing.T) {
f := func(key string) (string, bool) {
if key == "interpolate" {
return "true", true
}
return "", false
}
p := NewYAMLProviderFromReaderWithExpand(f, ioutil.NopCloser(strings.NewReader("val: ${interpolate:false}")))
assert.True(t, p.Get("val").AsBool())
}
It would duplicate a lot if several config structs are sharing the same set of fields, so that support for struct embedding would be very useful.
One solution is to add support for yaml tag "inline", which will make these codes works:
package main
import (
"fmt"
"log"
"go.uber.org/config"
)
type Nested struct {
N string
}
func main() {
p, _ := config.NewStaticProvider(map[string]interface{}{
"example": map[string]interface{}{
"name": "uber",
"founded": 2009,
"n": "nnnn",
},
})
var c struct {
Nested `yaml:",inline"`
Name string
Founded int
}
if err := p.Get("example").Populate(&c); err != nil {
log.Fatal(err)
}
fmt.Println(c)
}
Config code panics a lot when something doesn't go it's way: can't open yaml files, can't marshal, etc.
Before start the release cycle I'd like for all the providers to return (Provider, error)
.
For example, one of the must uses providers: func NewYAMLProviderFromFiles(files ...string) Provider
has multiple ways it can panic.
I get that configuration loading is a pretty crucial process and more often than not, users will chose to panic right away. However, let's not make that choice for them and at least give them the option to do something else.
We're a library, and like dig, we should try very very hard to reserve panics for unexpected conditions, rather than trying to use it as an exception.
I have a project where I am loading large env variables using the os.LookupEnv
function. When doing so I see this error:
couldn't expand environment: transform: short destination buffer
The library would ideally be able to handle arbitrarily large env variables.
The original vision was to allow dynamic providers.
In reality, however, we ended up with the traditional YAGNI world as they are completely unused. Since they literally account for 50% of the API surface of the Provider
and are unused, lets be ruthless and throw them away before it bites us.
Command line provider is using maps to store slices right now, which makes it impossible to use in combination with a group provider and unmarshalers: with roles defined in e.g. base.yaml and overriden later via command line group provider will fail to merge a map and a slice.
In previous versions of config values where HasValue == false were never deserialized (theory, haven't confirmed that). In 1.0.0 every field is getting deserialized, even if it has no corresponding value in the yaml.
This breaks deserializing into a struct like xtchannel.Configuration which contains UnmarshalText
methods that don't accept default values as valid.
Ran into this migrating one of my services to FX.
If we have two configuration files that we want to merge together like:
base.yaml
test:
testmap:
testkey: "test"
and
development.yaml
test:
testmap:
testkey2: "anothervalue"
In our current internal config library, the resulting merged config will be:
test:
testmap:
testkey2: "anothervalue"
the testmap
key of the development.yaml configuration takes precedence and removes the testkey
map key from the base.yaml.
Using go.uber.org/config
the result config is:
test:
testmap:
testkey: "test"
testkey2: "anothervalue"
Which is different than format we've pushed for internally in all go and python services. And will likely cause a large amount of developer confusion as we move people away from the internal config system and towards the fx world. Unless there is a compelling reason to keep the new way it will be advantageous to keep parity with the current internal config merging.
This is a little invoke function I used to test this:
type Config struct {
Test map[string]interface{} `yaml:"test"`
}
func testConfig(cfg config.Provider, logger *zap.Logger) {
var cfgData map[string]interface{}
if err := cfg.Get("test").Populate(&cfgData); err != nil {
logger.Error(err.Error())
return
}
logger.Sugar().Infof("cfg: %v", cfgData)
var xcfg Config
if err := xconfig.Load(&xcfg); err != nil {
logger.Error(err.Error())
return
}
logger.Sugar().Infof("xcfg: %v", xcfg.Test)
}
I'm reading a single config file where I want to override values based on priority. My initial code below worked with 1.1.0, but does not with 1.3.0. I understand that some functions have been marked deprecated in 1.3.0, but the README explicitly states that "no breaking changes will be made in the 1.x series of releases".
I see that the new NewYAML
function can take multiple files with priority, but I want to read and merge a single file. Is there another way to do this in 1.3.0?
Reproduction:
config.yml file:
---
default:
foo: Base
other:
foo: Override
main.go:
package main
import (
"fmt"
"go.uber.org/config"
)
func main() {
var conf struct {
Foo string `yaml:"foo"`
}
provider, err := fromSpecificEnvs("config.yml", "default", "other")
if err != nil {
panic(err)
}
if err := provider.Get(config.Root).Populate(&conf); err != nil {
panic(err)
}
fmt.Printf("Foo: %q\n", conf.Foo)
}
func fromSpecificEnvs(filepath string, envs ...string) (config.Provider, error) {
baseProvider, err := config.NewYAMLProviderFromFiles(filepath)
if err != nil {
return nil, fmt.Errorf("error parsing config file: %v", err)
}
var subProviders []config.Provider
for _, env := range envs {
sp := config.NewScopedProvider(env, baseProvider)
subProviders = append(subProviders, sp)
}
return config.NewProviderGroup("config", subProviders...)
}
Result using 1.1.0:
Foo: "Override"
Result using 1.3.0:
Foo: ""
Fields tagged with both a name and the omitempty
tag, e.g.
type Foo struct {
Name string `yaml:"name,omitempty"`
}
aren't populated.
Default values set using WithDefault
are used even if a YAML provider tries to null out the value using yaml ~
syntax.
Example code:
cfgP, err := config.NewYAMLProviderFromBytes([]byte(`
subkey: ~
`))
if err != nil {
panic(err)
}
type config struct {
Initial int `yaml:"initial"`
Frequency int `yaml:"frequency"`
}
v, err := cfgP.Get("subkey").WithDefault(config{1000, 2000})
if err != nil {
panic(err)
}
var c config
if err := v.Populate(&c); err != nil {
panic(err)
}
fmt.Printf("%+v\n", c)
}
Expected output was to have empty config, but we get:
{Initial:1000 Frequency:2000}
In comparison, a YAML snippet which does something similar:
package main
import (
"fmt"
yaml "gopkg.in/yaml.v2"
)
func main() {
type subkey struct {
Initial int `yaml:"initial"`
Frequency int `yaml:"frequency"`
}
type config struct {
Subkey *subkey `yaml:"subkey"`
}
bs := []byte(`
subkey: ~
`)
c := config{&subkey{1000, 2000}}
if err := yaml.Unmarshal(bs, &c); err != nil {
panic(err)
}
fmt.Println(c.Subkey)
}
Prints out &{0 0}
.
I'm having trouble making the Value
type's WithDefault
work with Populate
. I expect this test to pass, but it doesn't:
func TestConfigDefaults(t *testing.T) {
type cfg struct{ N int }
var c cfg
require.NoError(
t,
config.NewStaticProvider(nil).Get("metrics").WithDefault(cfg{42}).Populate(&c),
"Failed to populate config.",
)
if c.N != 42 {
t.Fatalf("Expected to see default N of 42, got %+v.", c)
}
}
Am I using these APIs wrong, or is this a bug?
/cc @blampe (can't get your username to populate in the assignees drop-down)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.