Configuration files: ini, json, yaml or python about aiohttp-devtools HOT 12 CLOSED

I vote on yaml.
Please also think about config validation via tools like trafaret or json-schema.
See https://github.com/tailhook/trafaret_config as an example.

from aiohttp-devtools.

Cerberus is good for stuff like that, and pretty stable.

How would you suggest dealing with environment variables?

Could do something like:

db_password:
  env: DBPASS
  default: foobar

Or just update the config object based on a defined set of defaults?

The more logic like this required, the more #3 becomes required.

from aiohttp-devtools.

A vote for python, primarily for its transparency in environment access. See for example gunicorn's config file format.

Though I have a soft spot for yaml, I think that environment variables should be supported out-of-the-box with this tool without requiring a specific syntax in the configuration file (such as ${VAR}: https://github.com/sseg/heroku-aiohttp-web/blob/master/web/utils/settings.py#L5-L16).

from aiohttp-devtools.

@samuelcolvin Cerberus is not older than aiohttp itself :)
If you need a reaaly stable solution there is collander.
I use trafaret for five years and pretty happy with its very straightforward schema definition DSL.

Maybe we need a pluggable validator -- but I pretty sure that the project requires a config file validation in-the-box. Maybe not in very first version though -- we can live without validation for a while but keep in mind this requirement.

Also please take a look on https://github.com/tailhook/trafaret_config twice.
Yes, it uses trafaret as schema specification -- but I've mentioned trafaret_config for another reason. It extends yaml loader for generating errors with line numbers, e.g.:

bad.yaml:2: smtp.port: value can't be converted to int
bad.yaml:3: smtp.ssl_port: value can't be converted to int
bad.yaml:4: port: value can't be converted to int

That's what we must have. I pretty sure the similar report could be implemented on top of any validator.

from aiohttp-devtools.

@sseg python is the worst available option, sorry.
When you are configuring nginx you don't edit config.c file with recompiling nginx after config changing, right?

from aiohttp-devtools.

python isn't compiled :-)

I've need using django's settings.py as the closest equivalent to a config file on a large project for the last two years and it's worked well.

Also a major advantage of a python config is how simple it would be to understand and extend. That extension could include loading a .yaml file.

I'm happy with yaml but we do need a good solution to for environment variables. What would you suggest?

from aiohttp-devtools.

The logic on the start command I'm building would make it very easy to add options for multiple types of config.

Most likely different people have different preferences and different types of solution will work better with setups.

Lets leave this for now and discuss when we have some concrete options.

from aiohttp-devtools.

python is not validated and schema-less solution.

I've run out of time and need to take a pause.
Sorry, I'll be not available until after tomorrow most likely.

@samuelcolvin please go ahead but don't invest too much time into config file polishing.

from aiohttp-devtools.

yaml config seems to be working well.

from aiohttp-devtools.

Good post @samuelcolvin, I'm also looking for the best config approach in Python. I've been using config.py in our project (Test Automation Harness), here is my observation so far:
Pros:

• simple to implement, easy to be understand by Non-programmers
• can have comments
• less code in your main program to access the values
• support all python data structures

Cons:

• too flexible, need rules to keep it for config only

I'd say for small to medium application, Python config file has it's ground.

from aiohttp-devtools.

After much thought on this exact question I built pydantic to solve this exact problem (as well as some others).

It allows you to use a simple python file to define settings but with advanced validation.

See here for an example of using pydantic for settings management.

from aiohttp-devtools.

to use yaml env configs you can do so

def construct_yaml_env(self, node):
    value = [x.strip() for x in self.construct_scalar(node).split(',')]
    if not (3 > len(value) > 0) or not value[0]:
        raise yaml.YAMLError(
            u'!!env {}, not trafaret '
            u'!!env ENV_VARIABLE, DEFAULT_VAL_OPTIONAL'.format(
                self.construct_scalar(node)))

    if len(value) == 1:
        value = os.environ[value[0]]

    value = os.environ.get(*value)

    return yaml.load(value)

yaml.add_constructor(u'tag:yaml.org,2002:env', construct_yaml_env)

test

_template = """
var1: !!env VAR1, true 
var2: !!env VAR2, False
var3: !!env VAR3, null
var4: !!env VAR4, 456
"""


def test_env_default_type():
    data = yaml.load(_template)

    assert data['var1'] is True
    assert data['var2'] is False
    assert data['var3'] is None
    assert isinstance(data['var4'], int)
    assert data['var5'] == 6


def test_env_type():
    os.environ['VAR1'] = 'false'
    os.environ['VAR2'] = 'True'
    os.environ['VAR3'] = 'null'
    os.environ['VAR4'] = '4.5'

    data = yaml.load(_template)

    assert data['var1'] is False
    assert data['var2'] is True
    assert data['var3'] is None
    assert isinstance(data['var4'], float)

from aiohttp-devtools.