[FEATURE] Load command line options from config file about typer HOT 15 CLOSED

UPDATE: I made a package for the solution below: maxb2/typer-config.

I found a simple solution to this inspired by phha/click_config_file.
In short, the config option is set with is_eager so that it's callback is called before the rest.
The callback then sets the default_map in the underlying click context based on the config file.
You can still set args and options to override the config file.

# typer_config.py
import typer
import yaml

app = typer.Typer( )

def conf_callback(ctx: typer.Context, param: typer.CallbackParam, value: str):
    if value:
        typer.echo(f"Loading config file: {value}")
        try: 
            with open(value, 'r') as f:    # Load config file
                conf = yaml.safe_load(f)
            ctx.default_map = ctx.default_map or {}   # Initialize the default map
            ctx.default_map.update(conf)   # Merge the config dict into default_map
        except Exception as ex:
            raise typer.BadParameter(str(ex))
    return value

@app.command()
def main(
    arg1: str,
    config: str = typer.Option("", callback=conf_callback, is_eager=True),
    opt1: str = typer.Option(...),
    opt2: str = typer.Option("hello"),
):
    typer.echo(f"{opt1} {opt2} {arg1}")


if __name__ == "__main__":
    app()

With a config file:

# config.yaml
arg1: stuff
opt1: things
opt2: nothing

And invoked with python:

$ python typer_config.py --config config.yml
things nothing stuff

$ python typer_config.py --config config.yml others
things nothing others

$ python typer_config.py --config config.yml --opt1 people
people nothing stuff

from typer.

I'm glad you're liking Typer!

So, this would actually fall in the scope of each specific CLI tool, how to handle it, etc.

But you can easily build something like that.

You could install:

$ pip install pyyaml

Then let's say you have a file dl.py:

import typer
import yaml
from pathlib import Path


def main(config: Path = None, num_layers: int = None):
    config_data = {}
    if config:
        config_data = yaml.load(config.read_text(), Loader=yaml.Loader)
    if num_layers:
        config_data["num_layers"] = num_layers
    typer.echo(f"DL config: {config_data}")

And let's say you also have a config.yml file:

version: "2.4"
num_layers: 5

And let's imagine you are using Typer CLI to get autocompletion for it.

You could use it like:

$ typer ./dl.py run

DL config: {}

$ typer ./dl.py run --config ./config.yml

DL config: {'version': '2.4', 'num_layers': 5}

$ typer ./dl.py run --config ./config.yml --num-layers 2

DL config: {'version': '2.4', 'num_layers': 2}

(I just tried all that) 🎁

from typer.

I think this feature will be very useful. @tiangolo many thanks for this awesome library. But the proposed solution is not convincing to me. Ideally you would like to have just a reverse of what is proposed. IMHO, someone will expect that config file integration will let you use the function arguments directly instead of using the new config dictionary which may be overridden by the given function arguments. Even more, no default values can be given to function arguments because None is used in the proposal to decide whether or not override the cofnig dictionary.

Is it possible to put a wrapper over the main function, so such wrapper will look for config file option, and if found it, it will call main with the values given there?

I was thinking in some kind of "partialization" of the main function, but I'm not sure how typer will react. Something like this (no real code):

import functools
import typer
import yaml
from pathlib import Path


def main(num_layers: int = None):
    typer.echo(f"DL num layers: {num_layers}")

def config_wrapper(main, *args):
    if config options are in command line:
        config = load yaml from given config path
        return functools.partial(main, **config)
    return main

typer.run(config_wrapper(main, '-c', '--config'))

Will something like this work? How far is typer of accepting something like this? Is it just something dumb what I'm proposing?

from typer.

In principle, yes. How you implement it depends on the desired behavior. The only requirement for this work-around is that you can manipulate the ctx.default_map before the main parameters are parsed (essentially rewriting the default values you gave in the source code). That's why you have to use is_eager=True for the --config option. It parses that option before any others.

• You could keep the --config option but it loads from a pyproject.toml by default. Something like:

  if value:
      with open(value, 'r') as f:    # Load config file
          conf = yaml.safe_load(f)
  else:
      conf = load_conf_from_pyproject()
  ctx.default_map = ctx.default_map or {}   # Initialize the default map
  ctx.default_map.update(conf)   # Merge the config dict into default_map

• You could use a callback for the app itself which would run before any command is invoked.

I'd suggest the first option, as it gives the end user some flexibility. Plus, you can hide the option with config: str = typer.Option("", hidden=True) if you don't want to advertise it to the end user. In the second option, it would be really bad if the callback unexpectedly raised an exception because it would block the cli from ever running (including --help).

from typer.

UPDATE: See typer-config=0.5.0 for a fix.

@imagejan That is both intended and not. I certainly want it to do that for --config non-existent.yml but not when --config is not provided. Failing during --help is even worse 🙃, so I'll definitely fix that.

However, you can easily get around this by defining your own config file loader with a conditional:

def new_loader(param_value: str) -> Dict[str, Any]:
    if not param_value:
        # Nothing provided, so return an empty dictionary
        # which is a no-op in the parameter callback function
        return {}
                        
    return yaml_loader(param_value)

new_callback = conf_callback_factory(new_loader)

# etc.

Could you open a new issue in maxb2/typer-config where we can discuss default behaviors?

from typer.

@pakozm I'm pretty sure that should work as Typer doesn't modify the original function. I think the original signature is also kept in the wrapped function, but I actually wouldn't be certain.

It's an interesting experiment for sure 😄 🧪 🥽

from typer.

Assuming the original issue was solved, it will be automatically closed now. But feel free to add more comments or create new issues.

from typer.

I am looking for something like https://pypi.org/project/ConfigArgParse/ but that integrates with typer

from typer.

hmm. hydra seems to handle them better https://hydra.cc/docs/tutorial/config_file

from typer.

In short, the config option is set with is_eager so that it's callback is called before the rest.

Can this work-around be used for config files that aren't passed through cli e.g. pyproject.toml?

from typer.

In the second option, it would be really bad if the callback unexpectedly raised an exception because it would block the cli from ever running (including --help).

That is a problem. However qa tools should load their config from pyproject.toml by default without the need to specify a flag. At the same time I'd like to have the option to specify the --config flag to force a custom config file without loading pyproject.toml. Since I am still trying to decide whether to use typer, can tell me whether that's possible?

from typer.

At the same time I'd like to have the option to specify the --config flag to force a custom config file without loading pyproject.toml.

That's exactly what the first approach would do. If --config FILE is provided, use that otherwise use pyproject.toml.

from typer.

@real-yfprojects I put this workaround in an easy-to-use package: maxb2/typer-config. Right now it only supports the first approach that I presented, but I'll be adding the second soon.

from typer.

Thanks! Looking good 💯

from typer.

Thanks @maxb2 for providing typer-config.

When I try to run an app containing a config option without providing --config (or just with --help), I get:

 Invalid value for '--config': [Errno 2] No such file or directory: ''

Is this intended? Is there a way to make the --config option optional? (I can open an issue in typer-config if it's not me doing something wrong here.)

from typer.