eladrich / pyrallis Goto Github PK

Hi,

When serializing a dataclass, there are cases in which we would like the output to be as concise as possible, and omit config entries if their value is the default one.

What do you think about adding a parameter to the encode/dump functions so that this behavior could be toggled on/off by the user?

Hey,

Thanks for this library, I will try it in my next project.

One problem I often face is configuring which class or function my code should use, e.g. use ResNet50 or VGG as backbone. This often ends up in if-clauses, and in gin-config one could use the @-Syntax in the config file to bind to functions and classes.

Do you think something similar would be possible with this library too, since we can simply type custom classes like nn.Module/tf.Module/Callable/etc.?

This is an awesome project! One small enhancement - it would be really nice to support "configurable" population orders for help text beyond the default (look for docstring above argument, then below, then inline).

A common pattern for many codebases I'm working with ends up looking like:

@dataclass
class Config:
    # fmt: off
    mode: str = "train"                         # Mode in < train | evaluate | visualize >

    # Data / Preprocessing Parameters
    data: Path = Path("data/")                  # Path to data directory with MNIST images / labels
    download: bool = True                       # Whether to download MNIST data (if doesn't already exist)

    # Model Parameters
    hidden_dim: int = 256                       # Hidden Layer Dimensionality for 2-Layer MLP
    # fmt: on


@pyrallis.wrap()
def main(cfg: Config) -> None:
    print(cfg)

Because of style directives (fmt: off) and "factored" configuration arguments, the help text ends up populated as:

usage: main.py [-h] [--config_path str] [--mode str] [--data str] [--download str] [--hidden_dim str]

optional arguments:
  -h, --help         show this help message and exit
  --config_path str  Path for a config file to parse with pyrallis

Config:

  --mode str         fmt: off
  --data str         Data / Preprocessing Parameters
  --download str     Whether to download MNIST data (if doesn't already exist)
  --hidden_dim str   Model Parameters

Notably the mode and data and hidden_dim parameters are all populated incorrectly, when ideally they'd be populated inline (e.g. `mode --> "Mode in < train | evaluate | visualize >").

Happy to PR with a fix if that would be easy! I think we just need to pass an additional argument to the initializer in dataclass_wrapper.py!

Hi, thank you for developing this powerful toolbox. I have a problem when trying to import a package which relies on the pyrallis into my own code. Because I use argparse in my script, pyrallis in that package seems also get my args from the command line, which leads to errors.
I wonder if it is possible to disable the pyrallis to read the args from command line(only get them from config_file)? Thank you!

Hi!

Is there any way to specify that some field is required (see, e.g., click's required option)?

Also, by default one cannot use field from dataclasses with no default value (with every other parameter left also by default) while with pyrallis it is replaced with None value instead. Although, personally I am not having any problem with that (even more, with that behaviour, I can somewhat manually check for required fields via assert some_value is not None for example) but it would be much cleaner if such required options are available.

Hey,

I really like Pyrallis, and I've been using it happily since it came out. I did run into a place where I needed a new feature (subclass registries), that spiraled into a bunch of other issues. Some of it's kinda big.

Are you open to patches/new features at this point or are you doing other things? (No shame, I run out of time/energy/context to work on things too). I might fork the project if you're done with it...

Thanks alot for an amazing package!! Has most of the stuff I've been looking around for in (many) other packages.
Is it possible to replace python built-in dataclass with pydantic dataclass to allow for run-time data validation with pyrallis?
It would be a great feature improvement. I havent found the argparse+pydantic combination anywhere else.

This issue will be used for contributors editing using allcontributors https://allcontributors.org/

Hey there! I've been putting pyrallis to action lately and can't over emphasize how much cleaner it's been to integrate and apply configurations.

That said, there are a couple use cases I'm interested to hear your thoughts on. First being how pyrallis handles typing.Literal. I see in decoding/decode_field that the Literal field will decode to 'Any' and that Literal[arg1, arg2, ..] will decode into arg1. In the case of Literal["constant"] this decodes to "constant", which is not a type.

More succinctly, does the patternLiteral[object] make good sense to use and if so, what might be the way for pyralis to handle this type appropriately?

Traceback (most recent call last):
  File "/anaconda3/envs/spec/lib/python3.9/site-packages/pyrallis/parsers/decoding.py", line 65, in decode_dataclass
    field_value = decode_field(field, raw_value)
  File "/anaconda3/envs/spec/lib/python3.9/site-packages/pyrallis/parsers/decoding.py", line 102, in decode_field
    return decode(field_type, raw_value)
  File "/anaconda3/envs/spec/lib/python3.9/functools.py", line 877, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/anaconda3/envs/spec/lib/python3.9/site-packages/pyrallis/parsers/decoding.py", line 33, in decode
    return get_decoding_fn(t)(raw_value)
  File "/anaconda3/envs/spec/lib/python3.9/site-packages/pyrallis/parsers/decoding.py", line 176, in get_decoding_fn
    raise Exception(f"No decoding function for type {t}, consider using pyrallis.decode.register")
Exception: No decoding function for type typing.Literal['onecycle', 'constant'], consider using pyrallis.decode.register

my code is like this.

@dataclass
class Person:
    name: str = 'John'
    age: int = 18

if __name__ == '__main__':
    import sys
    print(sys.argv)
    #main()
    parser = argparse.ArgumentParser()

    parser.add_argument('--config_path', type=str)
    parser.add_argument('hps', nargs=argparse.REMAINDER)
    args = parser.parse_args()
    print(args)
    cfg = pyrallis.parse(Person, config_path = args.config_path, args = args.hps[1:])
    print(cfg)

when I run python simple_test.py hps --help, it give right help messages.
but when I run python simple_test.py hps -h, it raise an exception.

Is this a bug?

I would like to be able to choose a model config if there are several different models. Example:

# main.py
@dataclass
class Model0Config:
    input_dim: int = 32
    output_dim: int = 32
    hidden_dim: int = 32

@dataclass
class Model1Config:
    input_dim: int = 32
    some_other_arg: str = "test_arg"

class ModelConfigs(Enum):
    model0 = Model0Config
    model1 = Model1Config

@dataclass
class TrainConfig:
    wandb_name: str = "some default name"
    model_config: ModelConfigs = field(default=ModelConfigs.model0)
   

Problem here is than I cannot setup sub-config for model is this way, as this will raise an error:

python main.py --model_config=model1 --model_config.input_dim=128

How would you implement something like this? Perhaps there is some other way? Thanks!

First of all, thanks for this amazing library.

I try to run on Jupyter notebook with a config file, but I get the following error message:

AttributeError: 'str' object has no attribute 'copy'

I use this kind of code: args = pyrallis.load(TrainConfig, "a.yaml")

Do you have any idea of using pyrallis on notebooks?

Hello,

Is there a chance to support different configuration formats, e.g. TOML or JSON? I wonder how deeply yaml is integrated.

Best,
-Justin

PS: Nice project. :)