This is more of a question... possibly pointing to an issue relating to a documentation gap (which I could do a PR for).

When unit testing my project, I configure environ-config using a dictionary rather than os.environ, similar to this:

environ.to_config(Configuration, {"SOME_VAR": "FOO"})

I got a little stuck after I added SecretsManagerSecrets, as it seems to always attempt to read from AWS Secrets Manager with boto3. At least, that's what I experienced with my particular setup.

I attempted to use moto and pass that client to SecretsManagerSecrets. However, I wasn't able to get that to work... it's possible my issues were due to me not using the library the way it is supposed to be used.... my rough setup is like this:

from environ.secrets.awssm import SecretsManagerSecrets

sm: SecretsManagerSecrets = SecretsManagerSecrets()

@environ.config(prefix='')
class Configuration:
    client_id = environ.var()
    client_secret = sm.secret()

In the course of trying to get moto in there, I did some work to initialize sm in my code's "bootstrap" methods, and I also tried patching.

But at the end of the day and after a bourbon, it seemed like it would be easier to simply set a default value:

client_secret = sm.secret('unit_test_default')

So... I guess I have two questions

• Is there any high-level guidance for how SecretsManagerSecrets is supposed to be used? i.e. is my code reasonable :)
• If I'm on the right track, how is SecretsManagerSecrets intended to work with unit testing?

Seeing your presentation from PyCon 2018 (thanks a lot for that), I jumped nto your library to test environ_config. It works and helps.

Anyway, I got confused a bit and will try to describe that and propose alternative approach.

What confused me

I refer to the example in README.

First I expected, that instance of AppConfig class will already be populated config object, but I found, that e.g. env property is a Raise instance.

Then I understood, environ.to_config shall be called. At the end, the directly instanciated AppConfig is not what one expects.

Proposal 1: Full initialization using os_environ

What about:

cfg = AppConfig()

or with explicit os_environ:

os_environ={
 "APP_ENV": "dev",
 "APP_DB_HOST": "localhost",
 "LANG": "C",
 "APP_AWESOME": "yes",
 "SECRET_YOUR_APP_DB_PASSWORD": "s3kr3t",
}  
cfg = AppConfig(os_environ=os_environ)

pros:

• looks very simple and intuitive
  cons:
• implementation may become complex (att values may come from two sources: environ and specific argument directly)

Proposal 2: Add class method from_environ

It would work as follows:

cfg = AppConfig.from_environ()

or using explicit os_environ value:

os_environ={
 "APP_ENV": "dev",
 "APP_DB_HOST": "localhost",
 "LANG": "C",
 "APP_AWESOME": "yes",
 "SECRET_YOUR_APP_DB_PASSWORD": "s3kr3t",
}  
cfg = AppConfig.from_environ(os_environ)

One advantage would be, direct instantiation with explicit arguments could work as expected:

cfg = AppConfig(env="dev", awesome=True, db=AppConfig.DB(host="localhost", password="s3kr3t"))

The from_environ could get kwargs argument to allow mixed creation:

cfg = AppConfig.from_environ(os_environ=os_environ, kwargs=dict(env="dev", awesome=True))

pros:

• easy to create AppConfig instance without environmental variables
• implementation would be simpler
  cons:
• direct instantiation could surprise (raising Exception on missing arguments).

Expected modifications would be:

• add class method (could be very similar to existing environ.to_config
• modify handling of arguments without defaults not to return Raise instance, but really raise an exception.

Conclusions

First, I was in favour of Proposal 1, but the Proposal 2 seems simpler to implement and in a way looks cleaner. What do you think?

Hi, I am using the latest version but when I do help_str = environ.generate_help(AppConfig) where

@environ.config(prefix="")
class Parent:
      example1= environ.var(default="example 1")
      example2 = environ.var(default="example2")

I getting:

_EXAMPLE_1
_EXAMPLE_2

It just happend when showing vars, because when setting EXAMPLE_1 and EXAMPLE_2, are right.

In PR #7 and one previous one I have run into an issue, that isort pre-commit check passed locally (using tox -e ling) and failed on Travis or vice versa.

As @hynek noted, the cause was local isort configuration. In my own case, there were .editorconfig and .isort files, which participated on the problem. After the files were renamed, the issue was gone.

It seems like isort (and possibly black?) could be affected by higher level configuration files.

One option to try is to specify complete isort configuration on project level. This way higher level configs would have no chance to influence the check.

Use case

I'm using environ-config to conveniently maintain configuration data and display config options in my docs. The software I'm writing is basically a scientific analysis library, meant to be used in an interactive console such as a Jupyter notebook. In this workflow, it may be desirable to modify the configuration during execution.

The issue

I'd like converters to be executed when setting variables. Is there a built-in option similar to attrs' on_setattr to do that? Or should I work around it?

Hi,

For some reason I just found out this library and loved it. It allows me to setup attr.s settings classes in way more better fashion than before!

However I think one of most important config characteristics is immutability, while by default @environ.config creates mutable attr.s class,

>>> import environ
>>> @environ.config(prefix="")
... class Settings:
...     LEVEL: str = environ.var()
... 
>>> settings = environ.to_config(Settings)
>>> settings.LEVEL
'dev'
>>> settings.LEVEL = 'prod'
>>> settings
Settings(LEVEL='prod')

With that in mind I believe there should be a simple way of passing frozen=True to attr.s call, which resulted in creation of immutable config instance.

If you're ok with the idea, I can provide the PR to satisfy that and allow @environ.config be immutable.

Thanks again!

Hi,

I think there might be a bug in the from_path() constructor of DirectorySecrets class.

Code to reproduce

from environ import secrets
import environ
dir_secrets = secrets.DirectorySecrets.from_path("/run/secrets")

@environ.config(prefix="")
class TestSettings:
    dummy = dir_secrets.secret()

a = TestSettings.from_environ()

Fails with an exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/venv/lib/python3.10/site-packages/environ/_environ_config.py", line 93, in from_environ_fnc
    return __to_config(cls, environ)
  File "/opt/venv/lib/python3.10/site-packages/environ/_environ_config.py", line 307, in to_config
    return _to_config_recurse(config_cls, environ, app_prefix)
  File "/opt/venv/lib/python3.10/site-packages/environ/_environ_config.py", line 268, in _to_config_recurse
    got[name] = getter(environ, attr_obj.metadata, prefixes, name)
  File "/opt/venv/lib/python3.10/site-packages/environ/secrets/__init__.py", line 189, in _get
    secrets_dir = environ.get(self._env_name, self.secrets_dir)
  File "/usr/local/lib/python3.10/_collections_abc.py", line 819, in get
    return self[key]
  File "/usr/local/lib/python3.10/os.py", line 676, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/local/lib/python3.10/os.py", line 756, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType

I think it has to do with the fact that _env_name attribute is used to get default secret_dir value. _env_name is None by default, hence the error. Code in question.

def _get(self, environ, metadata, prefix, name):
        ce = metadata[CNF_KEY]
        # conventions for file naming might be different
        # than for environment variables, so we don't call .upper()
        filename = ce.name or "_".join(prefix[1:] + (name,))

        secrets_dir = environ.get(self._env_name, self.secrets_dir) < ---- failing line
        secret_path = os.path.join(secrets_dir, filename)
        log.debug("looking for secret in file '%s'.", secret_path)

        try:
            with _open_file(secret_path) as f:
                val = f.read()
            return _SecretStr(val)
        except FileOpenError:
            return _get_default_secret(filename, ce.default)

Workaround

I was able to overcome the issue using from_path_in_env() with a default value:

dir_secrets = secrets.DirectorySecrets.from_path_in_env(
    env_name="dummy", default="/run/secrets"
)

Hi

Just pip installed this library and tried it out with your test example inside PyCharm CE 2023 on OSX.

Question 1: Where are the settings saved?

import environ
import os

# Extracts secrets from Vault-via-envconsul: 'secret/your-app':
vault = environ.secrets.VaultEnvSecrets(vault_prefix="SECRET_YOUR_APP")


@environ.config(prefix="APP")
class AppConfig:
    @environ.config
    class DB:
        name = environ.var("default_db")
        host = environ.var("default.host")
        port = environ.var(5432, converter=int)  # Use attrs's converters and validators!
        user = environ.var("default_user")
        password = vault.secret()

    env = environ.var()
    lang = environ.var(name="LANG")  # It's possible to overwrite the names of variables.
    db = environ.group(DB)
    awesome = environ.bool_var()


cfg = environ.to_config(
    AppConfig,
    environ={
        "APP_ENV": "dev",
        "APP_DB_HOST": "localhost",
        "LANG": "C",
        "APP_AWESOME": "yes",  # true and 1 work too, everything else is False
        # Vault-via-envconsul-style var name:
        "SECRET_YOUR_APP_DB_PASSWORD": "s3kr3t",
    })  # Uses os.environ by default.

print(cfg)

print(cfg.db.password)

for key, value in os.environ.items():
    if key[:4] == "APP_":
        print(f'{key}: {value}')

The first I noticed is a warning from PyCharm:
Package containing module 'environ' is not listed in the project requirements

the execution result looks like:

AppConfig(env='dev', lang='C', db=AppConfig.DB(name='default_db', host='localhost', port=5432, user='default_user', password=<SECRET>), awesome=True)
s3kr3t

Process finished with exit code 0

Anyhow, as you can see at the end of the script, I loop over os.environment items for those starting with APP_.
However, there are none.

Even if getting all os.environ items and search manually for any of those items, there are none.

Question 2: How are such vars saved at all ?

In case I replace the whole cfg = environ.to_config(...) block with cfg = environ.var(), then I get as execution result:

_CountingAttr(counter=48, _default=Raise(), repr=True, eq=True, order=True, hash=None, init=True, on_setattr=None, alias=None, metadata={'environ_config': _ConfigEntry(name=None, default=Raise(), sub_cls=None, callback=None, help=None)})
Traceback (most recent call last):
  File ".../test2.py", line 39, in <module>
    print(cfg.db.password)
          ^^^^^^
AttributeError: '_CountingAttr' object has no attribute 'db'

Process finished with exit code 1

Or did I misunderstand the whole thing ?

thanks and regards
Wolfgang

Hi,

I've noticed that django-environ uses the same naming for their package:

import environ
...

Perhaps you might want to avoid such collision. Just documenting this.

Hey there, thanks for this nice lib.

I had some issues to get some log output. The doc says:

The other option is to activate debug-level logging for the environ_config logger by setting its level to logging.WARNING.

It took me some time to understand that I need to set logging.getLogger('environ_config').setLevel('DEBUG'), maybe this would be a nice addition to the docs.

Furthermore it says, that the log level should be WARNING, however the messages are logged as DEBUG.

Finally, it would be nice to not only log that the variables are looked up, but also whether they are found and what their value is.

Cheers!

Currently, one can print help calling `environ.generate_help(AppConfig).

Is it worth to extend PR #2 and beside AppConfig.from_environ provide also AppConfig.generate_help?

Hello,
This isn't really a code issue, but I've been installing the library via a Git commit that addressed #24, and was hoping to start referencing PyPI.

Thanks,
Jason

Hi,
thanks for maintain this project, nice work!

I'm having next exception using environ with in a python 3.10.12 environment:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/soto/Source/PROJECT/src/apps/api/PROJECT_api/settings/__init__.py", line 1, in <module>
    import environ
  File "/home/soto/.virtualenvs/PROJECT-api/lib/python3.10/site-packages/environ.py", line 114
    raise ValueError, "No frame marked with %s." % fname
                    ^
SyntaxError: invalid syntax

I've seen in the pyproject.toml that this module requires-python = ">=3.7",
Any idea why this exception is raised?

What do you think about adding a method to generate a JSON string from a configuration class?

I created a method for environ.config classes in a couple of recent projects to take their help strings and convert it to JSON. My teammates and I loved the convenience of knowing for sure we had all configuration values covered.

We found it most helpful when adding someone to the team or preparing to deploy to a new environment. We could run AppConfig.generate_json(), pipe the output into a file, and edit the file with the needed configuration values.

This was inspired by similar functionality in goodconf.

I have created a nested config structure:

@environ.config(prefix="", frozen=True)
class Config:

    @environ.config(frozen=True)
    class MongoDB:
        mongo_connection_uri: str = environ.var(help="Your MongoDB connection URI")
    ...

    db = environ.group(MongoDB)
    ...

However, accessing a nested group member like this:
Config.db.mongo_connection_uri

Produces following error:

 AttributeError: 'member_descriptor' object has no attribute 'mongo_connection_uri'

I could not find an example of accessing a member of a group in the docs, so I created this issue. How should I access a member of a config group?

Thank you.

Hi,

After adding environ-config to my projects with strict mypy config, I needed to provide numerous # type: ignore comments as environ is not typed yet.

I would like to provide type annotations for environ-config, but need to know several moments,

1. Should I provide annotations in Python files or via *.pyi files as done in attrs?
2. I’ve seen that in attrs typing annotations checked by running mypy over src/attr/*.pyi files and over tests/typing_example.py file. I understand necessity of providing similar typing_example.py file for environ-config as well, but not sure in its content. Can you provide suggestions on what needed to be “tested” there?
3. I’ve seen that tox.ini contains unpinned mypy. Would it be better to run typing checks agains pinned mypy version instead? What about adding mypy pre-commit hook instead of tox env or alongside?

Any other suggestions on a topic are welcome

Hi!

I built an api with fastpi, and I am using this module to set some variables.

My issue is that I am trying to pytest it by setting other values for the same environment variables, but the environ-config only read once per pytest execution.

For example, we have different places to test the API, e.g: APP_PLACE=EU, APP_PLACE=Americas etc.

So we can use os.environ to set pytest and environ-config to get the new value for that environment variable to check.. but the environ-config only reads once per pytest execution.

What do I need to do to fix that?

PS: bellow a "hello world" app as an example of what I need to test.

• pytest conftest.py:

# conftest.py
from fastapi.testclient import TestClient
import os
import pytest

@pytest.fixture()
def client():
    from main import app

    yield TestClient(app)
    

@pytest.fixture()
def client_env():
    os.environ['APP_PLACE'] = 'EU'
    from main import app as app_env
    
    yield TestClient(app_env)

• pytest test case hello_test.py:

# hello_test.py
from fastapi.testclient import TestClient
import os

class TestHello:
    
    def test_get_main_index(self, client: TestClient):
        response = client.get("/")
        assert response.status_code == 200
        assert response.json() == {'Hello': 'World'}


    def test_get_main_env_world_placeed(self, client_env: TestClient):
        response = client_env.get("/My World place")
        
        assert response.status_code == 200
        place = os.environ['APP_PLACE']
        assert response.json() == {'Hello': 'My World place', "place": place}

• config.py:

# config.py
from environ import config, to_config, var

@config
class Config:
    PLACE: str = var(default=None)
    
CONFIG: Config = to_config(Config)

• main.py:

# main.py
from fastapi import FastAPI

from app.config import CONFIG

app = FastAPI()

@app.get("/")
def hello():
    return {'Hello': 'World'}


@app.get("/{env_world}")
def hello_env_world(env_world: str):
    place = CONFIG.PLACE
    result = {"Hello": env_world} if not place else {"Hello": env_world, "place": place}
    return result

You can also get or see this code on my repo: https://github.com/adrianovieira/hello-pytest-env

With (probably) 683b129 typehints for the public apis where introduced (which i really like).

However it seems like that these typehints produce issues if you use environ.config().

Using the following example (which works technically fine) we see issues when validating it with mypy.

import environ

@environ.config()
class TestConfig:
    test_var = environ.var()

print(environ.to_config(TestConfig))

mypy outputs:

$ mypy test.py 
test.py:4: error: <nothing> not callable  [misc]
Found 1 error in 1 file (checked 1 source file)

This can also be reproduced allthought with a slightly different error:

import environ

class TestConfig:
    test_var = environ.var()

Test2 = environ.config(TestConfig, prefix="a")

print(environ.to_config(Test2))

$ mypy test.py
test.py:8: error: Argument 1 to "to_config" has incompatible type "TestConfig"; expected "Type[<nothing>]"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

Python2 support officially ended, this project could support Python 3.6+ only.

If you decide to drop Python2, I'm happy to do it!

I would personally just remove everything Python 2 related and just bump major version, but let me know how do you want to approach this.