alunduil / crumbs Goto Github PK

View Code? Open in Web Editor NEW

18.0 18.0 3.0 161 KB

Generalized all-in-one parameters module

License: MIT License

Python 100.00%

crumbs's People

Contributors

Stargazers

Watchers

Forkers

perniferuse b0d0nne11 swarren83

crumbs's Issues

online version of docs would be nice

So people don't have to download and build them just to see if your project is worth downloading in the first place. Or at least some basic usage.

Add ability to re-read configurations

Currently it is not possible to re-read configurations with out possibly using inotify.

log caller with warning about unparsed parameters

Install fails if pyinotify is not present

Steps to reproduce:

Bring up an Ubuntu machine without pyinotify installed.
Run pip install crumbs
Notice an error because logging has been set up incorrectly.

environment prefix = none doesn't parse

AttributeError in del

Exception AttributeError: "'NoneType' object has no attribute 'info'" in <bound method Parameters.del of <crumbs.Parameters object at 0x238e490>> ignored

Looks like we need to check if the logger still exists in the destructor for Parameters.

The previous package for inotify support, pyinotify, is no longer maintained and doesn't work well with recent versions of python. I've dropped support for inotify in the meantime but we should consider adding this back in with the inotify package.

ResourceWarning not defined when `inotify = True`

======================================================================
ERROR: Failure: NameError (global name 'ResourceWarning' is not defined)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib64/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib64/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/alunduil/Projects/savvy/margarine/test_margarine/test_unit/test_spread/test_users.py", line 19, in <module>
    from margarine.spread.users import create_user_consumer
  File "/home/alunduil/Projects/savvy/margarine/margarine/spread/__init__.py", line 9, in <module>
    from margarine.parameters import Parameters
  File "/home/alunduil/Projects/savvy/margarine/margarine/parameters/__init__.py", line 42, in <module>
    PARAMETERS.add_configuration_file(PARAMETERS['margarine.configuration_file_path'])
  File "/usr/lib64/python2.7/site-packages/crumbs/__init__.py", line 227, in add_configuration_file
    warnings.warn('could not read {}'.format(file_name), ResourceWarning)
NameError: global name 'ResourceWarning' is not defined

Add support for rereading configuration files with inotify.

It'd be nice if the added configuration files could be watched and reread as they change for the program using crumbs. I think pyinotify fits this bill but have begun research to verify if this is the case.

Parameters in different groups with the same name conflict.

When specifying two different parameters with the same name in different groups, the parameters conflict. For example::

PARAMETERS.add_parameter(
        group = 'pmort',
        options = [ '--configuration-file-path', '-c' ],
        metavar = 'FILE',
        default = os.path.join(CONFIGURATION_DIRECTORY, 'pmort.ini'),
        help = \
                'Location of the configuration file to search for pmort ' \
                'parameters.  Default: %(default)s'
        )

and

PARAMETERS.add_parameter(
        options = [ '--configuration-file-path' ],
        group = 'logging',
        metavar = 'FILE',
        default = os.path.join(CONFIGURATION_DIRECTORY, 'logging.ini'),
        help = \
                'Specifies the file path for the logging configuration file. ' \
        )

Yields the following help message:

usage: execute.py [-h] [-c FILE] [--output-directory DIR]
                  [--cache-directory DIR] [--configuration-file-path FILE]
                  [--active ACTIVE] [--maximum-interval MAXIMUM_INTERVAL]
                  [--directory DIRECTORY]

optional arguments:
  -h, --help            show this help message and exit

pmort:
  -c FILE               Location of the configuration file to search for pmort
                        parameters. Default: /etc/pmort/pmort.ini
  --output-directory DIR, -o DIR
                        Specifies the output directory that the collection
                        output is collected into.
  --cache-directory DIR
                        Specifies the output directory for any learned
                        information. Default:
                        /var/cache/pmort/collectors/execute.py

logging:
  --configuration-file-path FILE
                        Specifies the file path for the logging configuration
                        file.

learner:
  --active ACTIVE       The learner algorithm to utilize for determining
                        inter-run timings. Default linear
  --maximum-interval MAXIMUM_INTERVAL
                        Set the maximum time between collections by
                        execute.py. This specifies the maximum number of
                        seconds between collections. Default 600

collector_execute:
  --directory DIRECTORY
                        Directory in which to search for non-standard
                        collectors. Default: /etc/pmort/collectors.d

Should the parameters instead be prefixed on the command line by their group name if they have one?

Build Broken on python-2.7.7

alunduil@elijah ~/Projects/personal/crumbs $ python2.7 setup.py build
Traceback (most recent call last):
  File "setup.py", line 104, in <module>
    setup(**PARAMS)
  File "/usr/lib64/python2.7/distutils/core.py", line 124, in setup
    dist.parse_config_files()
  File "/usr/lib64/python2.7/distutils/dist.py", line 397, in parse_config_files
    val = parser.get(section,opt)
  File "/usr/lib64/python2.7/ConfigParser.py", line 623, in get
    return self._interpolate(section, option, value, d)
  File "/usr/lib64/python2.7/ConfigParser.py", line 669, in _interpolate
    option, section, rawval, e.args[0])
ConfigParser.InterpolationMissingOptionError: Bad value substitution:
        section: [nosetests]
        option : logging-format
        key    : pathname
        rawval : %%(pathname)s:%%(lineno)d: %%(process)d: %%(levelname)s %%(message)s

Looks like the monkey patch added to work between nose and distutils doesn't work in python-2.7.7. We'll need to investigate how to make it work again or find a better work around for this issue.

Group names with '_' should be '-' in argument.

When creating parameters that have a group including an '_' (i.e. foo_bar) the resulting parameter gets rendered with the underscore in the command line help (i.e. --foo_bar-baz).

This is more of an aesthetic item as it doesn't break anything to have it this way but it would make the command line a little nicer if this was hyphenated instead.

Setting only_known leads to loads of `retrieving values from unparsed Parameters` warnings

When you parse with only_known set, every time we lookup a valid value we get warnings:

retrieving values from unparsed Parameters

The docs explain that parsed is not set when only_known is set. Would it make sense to disable this warning in the case that only_known is set? Or consider the parameters as parsed anyway? I can't find the reason for parsed not being set alongside only_known.

from crumbs import Parameters

parameters = Parameters()
parameters.add_parameter(options=["--foo"])
parameters.parse(only_known=True)
parameters["foo"]

Extract Basename for Environment Variable Keys

Looks like we don't do enough manipulation of the sys.argv[0] component of environment value key names. Let's do the following:

extract basename (everything after the last '/')
convert any '-' to '_'

Allow for configuration parameter to map to a non-prefixed environment variable

Sometimes I may want to re-use existing namespacing, such as TOKEN, etc. Or I just might not care if I conflict with parent namespaces.

It would be nice to be able to disable the ARGV0_ prefix of the environment variables.

Add grouped_parameters property to Parameters

It would be very convenient if the Parameters class had an instance property, grouped_parameters, that allowed one to get a grouped set of parameters (example below). This is merely cosmetic as it can be derived from the parameters property currently.

Example:

{
  'group': {
    GROUPS_PARAMETERS (from parameters)
  },
}

inotify should be an extra require

http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies

docstring for parse(only_known) could use some clarification

Running: python 2.7.5-11 (fedora 20) and crumbs 1.2.0

Current:
Help on method parse in crumbs.Parameters:

crumbs.Parameters.parse = parse(self, only_known=False) unbound crumbs.Parameters method
    Ensure all sources are ready to be queried.

    Arguments
    ---------

    :``only_known``: If True, prepare values for parameters that have been
                 added and do not error or fail when unknown parameters
                 are encountered.

    .. note::
        Once ``parse`` has been called ``Parameters.parsed`` will be True
        and it is inadvisable to add more parameters to the ``Parameters``.

    .. note::
        If ``only_known`` is True, the ``--help`` and ``-h`` options on the
        command line (``sys.argv``) will be ignored during parsing as it is
        unexpected that these parameters would be desired at this stage of
        execution.

My sample code:

#!/usr/bin/env python

from crumbs import Parameters
PARAMETERS = Parameters(conflict_handler = 'resolve', inotify = True)
PARAMETERS.add_parameter(
    options = [ '--uri', '-U' ],
    metavar = 'URI',
    help = 'Path to the signup uri '
)
PARAMETERS.parse(only_known=True)
print PARAMETERS['uri']

if i call that with:

python test.py --help

i get:

/usr/lib/python2.7/site-packages/crumbs/__init__.py:252: RuntimeWarning: retrieving values from unparsed Parameters
  warnings.warn('retrieving values from unparsed Parameters', RuntimeWarning)
  None

if i remove only_known:

PARAMETERS.parse()

I get this:

usage: test.py [-h] [--uri URI]

optional arguments:
  -h, --help         show this help message and exit
  --uri URI, -U URI  Path to the signup uri