A use case cited in this stackoverflow discussion is to define class fields, so that subclasses are restricted when they override them.

Maybe classfield() could be easily done, similarly to field()

ValueError(
"validation function should accept 1, 2, or 3 arguments at least. f(val), f(obj, val) or "
"f(obj, field, val)")

@<field>.validator(...) decorator.

Can be specified at the same time than validators arg, in which case they are combined

The error message and/or failure type is the argument of the decorator: @<field_name>.validator(<error message>)

See also @valid8.as_failure_raiser.

is it actually possible ?

For example see https://stackoverflow.com/questions/47931795/validation-of-read-only-attributes-in-python

As of now the order of arguments in generated inits is ancestor-first, but some users may wish to have the other order

when using make_init or init_fields or inject_fields

In all the above scenarii, having a skip_validation option could be nice to get fast object creation routes.

Originally posted by @smarie in #2 (comment)

Extracted from #5

for example <field>.trace_convert(...) that would not only return the converted value but the conversion details.

From #5

matching converters will be called even if received value is of a valid type, except if a specific option skip_matching_converters_if_valid=True is set

Currently PyCharm has an issue with type hints when the __get__ descriptor returns self for class access. See https://youtrack.jetbrains.com/issue/PY-38151

So this behaviour is currently disabled and replaced with an exception (raise ClassFieldAccessError). We should put it back when it is fixed on pycharm.

For example see https://github.com/smarie/python-pyfields/blob/master/pyfields/core.py#L800

class A(object):
    a = field(default='hello')

class B(A):
    b = field(default='world')

    def a(self):
        pass

    __init__ = make_init()


B(a='h', b='w')

yields TypeError: __init__() got an unexpected keyword argument 'a'

I'm facing an issue while using pyfields, it has to do with the fact that properties are lazily initialized. Some of my tests require doing an equality comparison of two objects.
Let's say I have the following class:

class A:
     field1: str = field(check_type=True, default=None)
     
    def __eq__(self, other):
        return type(self) is type(other) and self.__dict__ == other.__dict__

Now,

a1 = A()
a1.field1

a2 = A()

assert a1 == a2

Here, a1 !=a2 since calling a1.field would set a _field variable inside the instance.

This is not exactly a bug, but can you think of any clean way of handling this with pyfields?

From #5

by default after conversion, validation will happen as always. A validate_after_converting=False argument or converter option would also be needed to change this behaviour.

If I have a Nonable field, setting it to None, or calling a constructor with the field unspecified should not raise a validation error. However, this happens if I have typegaurd installed

Example code to reproduce:

class A:
     f: str = field(check_type=True, default=None, nonable=True)
     @init_fields
     def __init__():
            pass

a = A()

This results in the following traceback:

raceback (most recent call last):
  File "/Users/devashishshankar/Work/ddp_v1_venv/lib/python3.7/site-packages/pyfields/typing_utils.py", line 58, in assert_is_of_type
    check_type(field.qualname, value, typ)
  File "/Users/devashishshankar/Work/ddp_v1_venv/lib/python3.7/site-packages/typeguard/__init__.py", line 544, in check_type
    format(argname, qualified_name(expected_type), qualified_name(value)))
TypeError: type of A.f must be str; got NoneType instead

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<makefun-gen-36>", line 2, in __init__
  File "/Users/devashishshankar/Work/ddp_v1_venv/lib/python3.7/site-packages/pyfields/init_makers.py", line 446, in __init__
    setattr(self, field_name, field_value)
  File "/Users/devashishshankar/Work/ddp_v1_venv/lib/python3.7/site-packages/pyfields/core.py", line 1066, in __set__
    assert_is_of_type(self, value, t)
  File "/Users/devashishshankar/Work/ddp_v1_venv/lib/python3.7/site-packages/pyfields/typing_utils.py", line 63, in assert_is_of_type
    raise new_e
pyfields.typing_utils.FieldTypeError: Invalid value type provided for 'A.f'. Value should be of type <class 'str'>. Instead, received a 'NoneType': None

From #2

At the end of generated init methods, we could optionally enable validation that all mandatory fields have been set. That could be implicit or explicit. A prerequisite is to have a function able to provide this functionality on an object class (getting behind the scenes the list of fields applicable to this class).

This function could also be explicitly used by users, anywhere in their code (in an init method or elsewhere)

Note that this is not as simple as looking at the init arguments

Non-data descriptors seem to work with old-style classes (but cannot be debugged in pycharm as of today), but descriptors with a "set" method don't: we should alert users.

feature 5

user wants fully automatic init creation

5a with decorator

@auto_init('height')
class Wall(object):
    height: int = field(doc="Height of the wall in mm.")
    color: str = field(default='white', doc="Color of the wall.")

pros

• a simple decorator, and "all fields" is easy to declare: decorator without arguments.

cons

• fields are referenced by name: no autocompletion if a particular selection needs to be made.
• decorator = a bit harder to debug and can be frightening

5b with decorator but fields declare if they are in init

@auto_init
class Wall(object):
    height: int = field(doc="Height of the wall in mm.", in_init=True)
    color: str = field(default='white', doc="Color of the wall.", in_init=False)

pros

• more compact decoration

cons

• I do not like the fact that the inclusion in init is a property of the fields. This will cause trouble in inheritance and mixin scenarii.

5c explicit method

class Wall(object):
    height: int = field(doc="Height of the wall in mm.")
    color: str = field(default='white', doc="Color of the wall.")
    __init__ = make_init(height, color)

pros

• compact
• autocompletion works if an explicit list or order needs to be provided
• this can probably be smoothly extended to support scenario 4 as well (a __post_init__ method or even any method using make_init(..., post=<method_post>) or make_init(color, my_post_init_method, height) so as to control precisely which part of the signature will go where)

I do not see much cons here

Originally posted by @smarie in #2 (comment)

We should probably consider using __new__ on the descriptors as explained here https://stackoverflow.com/a/41523491.

A few requirements/ideas:

• both as argument and as decorator. both can be specified. Decorator is @<field_name>.converter(*types) will be handled in #28
• if argument: a possibly ordered dictionary of type: converter or (*types): converter.
  - For old python support, a list of key/val tuples is supported.
  - '*': <> can be used to indicate "the rest"
  - None should be used for the none type
  - Is(x) for constant-only converters
• if argument: a single converter can be passed, in which case it will be considered '*'
• if several converters match, they will be tried in order until no exception is raised / no CONVERSION_FAILED is returned (returning None is considered a successful conversion)
• matching converters will be called even if received value is of a valid type, except if a specific option skip_matching_converters_if_valid=True is set now in #30
• by default after conversion, validation will happen as always. A validate_after_converting=False argument or converter option would also be needed to change this behaviour. now in #29

that allows a factory to refer to another field value, for example "if this field is not provided, I copy this other field"

This is probably not a good idea in terms of perfs.

For example autoclass, see smarie/python-autoclass#28

We should probably use a creation_counter on fields like django did:

https://github.com/django/django/blob/129583a0d3cf69b08d058cd751d777588801b7ad/django/db/models/fields/__init__.py#L96

https://github.com/django/django/blob/f3c43ad1fd9556f0fd026a5dfa93c67a5cf186ca/django/forms/forms.py#L38

For example PositiveInt. or PositiveInt | Is('default')

See checktypes library maybe ?

If I have a field with type_hint being something like List[A], and check_type=True, I would expect it to validate the field. However, the code throws the following error on class init:

TypeError: Subscripted generics cannot be used with class and instance checks

This seems to be the default behavior of isinstance when dealing with any subscripted generic field, for e.g. List[str] or Optional[str]. See related SO answer. Consequently validate also suffers with the same issue.

There is an easy workaround, setting check_type=False, and adding a custom validator:

f = field(type_hint=List[A], check_type=False,
              validators=lambda l: isinstance(l, list) and all([isinstance(e, A) for e in l]),)

However, I still thought I would add this issue in case someone runs into it. (Don't know if this should be handled at a library level, given this is a limitation of the default isinstance)

This would provide a strict equivalent of attrs behaviour. But for which usage ?

I would rather enable this only for classes that are read-only, so that the behaviour is intuitive.

Ideally we should use typing.cast to help the type checker get it.

Spec:

• both as field() argument and as @<field>.validator(...) decorator. Both can be specified in which case the arguments are used first.
• if argument: a possibly ordered dictionary of <error_msg>: <validator>. For old python support, a list of key/val tuples is supported.
• <validator> should be a callable accepting one, two or three arguments (val), (obj, val) or (obj, field, val) so that it has access to the full object on value modification (it can check values of other fields)
• <validator> should return True or None in case of success (same definition than valid8)
• when provided as a decorator, the error message is the argument of the decorator: @<field_name>.validator(<error message>)

This will lower the risk of a field not being initialized. Indeed ancestor class do not know about the subclasses so there is no risk that they require a subclass' field to be initialized. The opposite is not true.

Steps to reproduce (on docker):

docker pull ubuntu
docker run -it ubuntu bash

# Inside docker shell
apt-get -y update
apt-get install -y python python3 python-pip python3-pip
pip3 install pyfields

Stacktrace:

root@eddc884167cf:/# pip3 install pyfields
Collecting pyfields
  Downloading https://files.pythonhosted.org/packages/9f/4b/1e3eb1b5f35f51117d8b9801a4c051bdbc1b5d9889d1b48cdb056b1f7ec4/pyfields-0.12.0-py3-none-any.whl (43kB)
    100% |################################| 51kB 140kB/s
Collecting sentinel (from pyfields)
  Downloading https://files.pythonhosted.org/packages/2e/12/867f97b68c2a541f1e0de6b3b8ff52a2c2bef8d88228efc885ad07f20968/sentinel-0.1.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-4q0121fl/sentinel/setup.py", line 5, in <module>
        long_description = readme.read()
      File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 64: ordinal not in range(128)

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-4q0121fl/sentinel/

From #43 :

As of now we seem to have to explicitly declare optionality for the "check type" to skip the type validation in case of a nullable field.

For example:

from typing import Optional

class A:
     field1: Optional[int] = field(check_type=True, default=None)
     field2: Optional[int] = field(check_type=True)

Note that this is completely independent from the fact that a field is optional or mandatory. However the usual python behaviour for typing is to consider nullable all arguments that have a None default value, see https://stackoverflow.com/a/57390124/7262247.

if the class contains a __validate_all_fields__(self) method, it will be called everytime a field is modified. This is quite heavy but might be useful in some rare cases where validation can not be done in each of the fields validators

• you can disable this temporarily by doing "with skip_global_validation(o): ..." (implementation note: this would set a __skip_global_validation__ flag on the instance, read by the fields setters)

• or permanently with set_global_validation(o, enabled=False) ?

Several cases to support

1. user writes his own init entirely. This is already supported and he can set the fields correctly (or not, for the optional)
2. user writes his own entirely but not the type hints, doc and defaults (they have already been declared, why copying again). Support the case where he uses all, or some of, the fields as constructor arguments. (will not be fixed for now)
3. user writes his own but does not want to write all the argument names so he could use a special variable fields in the constructor, and anywhere in his constructor do fields.assign() or fields.init(). (fixed in #13 - @inject_fields)
4. user writes his own but wished object creation to be faster than the above. In that case (to benchmark though), we would be rather interested by some kind of a __post_init__ support like in @dataclass (fixed in 0.5.0 - @init_fields and make_init)
5. user wants fully automatic init creation (fixed in 0.5.0 - make_init)

In all these cases,

• The initialization order can be by default the order of appearance in the class. We could also add a init_after=<fields> argument to field() so that a field requiring other fields to be set before validation, converter or default_factory execution could be initialized afterwards. Another option would be to support a special way to declare that one of these three arguments depends on another field. (now in #23)
• we could optionally enable validation that all mandatory fields have been set at the end of init. That could be implicit or explicit (i.e. in use case 3. above, that could be done in fields.assign ?) (now in #24)

So as to optionally transform it to a descriptor field if needed, or raise an error.

Indeed our fix_field does not look at parent classes so this will not work:

class A(object):
    a = field(default='hello')

class B(A):
    pass

b = B()
assert b.a == 'hello'  # first access should have fixed the field name

# make sure that for all python versions (especially 2 and 3.5) the name is now ok.
assert A.__dict__['a'].name == 'a'

The initialization order can be by default the order of appearance in the class. We could also add a init_after=<fields> argument to field() so that a field requiring other fields to be set before validation, converter or default_factory execution could be initialized afterwards. Another option would be to support a special way to declare that one of these three arguments depends on another field.

maybe one day, if one tries to add a validator or something else, allow them to do it and transform this field to a descriptor field automatically by copying the functions from the other class. Seems Hacky...

It could apply to validators, converters, etc.

It could also be done explicitly with a .to_descriptor_field() or .use_descriptor() method.

Let's say I have classes:

class A:
     f1 = field()
     __init__ = make_init()

class B(A):
     f2 = field()
     __init__ = make_init(f2, A.f1)

i.e. I want to change the order of arguments in init. The above fails with the error:

pyfields.core.ClassFieldAccessError: Accessing a `field` from the class is not yet supported. See https://github.com/smarie/python-pyfields/issues/12

Any way to make this work? Till then I'm working around by manually creating the __init__

Feature 3

user writes his own but does not want to write all the argument names so he could use a special variable fields in the constructor, and anywhere in his constructor do fields.assign() or fields.init().

        @inject_fields(height, color)
        def __init__(self, fields):
            fields.init()

pros:

• you can use autocompletion so it is fast to type the decorators
• you only declare the fields that you need, in the order that you like
• you control when init happens
• you can easily debug what happens during field initialization

cons:

• it does not seem possible to have @inject_fields work without arguments, or even with the class as argument, to say "use all fields from class", except if we allow introspection to be used. Indeed, at the time where the __init__ method is decorated, it is not yet bound to the class. EDIT actually it is possible: we can create a non-data descriptor on the class (this was inspired by this link).

Originally posted by @smarie in #2 (comment)

We need to clarify

• the callback signature: probably just (obj, field) for get and (obj, field, value, has_changed) for set.
• should the observer be called in the same thread ? probably yes for now. Later we could imagine using asyncio.loop.call_soon but that seems overkill for now
• should the observer be called after setting the value in case of a set observer ? Maybe better to support (A) generators with a single yield and (B) normal methods, in which case they will be called after* the field is set.
• the name of the option in field() and in the associated decorator (like for validators and converters). Maybe get_observers and set_observers (without the 's' for decorator and corresponding add_xxx method) ?

See https://traitlets.readthedocs.io/en/stable/using_traitlets.html#observe and https://python-3-patterns-idioms-test.readthedocs.io/en/latest/Observer.html

For example this raises an error because x is initialized first, and tries to copy y, which has not been initialized still.

class C(object):
    y = field()
    x = field(default_factory=copy_field(y))

    @init_fields
    def __init__(self):
        pass

c = C(y=1)