Spec: Annotating the `self` argument in `__init__` methods about typing HOT 19 CLOSED

I think you forgot to add is_int=True to the last line in the first code block?

Thanks for the catch, updated.

from typing.

Don't forget intersections with Self:

from typing import Self

class H:
    def __init__(self: Self):
        return


reveal_type(H())

and old-Self:

from typing import TypeVar

Self = TypeVar("Self")

class H:
    def __init__(self: Self, other: type[Self]):
        return


reveal_type(H(H))

from typing.

Is your Bar example incorrect? It works in neither Mypy nor Pyright

The example is correct. My point in providing that example is to demonstrate that there are edge cases that are currently undefined — and therefore produce behaviors that may be undesirable with the current type checker implementations.

The behavior in this case is unspecified, so arguably any behavior is "correct" according to the current spec.

from typing.

I think you forgot to add is_int=True to the last line in the first code block?

from typing.

Old self there is missing a bound=H

from typing.

Thanks for starting this discussion.

The example you posted above is fine, but it don't really demonstrate why __init__ needs to be treated as special by a type checker. With the examples you've provided above, the current type specification indicates the behavior that type checkers should provide, and there would be no need for an extension to the spec.

The unspecified behavior occurs when the annotation for self includes one or more type variable (class-scoped, method-scoped, or both). This is where the rules become unclear because the __init__ method acts differently from other methods.

Normally, type variables appears in input parameter types and the return types, and it's the job of a type checker to "solve" the type variables based on the arguments passed to the call and then specialize the return type based on the solved type variables.

In this example, T appears in the types for input parameters x and and y, and it is solved based on the arguments corresponding to those parameters. The return type (list[T]) is then specialized based on the solved value of T.

def func(x: T, y: T) -> list[T]:
    return [x, y]

reveal_type(func(1, 2)) # list[int]
reveal_type(func("", b"")) # list[str | bytes]

The return type of an __init__ method is always None, so its return type annotation never contains a type variable. However, the type annotation for the self parameter kind of acts like a return type in the case of __init__. This is different from every other method — and the reason why this needs special-cased behavior in a type checker.

T = TypeVar("T")
S = TypeVar("S")

# A class-scoped type variable is used in this example:
class Foo(Generic[T]):
    def __init__(self: Foo[T], value: T) -> None: ...

reveal_type(Foo(1)) # Foo[int]
reveal_type(Foo("")) # Foo[str]

# A method-scoped type variable is used in this example:
class Bar(Generic[T]):
    def __init__(self: "Bar[S]", value: S) -> None:
        ...

reveal_type(Bar(1))  # Bar[int]
reveal_type(Bar(""))  # Bar[str]

The __init__ method is also special in that it interacts in a complicated (and currently unspecified) way with the __new__ method if both are present on the class. When calling a class constructor, the __new__ method is invoked first, and it can potentially provide the type arguments for class-scoped type variables. (Mypy doesn't currently honor the return type of __new__, but it really should.)

That means we need to consider the intended behavior for cases like this:

class Foo(Generic[T]):
    def __new__(cls, value: T) -> Foo[T]: ...

    def __init__(self: Foo[T], value: T) -> None: ...

from typing.

@erictraut Is your Bar example incorrect? It works in neither Mypy nor Pyright

from typing.

The example you posted above is fine, but it don't really demonstrate why __init__ needs to be treated as special by a type checker. With the examples you've provided above, the current type specification indicates the behavior that type checkers should provide, and there would be no need for an extension to the spec.

Thanks for the detailed feedback. I made some updates and still need to continue working on the requested feature.

I'm not sure I follow you here. Is the following example (which I've removed from the first post) already valid and formalized somewhere in the spec?

from typing import Generic, Literal, TypeVar, overload

T = TypeVar("T")

class A(Generic[T]):
    @overload
    def __init__(self: A[int], is_int: Literal[True]) -> None:
        ...

    @overload
    def __init__(self: A[str], is_int: Literal[False] = ...) -> None:
        ...

Or is still a valid example regarding the requested feature? According to your comment here, it doesn't seem to be specified, but is a trivial example. In that case, I will probably have to add some more complex examples involving unions and type variables.

Regarding this example:

# A class-scoped type variable is used in this example:
class Foo(Generic[T]):
    def __init__(self: Foo[T], value: T) -> None: ...

Is there really a need to add support for this, as you could simply omit Foo[T] and type checkers would actually infer T with the value of value?

Staying on class-scoped type vars, it might be interesting to see what should happen with the following:

class Foo(Generic[T]):
    def __init__(self: Foo[T | None], value: T) -> None: ...

f = Foo(1)

Currently pyright discards the self annotation (i.e. f is Foo[int], not Foo[int | None]), which imo makes sense.

@A5rocks, Thanks, examples added in the already supported use cases.

from typing.

The following is a canonical example explaining the wanted behavior 1:

I may be coming late to this party, but if I take this example and simply remove the annotation for self, it gives the same results. So I am confused as to why you see a need for a change to the spec?

from typing.

I may be coming late to this party, but if I take this example and simply remove the annotation for self, it gives the same results. So I am confused as to why you see a need for a change to the spec?

My previous comment raised the same question, I'm waiting for an answer on this one.

However, the second example is making use of a method-scoped type variable where an explicit annotation on self is needed.

from typing.

However, the second example is making use of a method-scoped type variable where an explicit annotation on self is needed.

But why would you need this in real life? Please tell the story of the actual use case that led you to this proposal.

from typing.

I was the one who made the mypy issue that led to this; our motivation is proper typing for an exception group-friendly pytest.raises API.

See python/mypy#16752 which has a code sample, if that's not concrete enough for you.

We worked around this by subclassing ExceptionGroup for RaisesGroup but that's a massive hack because... it's not an exception group.

from typing.

But why would you need this in real life? Please tell the story of the actual use case that led you to this proposal.

I've updated the motivation section, providing a simplified example that I encountered when trying to correctly type hint Django fields, along with a list of already existing examples. The "introduction" was also updated to make it clear that it doesn't actually unblocks any existing issue (afaik), as a workaround is already available with __new__. The goal is mainly to standardize something already used to avoid diverging behavior between type checkers.

from typing.

Following my previous comment, I also updated the specification with examples that hopefully cover what needs to be specified. Feel free to add any other use cases that were missed.

There's a couple places where I wasn't sure what the wanted behavior is, and are open to discussion. These are marked with this Note mark:

from typing.

Thanks @Viicos. I've been working to fill in missing chapters in the typing spec. One of the next that I have on my list is a chapter on type evaluation for constructor calls. This mini-spec will fit nicely within that chapter. I'll post a draft in the Python typing discourse channel when I have something ready to review. That will be a good forum for us to pin down the remaining behaviors.

from typing.

Great to hear, and thanks for all the work on the typing spec lately. Indeed I think this "mini spec" would fit better in a more general chapter about constructor calls. As you mentioned earlier, will this include behavior with both a __init__ and __new__ method?

from typing.

@Viicos, I just posted a draft typing spec chapter on constructors. It incorporates parts of your earlier spec, although it deviates from your proposal in one significant way: it completely disallows the use of class-scoped TypeVars within a self type annotation in an __init__ method. When I was writing the spec and adding examples, I realized that allowing this construct creates ambiguities and nonsensical type evaluation results, so my proposal is to disallow it completely.

Please review the draft spec and add comments to this thread.

from typing.

The draft typing spec by @erictraut is merged, does that resolve the issues presented in the OP such that this can be closed?

from typing.

It does, thanks!

from typing.