Description

This PR will add support for the python 3.10 match statement to mypy.

Current progress:

• [x] parser and ast nodes
• [x] semantic analysis
• [x] type-checking
  • [x] as-pattern
  • [x] or-pattern
  • [x] literal-pattern
  • [x] value-pattern
  • [x] capture-pattern
  • [x] wildcard-pattern
  • [x] sequence-pattern
  • [x] mapping-pattern
  • [x] class-pattern
• [x] Lots of tests

Test Plan

Some tests have been added, but they don't pass yet. There will be a lot more tests in the future.

I'm currently developing and testing on cpython 3.10.0b1 and will make sure to update to newer versions every now and then.

PEP518 says that pyproject.toml is to support the minimum build requirements. I want to consider mypy as part of the minimum build on my package.

For example Pip uses a [tool.towncrier] section on pyproject.toml to generate a correctly updated NEWS.rst as part of its build. One can considers typing correctness as important to the distributed source pakcage as an well documented NEWS.rst. Other example is Black that also has it section on pyproject.toml.

Maybe you think that this is a silly feature request and that setup.cfg must be used instead, Pip or Flit or whatever can setup mypy after all. But an TOML module will be added to the standard library sooner or later, than could mypy accept this feature and antecipate TOML supporting pyproject.toml today?

I will be releasing mypy 0.920 soon. The release branch will be cut at a7d6e68541e1c6b52338dbe2e3f4a7055a033483. Please post here any PRs that need to be merged and cherry-picked.

This is an implementation of protocols proposed in PEP 544. This PR adds support for:

• generic protocols (including inference for protocols)
• recursive protocols
• special support for Type[] and isinstance()
• structural subtyping for Callable and Tuple
• other things

For example, now one can do this:

class Linked(Protocol[T]):
    val: T
    def next(self) -> 'Linked[T]': ...

class L:
    val: int
    def next(self) -> 'L': ...

def last(seq: Linked[T]) -> T:
    ...

reveal_type(last(L()))  # Revealed type is 'builtins.int*'

In general, the PR is practically feature complete, the main missing thing now is structural join/meet for partially overlapping protocols. For example, sometimes an explicit annotation is needed:

class A(Protocol):
    one: int
    def __len__(self) -> int: ...

class B(Protocol):
    another: int
    def __len__(self) -> int: ...

a: A
b: B
reveal_type([a, b])  # Revealed type is 'builtins.list[builtins.object*]'

y: List[Sized] = [a, b]  # This works OK

The reason for this is that I decided to re-use Instances for protocols, and real structural join/meet would require creating a "fake" TypeInfo to describe the resulting protocol. However, I have encountered problems with serialization/de-serialization of "fake" TypeInfos and postponed this feature (as proposed by Jukka). On the other hand, reusing Instances for protocols allowed to implement this in a very simple way. The PR is very small and 2/3 of it are extensive tests.

There is something I encountered while working on this PR. It looks like structural subtyping is significantly slower than nominal. This is why I also implement "subtype cache" for Instances, so that there is a net win in speed of few percent (even when I locally updated typeshed replacing all ABC in typing with protocols).

In terms of work-flow, I would be happy if people will try this PR and give also feedback on behaviour, not just a code review. I will soon submit a PR to typeshed with updates for typing stubs, and a PR with runtime implementation. As well, a PR to PEPs repo with an updated draft that takes into account comments that appeared so far.

@JukkaL @ambv @gvanrossum

Description

I was planning on adding a ParamSpec-based interface to something I'm making but... it turns out I need Concatenate and Z[[int]] syntax.

Closes #11833 Closes #12276 (needs test) Closes #12257 (needs test; not sure how to test cache issues) Refs #8645 External ref https://github.com/python/typeshed/issues/4827

This PR adds a new Parameters proper type to represent parameters (more later in this description), along with adding a Concatenate operator.

Test Plan

I copied all (seemingly relevant) tests from PEP 612. I plan on copying test cases from issues and thinking of a few of my own later.

Notes

(not a real section but seemed important enough)

• Concatenate support does not use a ProperType. That would be too much new code for an operator, in my opinion. Thus, it prepends to a ParamSpecType's .prefix.
• I would really really like CallableType to take in a parameters: ParamSpecType | Parameters -- this would simplify some code, and it fits better in your head. However, this would be a breaking change, so I did not do this (along with the fact that making basic changes in this direction led to 300 type errors when self-checking). ~~- Concatenate was super unspecified by the PEP in my opinion. I ended up making it work wherever ParamSpec could be (or so I think)... Which means Concatenate[int, Concatenate[str, P]] works!~~
• Error messages could be improved, but I'm not sure where I can get the context to improve those.

I tried mypy and was alarmed to see it unable to find imports that are clearly available. Reading the explanations in #1293 and #1339, I now understand why it doesn't merrily parse third-party code that will only add to a growing pile of unannotated code.

But this is a really bad first experience. I install something that claims to understand Python code, and it's confused by imports? Then it gives me two suggestions, neither of which is seems like the right advice:

note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)

I shouldn't try to set MYPYPATH, that's silly busy-work that if successful will only lead me into the unannotated third-party module trap. The imports aren't missing, so why would --ignore-missing-imports help? NOTE: using --ignore-missing-imports silences the warnings, but the option is misnamed: the imports are not missing.

Reading the help, I see --follow-imports, so I try those options. I try --follow-imports=silent, but that still complains about the imports. I try --follow-imports=skip, and that also still complains about the imports. Both of these sound like exactly what I need, and neither changes the behavior much:

$ mypy --version
mypy 0.560
$ mypy census.py
census.py:16: error: Cannot find module named 'aiohttp'
census.py:16: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
census.py:17: error: Cannot find module named 'async_timeout'
census.py:18: error: Cannot find module named 'attr'
census.py:20: error: Cannot find module named 'opaque_keys'
census.py:21: error: Cannot find module named 'opaque_keys.edx'
census.py:21: error: Cannot find module named 'opaque_keys.edx.keys'
helpers.py:3: error: No library stub file for module 'lxml'
helpers.py:3: note: (Stub files are from https://github.com/python/typeshed)
helpers.py:4: error: No library stub file for module 'lxml.html'
$ mypy --follow-imports=silent census.py
census.py:16: error: Cannot find module named 'aiohttp'
census.py:16: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
census.py:17: error: Cannot find module named 'async_timeout'
census.py:18: error: Cannot find module named 'attr'
census.py:20: error: Cannot find module named 'opaque_keys'
census.py:21: error: Cannot find module named 'opaque_keys.edx'
census.py:21: error: Cannot find module named 'opaque_keys.edx.keys'
$ mypy --follow-imports=skip census.py
census.py:16: error: Cannot find module named 'aiohttp'
census.py:16: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
census.py:17: error: Cannot find module named 'async_timeout'
census.py:18: error: Cannot find module named 'attr'
census.py:20: error: Cannot find module named 'opaque_keys'
census.py:21: error: Cannot find module named 'opaque_keys.edx'
census.py:21: error: Cannot find module named 'opaque_keys.edx.keys'
$ mypy --follow-imports=error census.py
census.py:16: error: Cannot find module named 'aiohttp'
census.py:16: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
census.py:17: error: Cannot find module named 'async_timeout'
census.py:18: error: Cannot find module named 'attr'
census.py:20: error: Cannot find module named 'opaque_keys'
census.py:21: error: Cannot find module named 'opaque_keys.edx'
census.py:21: error: Cannot find module named 'opaque_keys.edx.keys'
census.py:23: note: Import of 'html_writer' ignored
census.py:23: note: (Using --follow-imports=error, module not passed on command line)
census.py:24: note: Import of 'site_patterns' ignored
census.py:27: note: Import of 'sites' ignored
$ mypy --ignore-missing-imports census.py
$

Problem statement:

"Could there be some way to write a library like numpy so that a single codebase could simultaneously target CPython and the newer compilers, while achieving competitive speed in all cases? If so, what would it take to make that happen? If not, then what’s the next-best alternative?"**

Proposed Solution: a "PyIR" that can be consumed by various compilers. Details here: https://docs.google.com/document/d/1jGksgI96LdYQODa9Fca7EttFEGQfNODphVmbCX0DD1k/edit

Question for Mypy: A cython subset seems to be the recommended source format. Can a "strict mode" mypy be used instead to output this IR? Advantages include more expressive than cython (generics etc), bootstrap off mypy work and less fragmentation.

Excerpts from discussion on gitter:

@njsmith

at pycon this year Jukka and the mypy team were very interested in the idea of somehow using their static type stuff in (something like, this doesn't exist) "strict" mode to help with this

@kmod

I think the issue is that "programmer productivity types" fundamentally != "compiler-userful types"

For example, are subclasses subtypes?

If you pick either yes or no, then the type system becomes non-useful to one community or the other

bonus: Where would dynd's datashape play in? If so, can Dynd's datashape be used as a mypy plugin to annotate array types? @insertinterestingnamehere

[This is based on a discussion between me and @gvanrossum last week. I'm writing this down so that we won't forget about this.]

Several stdlib functions have signatures where the return type depends on the value of an argument. Examples:

• open returns an IO[str] or IO[bytes] depending on whether the mode argument has 'b' as a substring.
• subprocess.check_output returns a str or bytes depending on the value of the boolean universal_newlines argument.

A simple way to support these would be to introduce a few additional types that work with str and bool literals:

• A string pattern type that can describe the contents of a string literal, perhaps using a regular expression.
• Types for False and True.

For example, if we had FalseType and TrueType, we could write a function whose return value depends on the value of a boolean argument:

@overload
def f(x: FalseType) -> bytes: ...
@overload
def f(x: TrueType) -> str: ...

def f(x: Any) -> Any:
    if x:
        return 'x'
    else:
        return b'x'

reveal_type(f(False))  # bytes
reveal_type(f(True)  # str

Not sure about this one:

...
x = bool('')
reveal_type(f(x))  # Union[str, bytes] since we don't know the boolean value statically?

We could also have a generic class where a generic type argument depends on the value of an argument to __init__. Not sure how we should represent these -- here is one potential approach:

class C(Generic[AnyStr]):
    @overload
    @returntype('C[str]')
    def __init__(self, x: TrueType) -> None: ...
    @overload
    @returntype('C[bytes]')
    def __init__(self, x: FalseType) -> None: ...
    ...

reveal_type(C(True))  # C[str]

Alternatively, we could use __new__, but this would be problematic if there is no corresponding method defined at runtime.

Consider this example:

from typing import Callable

class A:
    f = None  # type: Callable[[int], int]

    def __init__(self, g: Callable[[int], int]) -> None:
        self.f = g

mypy reports:

trial.py: In member "__init__" of class "A":
trial.py, line 7: Cannot assign to a method
trial.py, line 7: Invalid method type
trial.py, line 7: Incompatible types in assignment (expression has type Callable[[int], int], variable has type Callable[[], int])

while I'd expect this to work.

Note that there is a second error in the reporting of the type of the variable.

Also, this passes type checking (correctly):

f = None  # type: Callable[[int], int]

def g(x: int) -> int: return 0

f = g

This is an attempt to implement the descriptor protocol (as per https://github.com/python/mypy/issues/244).

However, it currently is suffering from being being unable to resolve type variables during assignment, and I'm not sure what I would need to resolve that. For instance, using this implementation of properties:

from typing import Any, Text, TypeVar, Optional, Type, Generic, Callable, Union
from operator import attrgetter

T = TypeVar('T')
G = TypeVar('G')
S = TypeVar('S')

class MyProperty(Generic[T, G]):
    def __init__(
        self,
        fget: Callable[[T], G],
        fset: Optional[Callable[[T, G], None]] = None,
        fdel: Optional[Callable[[T], None]] = None,
        doc: Optional[Text] = None,
    ) -> None:
        self._getter = fget
        self._setter = fset
        self._deleter = fdel
        if doc:
            self.__doc__ = doc
        else:
            self.__doc__ = fget.__doc__

    def setter(self, fset: Callable[[T, G], None]):
        return MyProperty(self._getter, fset, self._deleter, self.__doc__)

    def deleter(self, fdel: Callable[[T], None]):
        return MyProperty(self._getter, self._setter, fdel, self.__doc__)

    def __get__(self, instance: Optional[T], owner: Type[T]) -> Union['MyProperty[T, G]', G]:
        if instance is None:
            return self

        return self._getter(instance)

    def __set__(self, instance: T, value: G) -> None:
        if self._setter:
            self._setter(instance, value)
        else:
            raise Exception()

    def __delete__(self, instance: T) -> None:
        if self._deleter:
            self._deleter(instance)
        else:
            raise Exception()

class NonGenericNonDataDescriptor(Generic[T]):
    def __init__(self, value: int) -> None:
        self._value = value  # type: int

    def __get__(self, instance: Optional[T], owner: Type[T]) -> int:
        return self._value

class NonGenericDataDescriptor(Generic[T]):
    def __init__(self, value: int) -> None:
        self._value = value  # type: int

    def __get__(self, instance: Optional[T], owner: Type[T]) -> Union['NonGenericDataDescriptor[T]', int]:
        if instance is None:
            return self

        return self._value

    def __set__(self, instance: T, value: int) -> None:
        self._value = value


class Test:
    my_unique_name = MyProperty(attrgetter('foo'))  # type: MyProperty[Test, int]
    ten = NonGenericNonDataDescriptor(10)  # type: NonGenericNonDataDescriptor[Test]
    some_int = NonGenericDataDescriptor(5)  # type: NonGenericDataDescriptor[Test]

    @MyProperty
    def my_other_name(self):
        return self.bar

    def __init__(self) -> None:
        self.foo = 0
        self.bar = "ten"

def f() -> None:
    t = Test()

    t.my_unique_name = 1
    t.my_unique_name + 1

    t.ten + 1

    t.some_int = 7
    t.some_int + 1

When checked with mypy, we get the following errors:

$ mypy --fast-parser test_descriptors.py --show-traceback

test_descriptors.py: note: In function "f":
test_descriptors.py:85: error: Incompatible types in assignment (expression has type "int", variable has type "G")
test_descriptors.py:86: error: Unsupported operand types for + ("Union[MyProperty[Test, int], int]" and "int")

The second error is because we don't have overloading outside of stub files (see https://github.com/python/mypy/issues/1136), I think, but the former is the one I'm not sure how to resolve.

There are some situations where a user wants to have more precisely typed **kwds. Current syntax only allows homogeneous **kwds:

def fun(x: int, *, **options: str) -> None:
    ...

However, in situations with many heterogeneous options listing all options in the signature could be verbose and will require rewriting some existing code. There is a vague idea to allow TypedDict for such situations. For example:

class Options(TypedDict):
    timeout: int
    alternative: str
    on_error: Callable[[int], None]
    on_timeout: Callable[[], None]
    ...

def fun(x: int, *, **options: Options) -> None:
    ...

Maybe for such cases the TypedDict used should be automatically understood as defined with total=False. Also it is worth mentioning that this feature will allow reusing the TypedDicts in modules where several functions have same (or similar) option sets.

Bug Report

mypy generates "error: Non-overlapping identity check" for identity comparison between optionals, even though None has the same identity as None. This is especially confusing for the x is y is None pattern, which I would expect to be valid as long as x and y are optional.

To Reproduce

def foo(x: int | None, y: str | None) -> None:
    if x is y:
        print('x and y have same identity!')

foo(None, None)

Actual Behavior

$ python repr.py
x and y have same identity!

$ mypy --strict repr.py
repr.py:2: error: Non-overlapping identity check (left operand type: "Optional[int]", right operand type: "Optional[str]")  [comparison-overlap]
Found 1 error in 1 file (checked 1 source file)

Your Environment

• Mypy version used: 0.991
• Mypy command-line flags: --strict
• Mypy configuration options from mypy.ini (and other config files): None
• Python version used: 3.10.8

Bug Report

When I transform a value into an enum, an error occurs in the match case construct. If you convert the value in advance, there will be no error. Same issue #14109 #14264 .

To Reproduce

import typing
import enum

class Color(enum.StrEnum):
    RED = enum.auto()
    GREEN = enum.auto()
    BLUE = enum.auto()

def test(color: str) -> str:
    match Color(color):
        case Color.RED:
            return 'red'
        case Color.GREEN:
            return 'green'
        case Color.BLUE:
            return 'blue'
        case _ as unreachable:
            typing.assert_never(unreachable)

https://mypy-play.net/?mypy=latest&python=3.11&gist=d28ae2af9f4545ca1c302476d8e4c782

import typing
import enum

class Color(enum.StrEnum):
    RED = enum.auto()
    GREEN = enum.auto()
    BLUE = enum.auto()

def test(color: str) -> str:
    color = Color(color)
    match color:
        case Color.RED:
            return 'red'
        case Color.GREEN:
            return 'green'
        case Color.BLUE:
            return 'blue'
        case _ as unreachable:
            typing.assert_never(unreachable)

https://mypy-play.net/?mypy=latest&python=3.11&gist=eab4deab65eb38a49649a46a2adc1e42

Actual Behavior

In the first example:

main.py:18: error: Argument 1 to "assert_never" has incompatible type "Color"; expected "NoReturn"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

In the second example:

Success: no issues found in 1 source file

Your Environment

• Mypy version used: mypy 0.991 (compiled: yes)
• Mypy command-line flags: N/A
• Mypy configuration options from mypy.ini (and other config files): N/A
• Python version used: 3.11.1

The following should be an error, but it currently is not.

from collections.abc import Callable
from typing import Any


def f(call: Callable[..., Any]) -> None:
    print(call.__name__)  # why is there no error here?

IMO the last line should cause a mypy error to the effect of "Callable" has no attribute "__name__". The callable protocol generally does not define __name__.

To further drive this point home, add the following piece:

class Bar:
    def __call__(self) -> None:
        print(f"hi mom")


f(Bar())  # this is valid
print(Bar().__name__)  # this is an error

This means that

1. Bar is correctly viewed as being a Callable subtype (structurally) and
2. it is also correctly identified to not have the __name__ attribute by mypy.
3. Yet when something is annotated as Callable, mypy just assumes it has the __name__ attribute. 🤨

Correct me if I am wrong, but these points cannot logically all be true.

The documentation of the Callable ABC clearly states that it is a class that defines the __call__ method. Nowhere does it mention the need for a Callable class to define __name__ (or anything else for that matter).

There is also no mention of any attributes/methods other than __call__ in the glossary entry for callable.

Lastly, the relevant section on Callable types of the standard type hierarchy chapter in the data model documentation again broadly defines them as

types to which the function call operation [...] can be applied

and further down even explicitly states the following:

Instances of arbitrary classes can be made callable by defining a __call__() method in their class.

I realize that user defined functions do always have the __name__ attribute (as well as a bunch of others listed in the aforementioned data model docs). But this should make them a subtype of Callable.

Yet it seems as though mypy treats a Callable as a user defined function. My example f above can be changed to any of the attributes of user defined functions, such as __kwdefaults__ for example and it would still pass mypy checks. This seems wildly inconsistent with all the documentation I could find, as well as logically inconsistent in itself.

I can only assume that there is some historic/pragmatic explanation for this, but I could not find it.

What is the reasoning behind this?

I found a discussion in #5958 that seems to be related, but found no answer to my question.

Avoid false "Incompatible return value type" errors when dealing with isinstance, constrained type variables and multiple inheritance (Fixes #13800).

The idea is to make a union of the current expanded return type and the previously expanded return types if the latter are among the bases of intersections generated by isinstance checks.

I work on a large legacy codebase with thousands of type errors. I have found recently that our configuration had ignore_missing_imports = True to account for the absence of a many of 3rd-party deps in our typechecking venv. In an attempt to improve typechecking strictness, I added a per-package stanza to enable ignore_missing_imports = False for import paths belonging to our internal code.

I found that:

• this revealed a handful of cases where import statements in if TYPE_CHECKING: block had invalid/outdated paths
• the presence of even a single such invalid import in the entire codebase, even in a file that is not imported anywhere, will mask most of the existing errors from mypy's output, even though mypy does appear to check the same number of files

For instance, prior to enabling the stricter import check I get:

...
Found 2830 errors in 815 files (checked 11667 source files)

with the stricter import check, if any import path is invalid, I get:

...
Found 182 errors in 78 files (checked 11667 source files)

This is on macOS, with mypy 0.971

Unfortunately the codebase is not public and I have not yet managed to create a minimal reproducer.

(Explain how this PR changes mypy.)

stubgen currently allows a variable to be an alias to another, but only at the module level.

For example:

def func() -> int:
    return 2

aliased_func = func

class Foo:
    def meth() -> int:
        return 2
    
    aliased_meth = meth

produces:

def func() -> int: ...
aliased_func = func

class Foo:
    def meth() -> int: ...
    aliased_meth: Incomplete   # <--- should be  `aliased_meth = meth`

At the class level, these aliased variables are marked as Incomplete.

In my experience, enabling aliases below the top level produces better stubs.

I'm not sure what the original reservations were around use of these aliases. If the concerns about class-level aliases are still valid, I can add an option to enable them.