pytest plugin for testing mypy types, stubs, and plugins

Last update: Dec 31, 2022

Overview

pytest plugin for testing mypy types, stubs, and plugins

Installation

This package is available on PyPI

pip install pytest-mypy-plugins

and conda-forge

conda install -c conda-forge pytest-mypy-plugins

Usage

Running

Plugin, after installation, is automatically picked up by pytest therefore it is sufficient to just execute:

pytest

Paths

The PYTHONPATH and MYPYPATH environment variables, if set, are passed to mypy on invocation. This may be helpful if you are testing a local plugin and need to provide an import path to it.

Be aware that when mypy is run in a subprocess (the default) the test cases are run in temporary working directories where relative paths such as PYTHONPATH=./my_plugin do not reference the directory which you are running pytest from. If you encounter this, consider invoking pytest with --mypy-same-process or make your paths absolute, e.g. PYTHONPATH=$(pwd)/my_plugin pytest.

You can also specify PYTHONPATH, MYPYPATH, or any other environment variable in env: section of yml spec:

- case: mypy_path_from_env
  main: |
    from pair import Pair

    instance: Pair
    reveal_type(instance)  # N: Revealed type is 'pair.Pair'
  env:
    - MYPYPATH=./pytest_mypy_plugins/tests/fixtures

What is a test case?

In general each test case is just an element in an array written in a properly formatted YAML file. On top of that, each case must comply to following types:

Property	Type	Description
`case`	`str`	Name of the test case, complies to `[a-zA-Z0-9]` pattern
`main`	`str`	Portion of the code as if written in `.py` file
`files`	`Optional[List[File]]=[]`*	List of extra files to simulate imports if needed
`disable_cache`	`Optional[bool]=False`	Set to `true` disables `mypy` caching
`mypy_config`	`Optional[Dict[str, Union[str, int, bool, float]]]={}`	Inline `mypy` configuration, passed directly to `mypy` as `--config-file` option
`env`	`Optional[Dict[str, str]]={}`	Environmental variables to be provided inside of test run
`parametrized`	`Optional[List[Parameter]]=[]`*	List of parameters, similar to `@pytest.mark.parametrize`
`skip`	`str`	Expression evaluated with following globals set: `sys`, `os`, `pytest` and `platform`
`expect_fail`	`bool`	Mark test case as an expected failure, like `@pytest.mark.xfail`
`regex`	`str`	Allow regular expressions in comments to be matched against actual output. Defaults to "no", i.e. matches full text.

(*) Appendix to pseudo types used above:

class File:
    path: str
    content: Optional[str] = None
Parameter = Mapping[str, Any]

Implementation notes:

main must be non-empty string that evaluates to valid Python code,
content of each of extra files must evaluate to valid Python code,
parametrized entries must all be the objects of the same type. It simply means that each entry must have exact same set of keys,
skip - an expression set in skip is passed directly into eval. It is advised to take a peek and learn about how eval works.

Example

1. Inline type expectations

# typesafety/test_request.yml
- case: request_object_has_user_of_type_auth_user_model
  main: |
    from django.http.request import HttpRequest
    reveal_type(HttpRequest().user)  # N: Revealed type is 'myapp.models.MyUser'
    # check that other fields work ok
    reveal_type(HttpRequest().method)  # N: Revealed type is 'Union[builtins.str, None]'
  files:
    - path: myapp/__init__.py
    - path: myapp/models.py
      content: |
        from django.db import models
        class MyUser(models.Model):
            pass

2. `@parametrized`

- case: with_params
  parametrized:
    - val: 1
      rt: builtins.int
    - val: 1.0
      rt: builtins.float
  main: |
    reveal_type({[ val }})  # N: Revealed type is '{{ rt }}'

3. Longer type expectations

- case: with_out
  main: |
    reveal_type('abc')
  out: |
    main:1: note: Revealed type is 'builtins.str'

4. Regular expressions in expectations

- case: expected_message_regex_with_out
  regex: yes
  main: |
    a = 'abc'
    reveal_type(a)
  out: |
    main:2: note: .*str.*

5. Regular expressions specific lines of output.

- case: expected_single_message_regex
  main: |
    a = 'hello'
    reveal_type(a)  # NR: .*str.*

Options

mypy-tests:
  --mypy-testing-base=MYPY_TESTING_BASE
                        Base directory for tests to use
  --mypy-ini-file=MYPY_INI_FILE
                        Which .ini file to use as a default config for tests
  --mypy-same-process   Run in the same process. Useful for debugging, will create problems with import cache
  --mypy-extension-hook=MYPY_EXTENSION_HOOK
                        Fully qualified path to the extension hook function, in case you need custom yaml keys. Has to be top-level.
  --mypy-only-local-stub
                        mypy will ignore errors from site-packages

License

MIT

Comments

Cut down on noise in expected strings

Many times the full expectation string is long and includes low value information. I'd like to introduce some way to cut down on that noise. One thought is to make the expected string a fullmatch regex. This could also handle #45.

- case: test_logic_promotion
  main: |
    from hdltypes.logic import StdLogic, X01Z, Bit

    a: StdLogic
    b: X01Z
    c: Bit

    reveal_type(c & a) # N: .* "hdltypes.logic.StdLogic(?:\\*|`-?\\d+)"
    reveal_type(c & b) # N: .* "hdltypes.logic.X01Z(?:\\*|`-?\\d+)"
    reveal_type(c & c) # N: .* "hdltypes.logic.Bit(?:\\*|`-?\\d+)"

It might also be possible to make the expected strings templates to enable reuse, and perhaps package some of the common patterns with the tool.

- case: test_logic_promotion
  main: |
    # ...

    reveal_type(c & a) # N: {reveal} "hdltypes.logic.StdLogic{_}"
    reveal_type(c & b) # N: {reveal} "hdltypes.logic.X01Z{_}"
    reveal_type(c & c) # N: {reveal} "hdltypes.logic.Bit{_}"
  meta:
    reveal: "Revealed type is"
    _: "(?:\\*|`-?\\d+)"

The language here is strings, so using the common tools for that: regexes and templates, makes a lot of sense.

opened by ktbarrett 19

Refactor and fix assert_expected_matched_actual

This PR:

Refactors assert_expected_matched_actual function to avoid repeated matching between expected and actual output
Fixes #63
Fixes #64

The following test file:

- case: all_mismatched
  main: |
    reveal_type(42)  # N: Revealed type is "Literal['foo']?"
    reveal_type("foo")  # N: Revealed type is "Literal[42]?"

- case: missing_message_then_match
  main: |
    reveal_type(42)
    reveal_type("foo")  # N: Revealed type is "Literal['foo']?"

- case: match_then_missing_message
  main: |
    reveal_type(42)  # N: Revealed type is "Literal[42]?"
    reveal_type("foo")

- case: missing_message
  main: |
    42 + "foo"

- case: mismatched_message_inline
  main: |
    1 + 1  # E: Unsupported operand types for + ("int" and "int")

- case: mismatched_messaged_in_out
  main: |
    1 + "foo"
  out: |
    main:1: error: Unsupported operand types for + ("int" and "int")

- case: match_then_mismatched_message
  main: |
    reveal_type(42)  # N: Revealed type is "Literal[42]?"
    reveal_type("foo")  # N: Revealed type is "builtins.int"

- case: mismatched_message_then_match
  main: |
    reveal_type("foo")  # N: Revealed type is "builtins.int"
    reveal_type(42)  # N: Revealed type is "Literal[42]?"

- case: match_between_mismatched_messages
  main: |
    reveal_type(42.0)  # N: Revealed type is "builtins.float"
    reveal_type("foo")  # N: Revealed type is "builtins.int"
    reveal_type(42)  # N: Revealed type is "Literal[42]?"

has been used to check for expected failures and gives output as shown below

test-expect-fail.yaml FFFFFFFFF                                                     [100%]

======================================== FAILURES =========================================
_____________________________________ all_mismatched ______________________________________
/home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:3: 
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Actual:
E     main:1: note: Revealed type is "Literal[42]?" (diff)
E     main:2: note: Revealed type is "Literal['foo']?" (diff)
E   Expected:
E     main:1: note: Revealed type is "Literal['foo']?" (diff)
E     main:2: note: Revealed type is "Literal[42]?" (diff)
E   Alignment of first line difference:
E     E: ...ed type is "Literal['foo']?"
E     A: ...ed type is "Literal[42]?"
E                               ^
_______________________________ missing_message_then_match ________________________________
/home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:9: 
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Actual:
E     main:1: note: Revealed type is "Literal[42]?" (diff)
E     main:2: note: Revealed type is "Literal['foo']?" (diff)
E   Expected:
E     main:2: note: Revealed type is "Literal['foo']?" (diff)
E   Alignment of first line difference:
E     E: main:2: note: Revealed type is "Literal['foo']?"
E     A: main:1: note: Revealed type is "Literal[42]?"
E             ^
_______________________________ match_then_missing_message ________________________________
/home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:12: 
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Actual:
E     main:2: note: Revealed type is "Literal['foo']?" (diff)
E   Expected:
E     (empty)
_____________________________________ missing_message _____________________________________
/home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:17: 
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Output is not expected: 
E   Actual:
E     main:1: error: Unsupported operand types for + ("int" and "str") (diff)
E   Expected:
E     (empty)
________________________________ mismatched_message_inline ________________________________
/home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:22: 
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Actual:
E     (empty)
E   Expected:
E     main:1: error: Unsupported operand types for + ("int" and "int") (diff)
_______________________________ mismatched_messaged_in_out ________________________________
/home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:26: 
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Actual:
E     main:1: error: Unsupported operand types for + ("int" and "str") (diff)
E   Expected:
E     main:1: error: Unsupported operand types for + ("int" and "int") (diff)
E   Alignment of first line difference:
E     E: ...rand types for + ("int" and "int")
E     A: ...rand types for + ("int" and "str")
E                                        ^
______________________________ match_then_mismatched_message ______________________________
/home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:33: 
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Actual:
E     ...
E     main:2: note: Revealed type is "Literal['foo']?" (diff)
E   Expected:
E     ...
E     main:2: note: Revealed type is "builtins.int" (diff)
E   Alignment of first line difference:
E     E: ...te: Revealed type is "builtins.int"
E     A: ...te: Revealed type is "Literal['foo']?"
E                                 ^
______________________________ mismatched_message_then_match ______________________________
/home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:37: 
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Actual:
E     main:1: note: Revealed type is "Literal['foo']?" (diff)
E     ...
E   Expected:
E     main:1: note: Revealed type is "builtins.int" (diff)
E     ...
E   Alignment of first line difference:
E     E: ...te: Revealed type is "builtins.int"
E     A: ...te: Revealed type is "Literal['foo']?"
E                                 ^
____________________________ match_between_mismatched_messages ____________________________
/home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:43: 
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Actual:
E     ...
E     main:2: note: Revealed type is "Literal['foo']?" (diff)
E     ...
E   Expected:
E     ...
E     main:2: note: Revealed type is "builtins.int" (diff)
E     ...
E   Alignment of first line difference:
E     E: ...te: Revealed type is "builtins.int"
E     A: ...te: Revealed type is "Literal['foo']?"
E                                 ^
================================= short test summary info =================================
FAILED test-expect-fail.yaml::all_mismatched - 
FAILED test-expect-fail.yaml::missing_message_then_match - 
FAILED test-expect-fail.yaml::match_then_missing_message - 
FAILED test-expect-fail.yaml::missing_message - 
FAILED test-expect-fail.yaml::mismatched_message_inline - 
FAILED test-expect-fail.yaml::mismatched_messaged_in_out - 
FAILED test-expect-fail.yaml::match_then_mismatched_message - 
FAILED test-expect-fail.yaml::mismatched_message_then_match - 
FAILED test-expect-fail.yaml::match_between_mismatched_messages - 
==================================== 9 failed in 2.77s ====================================

opened by zero323 12

Test that snippet passes

In mypy internal testing suite I can omit output to indicate that I expect snippet to pass

[case import]
from collections import namedtuple
[out]

This is very useful for basics checks.

However, I cannot get similar behaviour here. If I create a package

├── mypackage
│   └── __init__.py
├── mypy.ini
└── test
    └── test_mypackage.yaml

with

# ./mypackage/__init__.py
def bar(x: int) -> None:
    pass

and

./test/test_mypackage.yaml
- case: test_mypackage
  main: |
    from mypackage import foo

tests run without errors (pytest-mypy-plugins==1.9.1, mypy==0.910) despite incorrect import foo.

I have to put some output check

./test/test_mypackage.yaml
- case: test_mypackage
  main: |
    from mypackage import foo      
    reveal_type(True)  # N: Revealed type is "Literal[True]?"

to get an exception.

I also tried using empty out block

- case: test_mypackage
  main: |
    from mypackage import foo      
    reveal_type(True)
  out: ""

but it still silently ignores the broken import.

opened by 0x143 10

pdb not catching although --mypy-same-process

Hey @sobolevn, hope you're doing great in those crazy days !

Anyway, I want to keep this short. I am trying to debug a certain issue in my code but I have the problem. Neither pycharm nor pdb is able to break in a place I want. It makes no difference whether I make a breakpoint or do pdb.set_trace(). It just ignores that.

Here's relevant portion of my setup https://github.com/kornicameister/axion/blob/master/pytest.ini#L9 Any hints are welcome.

opened by kornicameister 10
Errors reported for 3rd-party dependencies
When I run type-checking of my code with mypy - everything works correctly. But, when I try to run type-tests on some module, then it breaks, example:

E ../../../../../../Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/hypothesis/vendor/pretty:706: error: Call to untyped function "_safe_getattr" in typed context (diff)
opened by sobolevn 8
Update path argument

This PR introduces compatibility with pytest 7.0.0rc1. As described in the link included in the original issue. fspath with LocalPath is deprecated in favor of path with pathlib.Path.

Closes #88

opened by zero323 7
Output-file rewriting removes '.py' in too many places
The last line of pytest_mypy_plugins.items.replace_fpath_with_module_name() does a blanket removal of all ".py" strings:

return line.strip().replace(".py", "")

This destroys paths in log information when mypy.verbosity > 0. Paths like "$HOME/.pyenv" and ".../python3.6/.../__init__.pyi" are converted into random strings.

I suspect you are only trying to replace "main.py: 1: note: ..." -type strings at the beginning of the line. If that is true (and I really haven't explored enough to be sure, so please check this) then a simple replacement my be replace(".py:", ":") to ensure the offending .py looks like a filespec.

Otherwise maybe only do the replacement in the first colon-delimited field?
bug
opened by aghast 7

AttributeError: type object 'YamlTestFile' has no attribute 'from_parent' in pytest_collect_file

The latest update to pytest_mypy_plugins broke my CI tests with the following error

==================================== ERRORS ====================================

______________________ ERROR collecting tests/__init__.py ______________________

../../../virtualenv/python3.8.0/lib/python3.8/site-packages/pluggy/hooks.py:286: in __call__

    return self._hookexec(self, self.get_hookimpls(), kwargs)

../../../virtualenv/python3.8.0/lib/python3.8/site-packages/pluggy/manager.py:92: in _hookexec

    return self._inner_hookexec(hook, methods, kwargs)

../../../virtualenv/python3.8.0/lib/python3.8/site-packages/pluggy/manager.py:83: in <lambda>

    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(

../../../virtualenv/python3.8.0/lib/python3.8/site-packages/pytest_mypy_plugins/collect.py:139: in pytest_collect_file

    return YamlTestFile.from_parent(parent, fspath=path)

E   AttributeError: type object 'YamlTestFile' has no attribute 'from_parent'

!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!

=============================== 1 error in 0.11s ===============================

The command "pytest" exited with 2.

view the log outputs here: https://travis-ci.org/github/lovasoa/marshmallow_dataclass/jobs/705848682

opened by lovasoa 7

How can I test that an assignment is legal? i.e that no output is generated
Thanks for this great tool, its been indispensable to my project!

I have a plugin that defines a protocol intersection type, and I want to test that it is invariant to argument order. My test case looks like this:

from typing import Protocol from my_package import Intersection class P1(Protocol): pass class P1(Protocol): pass i: Intersection[P1, P2] i2: Intersection[P2, P1] = i # I want to assert that no output is generated on this line
opened by suned 6
Test failures using Python 3.11 and Mypy 0.981
Hi!

Tests are failing on Python 3.11 due to a Mypy error and I'm not sure why. :thinking:

I forked pytest-mypy-plugins and added 3.11-dev Python version to CI tests: https://github.com/Delgan/pytest-mypy-plugins/commit/27f9037897507026519c67fcf016758bd2b3d7a6 You can see the failing test here: https://github.com/Delgan/pytest-mypy-plugins/actions/runs/3166219555/jobs/5155788328

E ../../opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/mypy/typeshed/stdlib/builtins.pyi:1865: error: Overloaded function signatures 1 and 2 overlap with incompatible return types (diff) E ../../opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/mypy/typeshed/stdlib/builtins.pyi:1885: error: Overloaded function signatures 1 and 2 overlap with incompatible return types (diff)

When I manually run mypy on the code base no error is reported, so I'm a bit confused about why it's failing when tests are ran through pytest and this plugin.
opened by Delgan 4

Does not work with pytest-cov

» pytest
================================ test session starts =================================
platform darwin -- Python 3.7.3, pytest-4.6.3, py-1.7.0, pluggy-0.12.0
Using --randomly-seed=1560429839
rootdir: /Users/sobolev/Documents/github/returns, inifile: setup.cfg
plugins: mypy-plugins-0.3.0, asyncio-0.10.0, cov-2.7.1, randomly-3.0.0
collected 49 items / 3 errors / 46 selected                                          
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/_pytest/main.py", line 206, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/_pytest/main.py", line 250, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/pluggy/manager.py", line 87, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/pluggy/manager.py", line 81, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/pytest_cov/plugin.py", line 229, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/pytest_cov/engine.py", line 171, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/coverage/control.py", line 818, in combine
INTERNALERROR>     self.data, aliases=aliases, data_paths=data_paths, strict=strict,
INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/coverage/data.py", line 736, in combine_parallel_data
INTERNALERROR>     data.update(new_data, aliases=aliases)
INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/coverage/data.py", line 488, in update
INTERNALERROR>     raise CoverageException("Can't combine line data with arc data")
INTERNALERROR> coverage.misc.CoverageException: Can't combine line data with arc data

Configuration: https://github.com/dry-python/returns/blob/master/setup.cfg

Test: test-data/typecheck/return.test

[CASE no_incompatible_meta_nested_class_false_positive]
1 + '2'
[/CASE]

opened by sobolevn 4

Feature proposal: Allow specifying out as list
I think the current format for output could be iterated on, and be made a little friendlier to work with. Instead of today's format using multiline strings:

out: | main:7: error: Argument 1 has incompatible type "int"; expected "str" [arg-type] main:9: note: Revealed type is "builtins.bool*"

We could allow specifying out as a list of objects with properties:

out: - error: 'Argument 1 has incompatible type "int"; expected "str" [arg-type]' line: 7 - note: 'Revealed type is "builtins.bool"' line: 9

This would unlock a few different things we could then implement at a higher level of abstraction. There are probably more that will make sense, these are just the ones I can think of for now.

Special support for error-codes

out: - error: 'Argument 1 has incompatible type "int"; expected "str"' code: "arg-type" line: 7

Special support for revealed type

out: - revealed_type: "builtins.bool" line: 9

Allow mixing regex with non-regex

out: - error: 'Argument 1 has incompatible type "int\*?"; expected "str\*?"' line: 7 regex: true

Omitting line numbers

This probably requires more knowledge of the internals than I currently have, but I think it might make sense to allow omitting line numbers. I'm proposing that this example would match if there's revealed type on line nine, followed by an error message on any line after that (strictly greater than line 9).

out: - note: 'Revealed type is "builtins.bool"' line: 9 - error: 'Argument 1 has incompatible type "int"; expected "str"' code: "arg-type"

What do you think? 🙂
opened by antonagestam 3
mypy_config documentation is confusing
The mypy_config field accepts a string as below.

- case: foo mypy_config: python_version = 3.8

So, if we want to provide multi fields, it would be like below, where the value of mypy_config in the YAML format is of type str.

- case: foo mypy_config: | python_version = 3.8 ignore_missing_imports = True

However, the document says its type is Optional[Dict[str, Union[str, int, bool, float]]]={} which I think means the following YAML format (dict syntax) is accepted.

- case: foo mypy_config: python_version: 3.8 ignore_missing_imports: true

but actually it is not.

My suggestion is either

to change the documentation to say the type of mypy_config is str, or

to change the implementation to accept the YAML dict.

because the files field, which is documented to accept Optional[List[File]]=[] type, can be specified with the YAML list syntax like

- path: myapp/__init__.py

, which is inconsistent and confusing.

The same problem is there on the env field. Its type is documented as Optional[Dict[str, str]]={}, but the YAML syntax for the field is list as follows:

env: - MYPYPATH=./pytest_mypy_plugins/tests/fixtures
opened by whitphx 2
Document cache directory usage
When tests are executed with typecheck_in_new_subprocess and disable_cache is False, --cache-dir is passed directly to mypy call as a path relative to root directory.

This means that the following are ignored

MYPY_CACHE_DIRin execution environment orenv` block of the test case.

cache_dir in mypy configuration.

are ignored.

This behavior is confusing, so it might be a good idea to document it.

In a long run, YamlTestItem should check if any of these are provided, and omit --cache-dir in such cases (this would be useful for example with remote cache).
opened by zero323 0
Broken mypy cache?
I am trying to investigate some issues related to caching behavior. When testing project with complex dependencies, I see serious performance degradation (roughly 20 fold on just 36 tests) compared to using mypy.test.testcheckTypeCheckSuite directly.

I thought the issue was a simple logic mistake (#82), but it seems it might be actually with find_dependent_paths

https://github.com/typeddjango/pytest-mypy-plugins/blob/a2d4adde12b0024e62f2e1661fd0dd5abb4f9191/pytest_mypy_plugins/item.py#L187

Since it uses at least main.py it includes all kinds of packages using main as a name (not necessarily as a module name, could be even an argument), for example

['/tmp/.mypy_cache/3.9/pdb', '/tmp/.mypy_cache/3.9/unittest/main', '/tmp/.mypy_cache/3.9/unittest/__init__', '/tmp/.mypy_cache/3.9/_pytest/pytester', '/tmp/.mypy_cache/3.9/_pytest/config/__init__', '/tmp/.mypy_cache/3.9/pytest/__init__', '/tmp/.mypy_cache/3.9/asyncio/runners']

This seems to escalate (in my case, to numpy annotations, for reason yet to be determined), and break caching in general.

Possibly related to #37

Originally posted by @zero323 in https://github.com/typeddjango/pytest-mypy-plugins/issues/82#issuecomment-945904250
opened by zero323 2

Support for mypy .test files.

This is a feature request.

It might be nice to get support for handling mypy-style .test files:

[case foo]

reveal_type(1 + 1)  # N: Revealed type is "builtins.int"

[out]]

-- Comment

[case bar]

reveal_type(1 + 1)

[out]
main:2: note: Revealed type is "builtins.int"

opened by zero323 3

Improve line matching behavior
This is a feature request.

Summary:

Currently rows are matched by their position in the file / output.

User experience could be improved, if matching was performed by (file, line-number).

Details:

Let's assume that I have a test case like this

- case: break_following_2 main: | reveal_type(1 + 1) reveal_type(1.0 + 2.0) # N: Revealed type is "builtins.float" reveal_type("foo" + "bar") # N: Revealed type is "builtins.str"

When I run tests I see:

E pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: E Expected: E main:2: note: Revealed type is "builtins.float" (diff) E main:3: note: Revealed type is "builtins.str" (diff) E Actual: E main:1: note: Revealed type is "builtins.int" (diff) E main:2: note: Revealed type is "builtins.float" (diff) E main:3: note: Revealed type is "builtins.str" (diff) E E Alignment of first line difference: E E: main:2: note: Revealed type is "builtins.float" E A: main:1: note: Revealed type is "builtins.int" E

If you analyze the test case, you'll see that actual state is like this:

| line| actual | expected | match | | ---- | ----------------------------------------- | ------------------------------------------- | ------------ | | 1 | Revealed type is "builtins.int" | | ✘ | | 2 | Revealed type is "builtins.float" | Revealed type is "builtins.float" | ✓ | | 3 | Revealed type is "builtins.str" | Revealed type is "builtins.str" | ✓ |

however alignment message

E Alignment of first line difference: E E: main:2: note: Revealed type is "builtins.float" E A: main:1: note: Revealed type is "builtins.int"

clearly shows that we start with comparing line 2 of expected and line 1 of actual.

This escalates to all the following lines and probably gets worse with multi-line messages (I wanted to investigate that, hence #66).

I am aware that this is consistent with behavior of the internal mypy test suite, which returns

Expected: main:2: note: Revealed type is "builtins.float" (diff) main:3: note: Revealed type is "builtins.str" (diff) Actual: main:1: note: Revealed type is "builtins.int" (diff) main:2: note: Revealed type is "builtins.float" (diff) main:3: note: Revealed type is "builtins.str" (diff) Alignment of first line difference: E: main:2: note: Revealed type is "builtins.float" A: main:1: note: Revealed type is "builtins.int" ^

for equivalent input, but it seems a bit counter-intuitive. While it detects presence of output mismatch, it cannot do much beyond that.

Ideally, I'd like to see something around these lines:

E pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: E Expected: E (Empty) E ... E Actual: E main:1: note: Revealed type is "builtins.int" (diff) E ...

(no alignment needed for an empty line).

This should generalize to multiple interleaved blocks of matching and not matching lines, where matching blocks are indicated, but omitted.

Furthermore, errors shouldn't propagate beyond current line, in case of multline output.

Originally posted by @zero323 in https://github.com/typeddjango/pytest-mypy-plugins/issues/65#issuecomment-938144873
opened by zero323 3

Releases(1.10.1)

1.10.1(Oct 26, 2022)
Bugfixes

Removes unused depenencies for python < 3.7

Fixes compatibility with pytest 7.2, broken due to a private import from py._path.

Source code(tar.gz)
Source code(zip)
1.10.0(Aug 15, 2022)
Features

Changes how mypy>=0.970 handles MYPYPATH

Bumps minimal mypy version to mypy>=0.970

Drops python3.6 support

Source code(tar.gz)
Source code(zip)
1.9.2(Oct 19, 2021)
Bugfixes

Removes usages of distutils #71

Fixes multiline messages #66

Fixes that empty output test cases was almost ignored #63

Fixes output formatting for expected messages #66

Source code(tar.gz)
Source code(zip)
1.9.1(Sep 8, 2021)
Bugfixes

Fixes that regex and dataclasses dependencies were not listed in setup.py

Source code(tar.gz)
Source code(zip)
1.9.0(Sep 8, 2021)
Features

Adds regex support in matching test output

Adds a flag for expected failures

Replaces deprecated pystache with chevron

Misc

Updates mypy

Source code(tar.gz)
Source code(zip)
1.8.0(Sep 7, 2021)

It was missed by mistake, see 1.9.0
Source code(tar.gz)
Source code(zip)
1.7.0(May 22, 2021)
Features

Adds --mypy-only-local-stub CLI flag to ignore errors in site-packages

Source code(tar.gz)
Source code(zip)
1.6.1(Oct 26, 2020)
Bugfixes

Changes how MYPYPATH and PYTHONPATH are calcualted. We now expand $PWD variable and also include relative paths specified in env: section

Source code(tar.gz)
Source code(zip)
1.6.0(Oct 10, 2020)
Features

Adds python3.9 support

Bumps required version of pytest to >=6.0

Bumps required version of mypy to >=0.790

Misc

Moves from Travis to Github Actions

Source code(tar.gz)
Source code(zip)
1.5.0(Oct 10, 2020)
Features

Adds PYTHONPATH and MYPYPATH special handling

Source code(tar.gz)
Source code(zip)
1.4.0(Jul 7, 2020)
Adds parametrized option

Fixes deprecation warnings for pytest

Bumps minimal pytest version to 5.4

Source code(tar.gz)
Source code(zip)
1.2.0(Dec 13, 2019)
Windows support

Allow to skip tests via skip: True

Source code(tar.gz)
Source code(zip)

Owner

TypedDjango

We make types for Django framework!

GitHub https://pypi.org/project/pytest-mypy-plugins/

ApiPy was created for api testing with Python pytest framework which has also requests, assertpy and pytest-html-reporter libraries.

ApiPy was created for api testing with Python pytest framework which has also requests, assertpy and pytest-html-reporter libraries. With this f

1 Jul 11, 2022

pytest plugin providing a function to check if pytest is running.

pytest-is-running pytest plugin providing a function to check if pytest is running. Installation Install with: python -m pip install pytest-is-running

21 Nov 1, 2022

Pytest-typechecker - Pytest plugin to test how type checkers respond to code

pytest-typechecker this is a plugin for pytest that allows you to create tests t

2 Aug 20, 2022

A pytest plugin to run an ansible collection's unit tests with pytest.

pytest-ansible-units An experimental pytest plugin to run an ansible collection's unit tests with pytest. Description pytest-ansible-units is a pytest

9 Dec 9, 2022

A command-line tool and Python library and Pytest plugin for automated testing of RESTful APIs, with a simple, concise and flexible YAML-based syntax

1.0 Release See here for details about breaking changes with the upcoming 1.0 release: https://github.com/taverntesting/tavern/issues/495 Easier API t

909 Dec 15, 2022

pytest plugin for testing mypy types, stubs, and plugins

Related tags

Overview

pytest plugin for testing mypy types, stubs, and plugins

Installation

Usage

Running

Paths

What is a test case?

Example

1. Inline type expectations

2. @parametrized

3. Longer type expectations

4. Regular expressions in expectations

5. Regular expressions specific lines of output.

Options

Further reading

License

Comments

Special support for error-codes

Special support for revealed type

Allow mixing regex with non-regex

Omitting line numbers

Releases(1.10.1)

1.10.1(Oct 26, 2022)

Bugfixes

1.10.0(Aug 15, 2022)

Features

1.9.2(Oct 19, 2021)

Bugfixes

1.9.1(Sep 8, 2021)

Bugfixes

1.9.0(Sep 8, 2021)

Features

Misc

1.8.0(Sep 7, 2021)

1.7.0(May 22, 2021)

Features

1.6.1(Oct 26, 2020)

Bugfixes

1.6.0(Oct 10, 2020)

Features

Misc

1.5.0(Oct 10, 2020)

Features

1.4.0(Jul 7, 2020)

1.2.0(Dec 13, 2019)

Owner

TypedDjango

ApiPy was created for api testing with Python pytest framework which has also requests, assertpy and pytest-html-reporter libraries.

pytest plugin providing a function to check if pytest is running.

Pytest-typechecker - Pytest plugin to test how type checkers respond to code

A pytest plugin to run an ansible collection's unit tests with pytest.

A command-line tool and Python library and Pytest plugin for automated testing of RESTful APIs, with a simple, concise and flexible YAML-based syntax

Pytest plugin for testing the idempotency of a function.

A grab-bag of nifty pytest plugins

Playwright Python tool practice pytest pytest-bdd screen-play page-object allure cucumber-report

Pytest-rich - Pytest + rich integration (proof of concept)

A collection of testing examples using pytest and many other libreris

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing

Set your Dynaconf environment to testing when running pytest

pytest_pyramid provides basic fixtures for testing pyramid applications with pytest test suite

Python Projects - Few Python projects with Testing using Pytest

:game_die: Pytest plugin to randomly order tests and control random.seed

pytest plugin for manipulating test data directories and files

pytest plugin that let you automate actions and assertions with test metrics reporting executing plain YAML files

A Django plugin for pytest.

Coverage plugin for pytest.

2. `@parametrized`