Local continuous test runner with pytest and watchdog.

Joe Esposito

Last update: Dec 23, 2022

Related tags

Testing pytest-watch

Overview

pytest-watch -- Continuous pytest runner

pytest-watch a zero-config CLI tool that runs pytest, and re-runs it when a file in your project changes. It beeps on failures and can run arbitrary commands on each passing and failing test run.

Motivation

Whether or not you use the test-driven development method, running tests continuously is far more productive than waiting until you're finished programming to test your code. Additionally, manually running pytest each time you want to see if any tests were broken has more wait-time and cognitive overhead than merely listening for a notification. This could be a crucial difference when debugging a complex problem or on a tight deadline.

Installation

$ pip install pytest-watch

Usage

$ cd myproject
$ ptw
 * Watching /path/to/myproject

Note: It can also be run using its full name pytest-watch.

Now develop normally and check the terminal every now and then to see if any tests are broken. Alternatively, pytest-watch can notify you when tests pass or fail:

OSX

$ ptw --onpass "say passed" --onfail "say failed"

$ ptw --onpass "growlnotify -m \"All tests passed!\"" \
      --onfail "growlnotify -m \"Tests failed\""

using GrowlNotify.

Windows
```
> ptw --onfail flash
```
using Console Flash

You can also run a command before the tests run, e.g. seeding your test database:

$ ptw --beforerun init_db.py

Or after they finish, e.g. deleting a sqlite file. Note that this script receives the exit code of pytest as an argument.

$ ptw --afterrun cleanup_db.py

You can also use a custom runner script for full pytest control:

$ ptw --runner "python custom_pytest_runner.py"

Here's an minimal runner script that runs pytest and prints its exit code:

# custom_pytest_runner.py

import sys
import pytest

print('pytest exited with code:', pytest.main(sys.argv[1:]))

Need to exclude directories from being observed or collected for tests?

$ ptw --ignore ./deep-directory --ignore ./integration_tests

See the full list of options:

$ ptw --help
Usage: ptw [options] [--ignore <dir>...] [<directory>...] [-- <pytest-args>...]

Options:
  --ignore <dir>        Ignore directory from being watched and during
                        collection (multi-allowed).
  --ext <exts>          Comma-separated list of file extensions that can
                        trigger a new test run when changed (default: .py).
                        Use --ext=* to allow any file (including .pyc).
  --config <file>       Load configuration from `file` instead of trying to
                        locate one of the implicit configuration files.
  -c --clear            Clear the screen before each run.
  -n --nobeep           Do not beep on failure.
  -w --wait             Waits for all tests to complete before re-running.
                        Otherwise, tests are interrupted on filesystem events.
  --beforerun <cmd>     Run arbitrary command before tests are run.
  --afterrun <cmd>      Run arbitrary command on completion or interruption.
                        The exit code of "pytest" is passed as an argument.
  --onpass <cmd>        Run arbitrary command on pass.
  --onfail <cmd>        Run arbitrary command on failure.
  --onexit <cmd>        Run arbitrary command when exiting pytest-watch.
  --runner <cmd>        Run a custom command instead of "pytest".
  --pdb                 Start the interactive Python debugger on errors.
                        This also enables --wait to prevent pdb interruption.
  --spool <delay>       Re-run after a delay (in milliseconds), allowing for
                        more file system events to queue up (default: 200 ms).
  -p --poll             Use polling instead of OS events (useful in VMs).
  -v --verbose          Increase verbosity of the output.
  -q --quiet            Decrease verbosity of the output (precedence over -v).
  -V --version          Print version and exit.
  -h --help             Print help and exit.

Configuration

CLI options can be added to a [pytest-watch] section in your pytest.ini file to persist them in your project. For example:

# pytest.ini

[pytest]
addopts = --maxfail=2


[pytest-watch]
ignore = ./integration-tests
nobeep = True

Alternatives

xdist offers the --looponfail (-f) option (and distributed testing options). This instead re-runs only those tests which have failed until you make them pass. This can be a speed advantage when trying to get all tests passing, but leaves out the discovery of new failures until then. It also drops the colors outputted by pytest, whereas pytest-watch doesn't.
Nosey is the original codebase this was forked from. Nosey runs nose instead of pytest.

Contributing

Check the open issues or open a new issue to start a discussion around your feature idea or the bug you found
Fork the repository, make your changes, and add yourself to Authors.md
Send a pull request

If you want to edit the README, be sure to make your changes to README.md and run the following to regenerate the README.rst file:

$ pandoc -t rst -o README.rst README.md

If your PR has been waiting a while, feel free to ping me on Twitter.

Use this software often? 😃

Comments

Multiple directories support, norecursedirs, passing args to pytest
I believe this PR closes all outstanding issues (#5, #6, #7 and #8). If merged in, this probably calls for a version bump.

The README will probably need to be updated as well..

Options added / things modified:

All positional arguments after -- are now passed as is to py.test executable.

Added support for multiple watched folders (passed as positional arguments).

Allow to ignore sub folders (currently via an exact match only at a first level of each watched folder, but this could be improved quite easily to support arbitrary patterns). Note that this is done at the observer level and not in the event handler because if you have tens of thousands of files in a subfolder like .tox for example, the handler would get completely swamped by that (so a simple pattern matching inside on_any_event is not going to cut it).

py.test subprocess no longer changes directory to a watched folder -- this no longer makes any sense given that any args can be passed to pytest and you can watch multiple folders at once.

py.test subprocess no longer runs with shell=True, to handle quoted arguments properly.

Catch KeyboardInterrupt even if it was caught after the first test run and exit gracefully.

Instead of catching all file events, only react to FileModifiedEvent and FileCreatedEvent. This fixed the problem where duplicate watchdog events would be fired upon a single file modification if using remote NFS mounts on Linux.

Added -p alias to --poll.

The exact py.test command is now printed to stdout upon the first test run.

Added event spooling with 200ms cooldown to work around atomic saves in different editors.

Added --verbose / --quiet options.

Added --no-spool option to disable spooling.

--ext option will prepend extensions with periods if needed.

An example using the new functionality:

ptw -v -p --no-spool --ignore=.tox,.git folder1 folder2 -- tests/*.py -svv
enhancement
opened by aldanor 26
Loops endlessly after one change was detected in Boxcryptor environment

I just tried this on my project and it looks good initially. However, once I update any file (i.e. re-save), pytest-watch just performs test after test after test ... until I ctrl-c.

It should definitely not do that.

There is either a loop in the tool itself or it detects changes to files that are generated during the build process and thus always attempts to rebuild.

The latter is unlikely however since it does not start looping until the very first change is detected. Furthermore, I checked which files had changed in the directory since the "original" run of pytest-watch and it was only the file I actually did change and its corresponding __pycache__/*.pyc file. And that file is very likely not regenerated if the source file didn't change either.

It would be nice if the changed file was displayed in the console before the command is rerun.

Environment: Windows7, Python 3.5.0 (64-bit), pytest 2.8.3, py 1.4.31, pytest-watch 3.8.0, watchdog 0.8.3

opened by FichteFoll 22
Suite reruns few times

If it's important I am using pytest-watch with https://github.com/pytest-dev/pytest-incremental.

When I update any source file sometimes pytest-watch reruns test twice, sometimes even three times. Is it a problem or a feature?
enhancement

opened by Glueon 18
Add --onsigint handler, run on KeyboardInterrupt

This gets run on KeyboardInterrupt/SIGINT, which could be sent by a --beforerun script, which kills all child processes (i.e. any currently running py.test instance).

This also introduces the explicit "failed" state, which is not "not passed" with py.test.

opened by blueyed 12
Regression: passes watch dir to py.test
With ptw dir -- path/to/test_file.py ptw will include the dir in the py.test run:

Running: py.test dir path/to/test_file.py

This however will cause py.test to run the whole directory, and not only the specified test file.

This was not the case with 3.10.0, which does:

Running: py.test path/to/test_file.py

btw: I found the best method is to specify testpaths = path/to/tests with the pytest options in setup.cfg, which gets used when running just py.test.

If this was done intentionally, I think it should only get done when there are no py.test args provided, but that feels a bit too magic.
opened by blueyed 11
Improve `--ignore`

Let's continue the discussion on how to improve the current single-level --ignore (alternatively, --norecursedirs). #6 was the original issue.

Summary

The core problem is that we need a way to exclude directory trees from the watch list. If we don't, ptw can crawl to a halt walking the entire tree or worse, use up the system's file resources.

Not all operating systems handle recursive watching natively, so using solutions like inotify (Linux) and kqueues (Mac) adds a new watch to each subdirectory instead of just one at the root. (Note that some systems do, like the Windows API and Mac's FSEvents.)

So we need to balance simplicity and efficiency. As mentioned above, not all platforms need a custom recursive solution, so using one in those cases hurts instead of helps.

Another idea is to shop around. If watchdog or another 3rd-party library exposes an efficient cross-platform solution (recursive watching/ignoring on Linux, and PatternMatchingEventHandler or a regex solution like #7 for Windows and Mac), we could pass it along instead of re-implementing it.
enhancement

opened by joeyespo 11
pytest-watch with debugger

Quite often I invoke pytest with the --ipdb option so I could see what is wrong with my code on first failure.

So I invkoed pytest-watch with ptw -- --ipdb. The problem is that if after droping into ipdb I modify a test or a file it starts a new session and the one with ipdb console goes to the background. In the end I end up with dozens of ipdb instances running.

Is it possible to block ptw if current session has dropped to the ipdb?
enhancement

opened by Glueon 9
Use the same interpreter when ptw is called outside of a virtual env
My primary use case here is pipenv. I would like to be able to do:

$ pipenv run ptw

But this attempts to find a globally installed pytest:

Running: py.test Error: [Errno 2] No such file or directory: 'py.test'

With this change, pytest-watch will call pytest as a module if not currently running in an activated virtual environment. This should also make running pytest-watch globally a nicer experience by ensuring the same Python interpreter is used to run both ptw and py.test even if the user's shell is misconfigured.
bug
opened by jacebrowning 8
Provide data about failed tests for onfail callback

It would be useful to have the name of failed tests available with the --onfail callback, e.g. in an environment variable.

This could be captured from subprocess.call and then parsed. Although that probably means to force / require specific options to py.test to get a useful short test summary, i.e. -rfEsxXw.

I could imagine using a py.test hook for this though: when pytest-watch is used as a plugin, it could capture this data itself.

The data would be useful to easily open the failing test in your editor, without having to copy'n'paste it (although this can be quite easy with a plugin like https://github.com/kopischke/vim-fetch already).
enhancement

opened by blueyed 8
Added feature to poll files for help with virtual machines/docker

We do a lot of our development in docker, and the events for changed files are generated in the host OS instead of inside docker, so ptw doesn't see that files have changed. Do you have any interest in adding a polling option, that will just scan the folder for modified file times and trigger tests? I may be able to write the code if you are.
enhancement

opened by carsongee 8
multiple directories support
It would be great if:

ptw [options] [<directory> ...]

rather than:

ptw [options] [<directory>]

I managed to patch this for my personal use. But I hope this could be done in simpler way with plugin. How about changing your main logic

from wrapping around and system-calling py.test ...

to being a pytest-plugin and adding entry_points={ 'pytest11': ... } to your setup.py? (like other pytest-something plugins do)

Anyway, this is my wish:

py.test --watch --onfail='...' --onpass='...' tests

for watching and testing tests/, or

py.test --watch-dir=mypackage --onfail='...' --onpass='...' tests examples

for watching mypackage/ but testing tests/ and examples/. Or some better interface could be possible...

Thanks for reading this and releasing v2!
enhancement
opened by rakjin 8

Directories passed to pytest-watch are also passed to pytest, but shouldn't be

When I run this command

python -m pytest_watch src/

I expect pytest-watch to run pytest without any additional args and for pytest-watch to watch the src/ directory. Instead, what happens is pytest-watch correctly only watches for changes in src/ but it runs the command py.test src/, making pytest only look for tests in src/, but it should just be running pytest without additional args.

Output from above command

+ python -m pytest_watch src/

[Tue Dec 20 18:49:33 2022] Running: py.test src/
============================================================================ test session starts ============================================================================
platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0
rootdir: [removed], configfile: pytest.ini
plugins: anyio-3.6.2, requests-mock-1.10.0, Faker-15.3.4
collected 0 items                                                                                                                                                           

=========================================================================== no tests ran in 0.04s ===========================================================================

This also happens when you pass additional pytest args, it just gets concatenated

+ python -m pytest_watch src/ -- tests/

[Tue Dec 20 18:52:09 2022] Running: py.test src/ tests/

opened by adibsaad 0

`ImportError` when running tests using `ptw`
I have a folder structure like this:

root |- title_api.py |- tests |- test_title_api.py

Content of test_title_api.py is:

from title_api import * def test_dummy(): assert False

Running pytest in the root directory successfully runs test_dummy. But when ptw is run in the root directory, it complains of an ImportError as Python can't find title_api module.

pytest-watch should work similarly to pytest here (add the parent directory to sys.path before running tests).

Notes:

I install latest commit of pytest-watch from its Github url.

I am running version 7.2.0 of pytest in Python 3.11.

My OS is Windows 11.
opened by sohang3112 1
Update version of `watchdog` library

Previously, old version of watchdog library was being used, which is not compatible with Python 3.3+. I have modified watchdog version to latest in requirements.txt. This fixes Issue 127.

opened by sohang3112 0

Error in Python 3.10: `module 'collections' has no attribute 'MutableSet'`

This is because MutableSet was moved from collections module to collections.abc module in Python 3.3. So collections.abc has to be used in Python 3.3+ instead of collections.

Full traceback:

Traceback (most recent call last):
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\site-packages\watchdog\observers\__init__.py", line 86, in <module>
    from .read_directory_changes import WindowsApiObserver as Observer
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\site-packages\watchdog\observers\read_directory_changes.py", line 38, in <module>
    from watchdog.observers.api import (
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\site-packages\watchdog\observers\api.py", line 23, in <module>
    from watchdog.utils.bricks import SkipRepeatsQueue
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\site-packages\watchdog\utils\bricks.py", line 175, in <module>
    class OrderedSet(collections.MutableSet):
AttributeError: module 'collections' has no attribute 'MutableSet'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\Scripts\ptw.exe\__main__.py", line 4, in <module>
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\site-packages\pytest_watch\__init__.py", line 14, in <module>
    from .command import main, doc, version
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\site-packages\pytest_watch\command.py", line 48, in <module>
    from .watcher import watch
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\site-packages\pytest_watch\watcher.py", line 18, in <module>
    from watchdog.observers import Observer
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\site-packages\watchdog\observers\__init__.py", line 88, in <module>
    from .polling import PollingObserver as Observer
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\site-packages\watchdog\observers\polling.py", line 43, in <module>
    from watchdog.observers.api import (
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\site-packages\watchdog\observers\api.py", line 23, in <module>
    from watchdog.utils.bricks import SkipRepeatsQueue
  File "C:\Users\SohangChopra\AppData\Local\Programs\Python\Python310\lib\site-packages\watchdog\utils\bricks.py", line 175, in <module>
    class OrderedSet(collections.MutableSet):
AttributeError: module 'collections' has no attribute 'MutableSet'

opened by sohang3112 3

Runner argument not parsed correctly on windows
Upon running the following command,

$ poetry run poe test-integ-watch Poe => ptw --runner "pytest -m integtest" [Wed Jun 15 10:02:38 2022] Running: "pytest -m integtest" '\"pytest -m integtest\"' is not recognized as an internal or external command, operable program or batch file.

my windows machine produces an argv that is formed as ['"poetry', 'run', 'poe', 'test-integ"'] instead of being formed as ['poetry', 'run', 'poe', 'test-integ']. The line where the watcher fails is shown below.

found at https://github.com/joeyespo/pytest-watch/blob/17375a1cf5be6d4dd30a583ffad61a0083d91b94/pytest_watch/watcher.py#L274

Running the same command on a Mac would have a well-formed list of arguments.

One possible solution would be to clean the arguments before passing them into the constructor, i.e.,

def cleanArgv(argv): return [x.replace('\"', '') for x in argv] argv = cleanArgv(argv) if is_windows else argv # Run tests p = subprocess.Popen(argv, shell=is_windows)
opened by MaxumCSmith 0