a wrapper around pytest for executing tests to look for test flakiness and runtime regression

Overview

bubblewrap

a wrapper around pytest for assessing flakiness and runtime regressions

a cs implementations practice project

How to Run:

First, install depedencies

# from the bubblewrap root
# first, install depdencies:
$ pip3 install -r requirements.txt`
# then, invoke with:
$ ./bubblewrap path/to/code

e.g.

$ ./bubblewrap .

or, if you're using add bubblewrap to your $PATH and call with just bubblewrap path/to/code, e.g.:

$ PATH="$HOME/path/to/bubblewrap:$PATH"

Demos

This package also has a couple quick demos for testing against:

# from the bubblewrap root
$ ./bubblewrap examples/stable
# from the bubblewrap root
$ ./bubblewrap examples/flake

Usage Notes

As of right now, imports in test modules need to import the full path of the module, e.g.:

# this is alright: 
from example.one.A import apple
# but this is currently supported:
from A import apple

This'll be changed in the future.

Also, bubblewrap doesn't yet support regression analysis

Local Setup

This project uses Python3.9, pip to manage dependencies, and runs in a virtual environment. Install with the following:

# create + activate venv + install dependencies
# see docs:  https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/
$ python3 -m pip install --user virtualenv
$ python3 -m venv env
$ source env/bin/activate
$ pip install -r requirements.txt
# note: to deactivate venv, use
$ deactivate

Code Formatting

This project uses Black for code formatting:

$ python3 -m black .

The pyproject.toml file includes configuration for Black.

And, the project is linted with flake8

$ python3 -m flake8 path/to/code

Tests

Actual unit tests for this project are also run with with pytest. Having installed the pip dependencies (see prior section), run with the following:

# unit tests, ignoring ones in the demo dirs that are designed to flake
$ python3 -m pytest -vv -k 'not flake' .
# test coverage report
$ python3 -m coverage run -m pytest -k 'not flake' .
$ coverage report -m
You might also like...
pytest plugin providing a function to check if pytest is running.

pytest-is-running pytest plugin providing a function to check if pytest is running. Installation Install with: python -m pip install pytest-is-running

Pytest-rich - Pytest + rich integration (proof of concept)
Pytest-rich - Pytest + rich integration (proof of concept)

pytest-rich Leverage rich for richer test session output. This plugin is not pub

Local continuous test runner with pytest and watchdog.

pytest-watch -- Continuous pytest runner pytest-watch a zero-config CLI tool that runs pytest, and re-runs it when a file in your project changes. It

pytest plugin for manipulating test data directories and files

pytest-datadir pytest plugin for manipulating test data directories and files. Usage pytest-datadir will look up for a directory with the name of your

Given some test cases, this program automatically queries the oracle and tests your Cshanty compiler!

The Diviner A complement to The Oracle for compilers class. Given some test cases, this program automatically queries the oracle and tests your compil

API Test Automation with Requests and Pytest

api-testing-requests-pytest Install Make sure you have Python 3 installed on your machine. Then: 1.Install pipenv sudo apt-get install pipenv 2.Go to

A set of pytest fixtures to test Flask applications

pytest-flask An extension of pytest test runner which provides a set of useful tools to simplify testing and development of the Flask extensions and a

A set of pytest fixtures to test Flask applications

pytest-flask An extension of pytest test runner which provides a set of useful tools to simplify testing and development of the Flask extensions and a

pytest plugin for a better developer experience when working with the PyTorch test suite

pytest-pytorch What is it? pytest-pytorch is a lightweight pytest-plugin that enhances the developer experience when working with the PyTorch test sui

Owner
Anna Nagy
software development engineer @twitter
Anna Nagy
pytest plugin that let you automate actions and assertions with test metrics reporting executing plain YAML files

pytest-play pytest-play is a codeless, generic, pluggable and extensible automation tool, not necessarily test automation only, based on the fantastic

pytest-dev 67 Dec 1, 2022
A pytest plugin to run an ansible collection's unit tests with pytest.

pytest-ansible-units An experimental pytest plugin to run an ansible collection's unit tests with pytest. Description pytest-ansible-units is a pytest

Community managed Ansible repositories 9 Dec 9, 2022
Selects tests affected by changed files. Continous test runner when used with pytest-watch.

This is a pytest plug-in which automatically selects and re-executes only tests affected by recent changes. How is this possible in dynamic language l

Tibor Arpas 614 Dec 30, 2022
Pytest-typechecker - Pytest plugin to test how type checkers respond to code

pytest-typechecker this is a plugin for pytest that allows you to create tests t

vivax 2 Aug 20, 2022
Thin-wrapper around the mock package for easier use with pytest

pytest-mock This plugin provides a mocker fixture which is a thin-wrapper around the patching API provided by the mock package: import os class UnixF

pytest-dev 1.5k Jan 5, 2023
pytest splinter and selenium integration for anyone interested in browser interaction in tests

Splinter plugin for the pytest runner Install pytest-splinter pip install pytest-splinter Features The plugin provides a set of fixtures to use splin

pytest-dev 238 Nov 14, 2022
:game_die: Pytest plugin to randomly order tests and control random.seed

pytest-randomly Pytest plugin to randomly order tests and control random.seed. Features All of these features are on by default but can be disabled wi

pytest-dev 471 Dec 30, 2022
ApiPy was created for api testing with Python pytest framework which has also requests, assertpy and pytest-html-reporter libraries.

ApiPy was created for api testing with Python pytest framework which has also requests, assertpy and pytest-html-reporter libraries. With this f

Mustafa 1 Jul 11, 2022
The pytest framework makes it easy to write small tests, yet scales to support complex functional testing

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries. An example o

pytest-dev 9.6k Jan 2, 2023
Playwright Python tool practice pytest pytest-bdd screen-play page-object allure cucumber-report

pytest-ui-automatic Playwright Python tool practice pytest pytest-bdd screen-play page-object allure cucumber-report How to run Run tests execute_test

moyu6027 11 Nov 8, 2022