A modern API testing tool for web applications built with Open API and GraphQL specifications.

Overview

Schemathesis

Build codecov.io status for master branch Version Python versions Documentation Status Gitter License

Schemathesis is a modern API testing tool for web applications built with Open API and GraphQL specifications.

It reads the application schema and generates test cases, which will ensure that your application is compliant with its schema.

The application under test could be written in any language; the only thing you need is a valid API schema in a supported format.

Simple to use and yet powerful to uncover hard-to-find errors thanks to the property-based testing approach backed by state-of-the-art Hypothesis library.

πŸ“£ Please fill out our quick survey so that we can learn how satisfied you are with Schemathesis, and what improvements we should make. Thank you!

Features

  • Content-Type, schema, and status code conformance checks for Open API;
  • Testing of explicit examples from the input schema;
  • Stateful testing via Open API links;
  • Concurrent test execution;
  • Targeted testing;
  • Storing and replaying network requests;
  • Built-in ASGI / WSGI application support;
  • Ready-to-go Docker image;
  • Configurable with user-defined checks, string formats, hooks, and targets.

Installation

To install Schemathesis via pip run the following command:

pip install schemathesis

Usage

You can use Schemathesis in the command line:

schemathesis run --stateful=links --checks all http://0.0.0.0:8081/schema.yaml

https://raw.githubusercontent.com/schemathesis/schemathesis/master/img/schemathesis.gif

Or in your Python tests:

import schemathesis

schema = schemathesis.from_uri("http://0.0.0.0:8081/schema.yaml")


@schema.parametrize()
def test_api(case):
    case.call_and_validate()

CLI is simple to use and requires no coding; the in-code approach gives more flexibility.

Both examples above will run hundreds of requests against the API under test and report all found failures and inconsistencies along with instructions to reproduce them.

πŸ’‘ See a complete working example project in the /example directory. πŸ’‘

Contributing

Any contribution to development, testing, or any other area is highly appreciated and useful to the project. For guidance on how to contribute to Schemathesis, see the contributing guidelines.

Support this project

Hi, my name is Dmitry! I started this project during my work at Kiwi.com. I am grateful to them for all the support they provided to this project during its early days and for the opportunity to evolve Schemathesis independently.

In order to grow the community of contributors and users, and allow me to devote more time to this project, please donate today.

Also, I occasionally write posts about Schemathesis in my blog and offer consulting services for businesses.

Commercial support

If you are interested in the effective integration of Schemathesis to your private project, you can schedule an appointment and I will help you do that.

Links

Additional content:

License

The code in this project is licensed under MIT license. By contributing to Schemathesis, you agree that your contributions will be licensed under its MIT license.

Issues
  • Store requests & responses

    Store requests & responses

    Closes #379

    • [x] Add command to cassette
    • [x] Documentation
    • [x] Reuse prepared request to avoid double computation
    • [x] CLI output about saved cassette file
    opened by Stranger6667 26
  • Add hooks to modify request before request is sent

    Add hooks to modify request before request is sent

    Description

    Schemathesis currently has hooks that allow users to extend the set of Schemathesis validation functions. I would like to be able to write hooks that allow me to alter the request before it is sent to the API. This will allow me to test specific behavior of the API.

    Example of one extended validation rule:

    Accept header MUST be used to indicate allowable request methods in a 405 response.
    

    To exercise this validation rule, we need to send a request with an operation that we know is not in the API definition for that endpoint. This idea is mentioned in #108, so this behavior may be included as part of the broader goal to generate invalid data in #65.

    However, some of the rules are likely too specific to be part of the default request generation. For example:

    Requests including Accept header with only formats not supported by the server MUST be
    rejected with a 406 status code and appropriate error response model.
    

    Design

    The validation hooks are simple and easy to use, and I think the request hooks should use the same design with different parameters. I think the should request hooks should take a request object and a case object (or any object that contains the API description for the endpoint associated with the request).

    Questions to Consider

    • Does mutating an initially valid request cause problems?
    • Do the extended validation rules still run when the status code is not in the API definition? 415 response, for example, may not be documented in the OpenAPI description.
    • Which requests do we modify? We only want to modify a subset of the requests. Maybe we generate a valid request for each hook in addition to the default requests. Each additional valid request will be modified by a single hook.
    Priority: Medium Type: Feature 
    opened by barrett-schonefeld 21
  • validation of response schema not working for recursive schemas

    validation of response schema not working for recursive schemas

    In response_schema_conformance (in checks.py) schemathesis is validating the response schema after it has tried to resolve it. If the response schema is recursive this doesnt work. It would be better to directly pass the response schema to jsonschema.validate() instead of the resolved one.

    if i replace schema = case.endpoint.schema._get_response_schema(definition) with schema = responses[str(response.status_code)]['content'][content_type] (will fail if response has no content type) the validation works fine.

    since jsonschema.validate() can handle recursive schemas, this might also be a better fix for https://github.com/kiwicom/schemathesis/issues/435, validate the raw schema and then resolve it as much as needed to safe time wasted in unnecessary recursions.

    PS: just wanted to say thank you for schemathesis and not just keep opening new issues. I t has really helped us ;)

    Priority: Medium Type: Bug 
    opened by jeremyschiemann 15
  • Support multiple examples specified in Open API schema

    Support multiple examples specified in Open API schema

    As mentioned in #582 we can generate multiple examples via the examples keyword:

    parameters:
      - in: query
        name: limit
        schema:
          type: integer
          maximum: 50
        examples:       # Multiple examples
          zero:         # Distinct name
            value: 0    # Example value
            summary: A sample limit value  # Optional description
          max: # Distinct name
            value: 50   # Example value
            summary: A sample limit value
    
    Priority: Medium Type: Enhancement 
    opened by Stranger6667 14
  • Validate error for nullable properties since v1.3.0

    Validate error for nullable properties since v1.3.0

    Hi, I have schemathesis==1.3.3 and I have the same issue as #335:

    Details: 
    
    None is not of type 'string'
    
    Failed validating 'type' in schema['items']['properties']['family_name']:
        {'maxLength': 50,
         'nullable': True,
         'title': 'Family Name',
         'type': 'string'}
    
    On instance[0]['family_name']:
        None
    

    I tested older versions, This is working in v1.2.0, but regressed since v1.3.0

    Priority: High Type: Bug 
    opened by grigi 14
  • .call method does not support the

    .call method does not support the "verify" argument.

    response = case.call(BASE_URL, verify=False, timeout=1000)

    does not take into consideration the verify=False argument.

    Priority: High Type: Bug 
    opened by xtakacsx 14
  • Response code range not supported

    Response code range not supported

    Thanks for adding check of response code!

    Response codes may contain a range of code in the format XXX https://swagger.io/specification/#responsesObject

    My output

            if errors:
    >           raise AssertionError(*errors)
    E           AssertionError: Received a response with a status code, which is not defined in the schema: 404
    E           
    E           Declared status codes: 4XX, 200
    
    Priority: High Type: Feature 
    opened by MissiaL 13
  • [BUG] Docker image can't connect to localhost

    [BUG] Docker image can't connect to localhost

    Describe the bug According to the Pypi docs, we should be able to run the schemathesis Docker image vs a localhost api. When I use schemathesis from the command line everything works as expected, but when using kiwicom/schemathesis:stable even with --networks="host" it refuses to connect.

    To Reproduce Run the api code as described in the README here

    docker run --network="host" kiwicom/schemathesis:stable run http://localhost:5000/swagger.json
    Failed to load schema from http://localhost:5000/swagger.json
    Error: requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /swagger.json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc8d77eeb50>: Failed to establish a new connection: [Errno 111] Connection refused'))
    
    Add this option to your command line parameters to see full tracebacks: --show-errors-tracebacks
    

    Full traceback is linked here

    If possible, please post a minimal version of your API schema that cause this behavior: n/a

    Expected behavior Docker container connects to localhost and executes Schemathesis tests

    Environment (please complete the following information):

    • OS: MacOS Catalina 10.15.7
    • Python version: 3.8.2
    • Schemathesis version: stable
    • Spec version: Open API 3.0.2
    Type: Bug 
    opened by GvandeSteeg 12
  • Allowing RequestBodies for GET Requests

    Allowing RequestBodies for GET Requests

    Hi,

    Is there any plan for the schemathesis library to allow for Request Bodies in GET requests? I know this is against the specifications of OpenApi, but many Get requests may require request bodies (e.g. ElasticSearch).

    Really enjoy this library! I think it would be very beneficial to have this option.

    Thank you!

    Priority: Medium Type: Enhancement 
    opened by charliehohenstein30 12
  • Using $ref causes schema parsing error

    Using $ref causes schema parsing error

    I'm using Open api v3.

    The issue occurs both using yaml and json representation of the spec.

    My yaml:

    openapi: "3.0.0"
    info:
      description: |
        Some description
      version: "1.0.0"
      title: "This yaml is causing issues"
    paths:
      "/s/stats/exchange":
        get:
          summary: "Gets currency exchange rates"
          operationId: "getCurrencyExchangeRates"
          responses:
            "200":
              description: "A JSON object with currency tag as key and rate as value"
              content:
                application/json:
                  schema:
                    type: object
                    additionalProperties:
                      $ref: '#/components/schemas/CurrencyCode'
                    
    components:
      schemas:
        CurrencyCode:
          type: string
          minLength: 3
          maxLength: 3
    
    

    I use swagger editor webpage to export my yaml into the json file and afterwards trying to test it by running command:

    schemathesis run ./swagger.yaml --base-url <base url goes here>

    And i receive an error:

    Traceback (most recent call last):
      File "c:\python38\lib\site-packages\jsonschema\validators.py", line 774, in resolve_from_url
        document = self.store[url]
      File "c:\python38\lib\site-packages\jsonschema\_utils.py", line 22, in __getitem__
        return self.store[self.normalize(uri)]
    KeyError: ''
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "c:\python38\lib\site-packages\jsonschema\validators.py", line 777, in resolve_from_url
        document = self.resolve_remote(url)
      File "c:\python38\lib\site-packages\jsonschema\validators.py", line 856, in resolve_remote
        result = self.handlers[scheme](uri)
      File "c:\python38\lib\site-packages\schemathesis\schemas.py", line 36, in load_file
        with open(location) as fd:
    FileNotFoundError: [Errno 2] No such file or directory: ''
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "c:\python38\lib\site-packages\schemathesis\schemas.py", line 190, in get_all_endpoints
        methods = self.resolve(methods)
      File "c:\python38\lib\site-packages\schemathesis\schemas.py", line 288, in resolve
        item[key] = self.resolve(sub_item)
      File "c:\python38\lib\site-packages\schemathesis\schemas.py", line 288, in resolve
        item[key] = self.resolve(sub_item)
      File "c:\python38\lib\site-packages\schemathesis\schemas.py", line 288, in resolve
        item[key] = self.resolve(sub_item)
      [Previous line repeated 4 more times]
      File "c:\python38\lib\site-packages\schemathesis\schemas.py", line 285, in resolve
        with self.resolver.resolving(item["$ref"]) as resolved:
      File "c:\python38\lib\contextlib.py", line 113, in __enter__
        return next(self.gen)
      File "c:\python38\lib\site-packages\jsonschema\validators.py", line 754, in resolving
        url, resolved = self.resolve(ref)
      File "c:\python38\lib\site-packages\jsonschema\validators.py", line 766, in resolve
        return url, self._remote_cache(url)
      File "c:\python38\lib\site-packages\jsonschema\validators.py", line 779, in resolve_from_url
        raise exceptions.RefResolutionError(exc)
    jsonschema.exceptions.RefResolutionError: [Errno 2] No such file or directory: ''
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "c:\python38\lib\runpy.py", line 193, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "c:\python38\lib\runpy.py", line 86, in _run_code
        exec(code, run_globals)
      File "C:\Python38\Scripts\schemathesis.exe\__main__.py", line 7, in <module>
      File "c:\python38\lib\site-packages\click\core.py", line 764, in __call__
        return self.main(*args, **kwargs)
      File "c:\python38\lib\site-packages\click\core.py", line 717, in main
        rv = self.invoke(ctx)
      File "c:\python38\lib\site-packages\click\core.py", line 1137, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "c:\python38\lib\site-packages\click\core.py", line 956, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "c:\python38\lib\site-packages\click\core.py", line 555, in invoke
        return callback(*args, **kwargs)
      File "c:\python38\lib\site-packages\schemathesis\cli\__init__.py", line 231, in run
        execute(prepared_runner, workers_num, show_errors_tracebacks)
      File "c:\python38\lib\site-packages\schemathesis\cli\__init__.py", line 285, in execute
        handler(context, event)
      File "c:\python38\lib\site-packages\schemathesis\cli\output\default.py", line 318, in handle_event
        handle_initialized(context, event)
      File "c:\python38\lib\site-packages\schemathesis\cli\output\default.py", line 279, in handle_initialized
        click.secho(f"collected endpoints: {event.schema.endpoints_count}", bold=True)
      File "c:\python38\lib\site-packages\schemathesis\schemas.py", line 86, in endpoints_count
        return len(list(self.get_all_endpoints()))
      File "c:\python38\lib\site-packages\schemathesis\schemas.py", line 202, in get_all_endpoints
        raise InvalidSchema("Schema parsing failed. Please check your schema.")
    schemathesis.exceptions.InvalidSchema: Schema parsing failed. Please check your schema.
    
    

    The moment i don't use $ref and put the type directly where the response is, the issue is solved

    opened by ArinCantCode 12
  • [BUG] @app.on_event(

    [BUG] @app.on_event("startup") not triggered when using schemathesis.from_asgi() for fastapi==0.70.0

    Checklist

    Describe the bug When using schemathesis.from_asgi(app, "/openapi.json") the @app.on_event("startup") in our init_app function is not triggered for fastapi==0.70.0.

    We have also followed the example for lazy loading, but we get the same problem.

    To Reproduce In server.py

    #!/usr/bin/env python3
    import uvicorn
    from fastapi import FastAPI, status
    
    
    def init_app():
        app = FastAPI()
    
        @app.on_event("startup")
        async def startup():
            raise RuntimeError
    
        @app.get("/health", status_code=status.HTTP_200_OK)
        async def healthcheck():
            return {"status": "ok"}
    
        return app
    
    
    app = init_app()
    
    if __name__ == "__main__":
        uvicorn.run("server:app", port=8000)
    
    

    And our tests

    import schemathesis
    
    from server import init_app
    
    app = init_app()
    
    schema = schemathesis.from_asgi("/openapi.json", app)
    
    
    @schema.parametrize()
    def test_openapi_specification(case):
        response = case.call_asgi()
        case.validate_response(response)
    

    And we tried initializing according to the lazy load example.

    import pytest
    import schemathesis
    
    from server import init_app
    
    @pytest.fixture
    def web_app():
        app = init_app()
    
        @app.on_event("startup")
        async def startup():
            raise RuntimeError
        return schemathesis.from_asgi("/openapi.json", app)
    
    
    schema = schemathesis.from_pytest_fixture("web_app")
    
    
    @schema.parametrize()
    def test_openapi_specification(case):
        response = case.call_asgi()
        case.validate_response(response)
    

    where both tests succeed when they shouldn't.

    Expected behavior We expect RuntimeError to be raised.

    Environment (please complete the following information):

    • OS: macOS 11.6, Manjaro, kernel: 5.10.70 and Arch Linux, kernel 5.10.74-1-lts
    • Python version: 3.9.7
    • Schemathesis version: 3.10.1
    • Spec version: Open API 3.0.2
    • FastAPI version: 0.70.0

    Additional context We have also tried using fastapi>=0.68.2 with the same result.

    Priority: Medium Type: Bug Status: Need More Info 
    opened by jschennings 3
  • [FEATURE] Relax requirements on schema validity

    [FEATURE] Relax requirements on schema validity

    Is your feature request related to a problem? Please describe. It often confuses people when they run Schemathesis on a seemingly valid schema. We might demote the failure-by-default behavior to a warning, and or even disable schema validation by default

    Also, we can improve the docs on this matter as well

    Status: Review Needed Type: Feature 
    opened by Stranger6667 0
  • [FEATURE] Generate headers for debugging

    [FEATURE] Generate headers for debugging

    Is your feature request related to a problem? Please describe. When the user debugs the test, it might be beneficial to have a separate unique request-id, that Schemathesis will send with each request.

    Describe the solution you'd like Generate X-Schemathesis-Id inside each call call

    Additional context Comes from the discussion with @ludeknovy

    Status: Review Needed Type: Feature 
    opened by Stranger6667 0
  • Feature/966 refresh auth

    Feature/966 refresh auth

    🚨 DRAFT 🚨 Before I finalise everything and diving deeper into existing project logic, please do a high level review for coding style and feature solving.

    Black and Linting are not included yet. It should be just a first review through pure functionality and logical project anti pattern if possible.

    🚨Please review the guidelines for contributing to this repository.

    Description

    Please explain the changes you made here.

    Checklist

    • [ ] Created tests which fail without the change (if possible)
    • [ ] All tests passing
    • [ ] Added a changelog entry
    • [ ] Extended the README / documentation, if necessary
    opened by JangasCodingplace 7
  • [FEATURE] Support for `jsonschema==4.0`

    [FEATURE] Support for `jsonschema==4.0`

    To support Open API 3.1 we first need to bring the new jsonschema & hypothesis-jsonschema versions into the repo.

    Update the dependencies & fix broken tests :)

    Priority: Medium Type: Enhancement Hacktoberfest Difficulty: Medium 
    opened by Stranger6667 0
  • [FEATURE] Add Python 3.10 support

    [FEATURE] Add Python 3.10 support

    We need:

    • Trove classifier in the pyproject.toml file
    • Builds
    Priority: Medium Type: Maintenance Hacktoberfest Difficulty: Easy 
    opened by Stranger6667 2
  • [FEATURE] Distinguish more errors for Hypothesis' shrinking logic

    [FEATURE] Distinguish more errors for Hypothesis' shrinking logic

    Ideally, Schemathesis would produce a single minimal example of each distinct error. We use Hypothesis' shrinking for this, which requires us to distinguish errors by exception type, code location, and __context__; and so Schemathesis creates new exception types for each distinct error message:

    https://github.com/schemathesis/schemathesis/blob/7e71b00f285139c79d96d2dfb5cb16f6463e2544/src/schemathesis/exceptions.py#L43-L47

    Collapsing errors by message seems fine in cases like {expected_contenttype}_{got_contenttype}, but collapsing all HTTP 500 errors together because they share a status code probably causes under-counting.

    Re-running on the fixed version will re-find the remaining errors, so that's not a disaster, but it would be a nicer UX (and paper result) to report them all in the first place.

    Possible fixes

    • If we can access the original exception when using the ASGI/WSGI integration, raise schemathesis_exc from the_inner_exc is sufficient. (and perhaps e.g. enable Flask's debug mode to ensure such exceptions are propagated)

    • Maybe we could distinguish HTTP 5xx by message as well status code, if we removed the dynamic response content? This risks not-deduplicating though, which I don't like.

    • Dynamically create a "from" exception using locations and messages from stringified stack traces and/or locations in responses, for example from Jupyter Server & Jupyterhub, and then raise from that. This seems promising to me, and reasonably easy to implement on a per-programming-language basis.

    Priority: Medium Type: Enhancement Difficulty: Medium 
    opened by Zac-HD 1
  • Docker container and cmd options

    Docker container and cmd options

    I just found that Schemathesis has a Docker version too, which is great. I can fuzz with Schemathesis using Docker, hassle-free :-)

    A few questions,

    1. Do options used in CLI work for the Docker version, for example, if I have a local openapi.yaml, how can I pass it into the Docker container to start fuzzing it?

    docker run schemathesis/schemathesis:stable run <__local open api??__>

    1. How can I pass in the JWT token to the Docker container that runs Schemathesis?
    2. How can I pass the fixed body UUIDs?
    3. How can I pass data_generation_methods option: data_generation_methods=[DataGenerationMethod.negative, DataGenerationMethod.positive]
    4. How can I pass base_url?
    Status: Review Needed Type: Feature 
    opened by mbiuki 2
  • [FEATURE] Avoid parsing all endpoints at once in multi-worker execution in CLI

    [FEATURE] Avoid parsing all endpoints at once in multi-worker execution in CLI

    Is your feature request related to a problem? Please describe. When there are multiple workers specified for CLI, then it will parse all API operations at once and submit them to the task queue. If the API schema is big, then it will take a lot of memory

    Describe the solution you'd like Generate it on demand, when workers are ready to take the next test

    Priority: Medium Type: Enhancement Difficulty: Medium 
    opened by Stranger6667 0
  • [FEATURE] Limit unicode categories used in generated strings

    [FEATURE] Limit unicode categories used in generated strings

    Is your feature request related to a problem? Please describe. When Schemathesis generates string data, it might contain a lot of non-ASCII chars that might not be much interesting for the end-user. E.g. often people ask how to exclude \x00 because those failures are not interesting for them.

    Describe the solution you'd like A public API that will allow the user to define what chars are used for strings (globally & per schema). Could be similar to register_string_format

    Currently, the format keyword could be abused for this purpose

    Priority: Low Type: Feature Difficulty: Medium 
    opened by Stranger6667 0
Releases(v3.10.1)
  • v3.10.1(Oct 4, 2021)

    :bug: Bug fixes

    • fcf6ef5 Unresolvable dependency due to incompatible changes in the new hypothesis-jsonschema release

    :wrench: Chores and Improvements

    • 49052ff Release 3.10.1
    • 975264d Update pre-commit
    • 76f66d7 Move data generation method to strategy factory mapping to the module level
    • 30d03c6 DataGenerationMethod.all shortcut to get all possible enum variants.

    :package: Other

    • 6edbae5 Document CLI options & add a recipe for HTTP(S) proxy
    • d1d47f3 Clarify communication language
    • f2f08d6 Disable deadline on flaky test
    • f0711b1 Use poetry<1.1.9
    Source code(tar.gz)
    Source code(zip)
  • v3.10.0(Sep 13, 2021)

    :rocket: Features

    • 2903305 New before_init_operation hook
    • f136a16 (internal) description attribute for all parsed parameters inside APIOperation.
    • f4576bd Schemathesis.io integration

    :bug: Bug fixes

    • cf63173 (example) Docker setup for example project
    • a80424b (example) Typo in example project's schema
    • df14316 Properly report all failures when custom checks are passed to case.validate_response
    • 54cce04 Stop worker threads on failures with exit_first enabled
    • b1172a1 Rewrite not resolved remote references to local ones
    • 167b8a9 Re-used referenced objects during inlining
    • 471cebd Stop workers on interrupted event (#1239)
    • c8179b9 TypeError on case.call with bytes data on GraphQL schemas
    • 6168c22 Internal error in make_case calls for GraphQL schemas.

    :wrench: Chores and Improvements

    • c7527af Release 3.10.0
    • 3f04815 Improve error messages when the loaded API schema is not in JSON or YAML
    • 576dba4 Improve error messages for Schemathesis.io
    • 84473eb Add an extra message in case of errors with Schemathesis.io
    • 669755c Show Schemathesis.io test report URL
    • 5780fd2 Open API schema loaders now accept single DataGenerationMethod instances for the data_generation_methods argument
    • 608216f Relax colorama and typing-extensions
    • e14df88 Add tool.poetry.urls section
    • 829fcb9 Reduce Docker images size
    • b6d75cd Add timeouts when loading external schema components or external examples
    • 2418188 Do not use header filtration when not necessary
    • affc9a8 Minor refactoring of reference inlining
    • e75345c Change OpenAPI20CompositeBody.definition type to List[OpenAPI20Parameter]
    • fd86bad Pin werkzeug to >=0.16.0
    • f196825 Remove unused parameter_cls attributes from BaseOpenAPISchema
    • 72a946c Remove duplicated keyword

    :package: Other

    • 7530ff0 Minor changelog correction
    • 212ef23 Update Service docs
    • e2c7a5f Add a link to a walkthrough
    • 97b47b5 Add tests for schema mutations
    • 822b084 Add extra tests for is_valid_header
    • 5980df1 Add missing changelog entry
    • e5178f3 Disable some health checks for slow tests
    • d7e90c0 Add more tests
    • 3080028 Fix typo in help message for --header
    • 975fb44 Fix Service tests on Windows
    Source code(tar.gz)
    Source code(zip)
  • v3.9.7(Jul 26, 2021)

    :rocket: Features

    • 3a730d0 New process_call_kwargs CLI hook

    :wrench: Chores and Improvements

    • 3b8ee7f Release 3.9.7
    • 7c50df3 Check non-string response status codes when Open API links are collected
    Source code(tar.gz)
    Source code(zip)
  • v3.9.6(Jul 15, 2021)

  • v3.9.5(Jul 14, 2021)

    :bug: Bug fixes

    • 0bec4b3 Preserve non-body parameter types in requests during Open API runtime expression evaluation

    :wrench: Chores and Improvements

    • d4560b4 Release 3.9.5
    Source code(tar.gz)
    Source code(zip)
  • v3.9.4(Jul 9, 2021)

    :bug: Bug fixes

    • 9395a74 TypeError during negative testing on Open API schemas with parameters that have non-default style value
    • 424dfa2 KeyError when the response_schema_conformance check is executed against responses without schema definition

    :wrench: Chores and Improvements

    • 4700a6c Release 3.9.4
    • e16d42a Remove unused imports

    :package: Other

    • 4a217d8 Use result.verbose_name unconditionally in display_subsection
    • b0ca226 Use cls in class methods instead of concrete classes
    Source code(tar.gz)
    Source code(zip)
  • v3.9.3(Jun 22, 2021)

    :rocket: Features

    • 36829e2 Add ExecutionEvent.is_terminal attribute that indicates whether an event is the last one in the stream

    :bug: Bug fixes

    • 703430c When EventStream.stop is called, the next event always is the last one

    :wrench: Chores and Improvements

    • 5e18c2b Release 3.9.3

    :package: Other

    • 47813ef Improve the way internal events are handled in CLI
    Source code(tar.gz)
    Source code(zip)
  • v3.9.2(Jun 16, 2021)

    :bug: Bug fixes

    • 4dbf750 Incorrect deduplication applied to response schema conformance failures that happen to have the same failing validator but different input values

    :wrench: Chores and Improvements

    • bbcbd92 Release 3.9.2
    • 9a6cc1d Return response from case.call_and_validate
    Source code(tar.gz)
    Source code(zip)
  • v3.9.1(Jun 13, 2021)

    :wrench: Chores and Improvements

    • be2f772 Release 3.9.1
    • 7ed2306 Convert all FailureContext class attributes to instance attributes
    • b6b3c59 Add SerializedCase.cookies
    • 01064a7 Add API schema to the Initialized event
    • 6fad51c ExecutionEvent.asdict adds the event_type field which is the event class name
    Source code(tar.gz)
    Source code(zip)
  • v3.9.0(Jun 7, 2021)

    :rocket: Features

    • b8d4484 A way to stop the Schemathesis runner's event stream manually via events.stop()
    • c76285c GraphQL support in CLI

    :wrench: Chores and Improvements

    • 6256be0 Release 3.9.0
    • a9fbccd Add EventStream.finish method
    • 250c20e Avoid ``pytest`warnings when internal Schemathesis classes are in the test module scope

    :package: Other

    • 3d5d093 Update GraphQL docs
    • 9fcad07 Fix RST syntax
    Source code(tar.gz)
    Source code(zip)
  • v3.8.0(Jun 3, 2021)

    :rocket: Features

    • e922209 Case.data_generation_method attribute that provides the information of the underlying data generation method
    • 9217358 Negative testing

    :bug: Bug fixes

    • cd91522 Compatibility with newer Hypothesis versions

    :wrench: Chores and Improvements

    • 7d8c0be Release 3.8.0
    • 622a1c9 Improve APIOperation.make_case behavior
    • 4d08fd2 Bump hypothesis-jsonschema version
    • 37b1000 Python values of True, False and None are converted to their JSON equivalents when generated for path parameters or query
    • 0b90566 Raise UsageError if schema.parametrize or schema.given are applied to the same function more than once

    :package: Other

    • 12b0049 Disable deadline for occasionally slow tests
    Source code(tar.gz)
    Source code(zip)
  • v3.7.8(Jun 2, 2021)

    :bug: Bug fixes

    • f9fa796 Open API style & explode for parameters derived from security definitions

    :wrench: Chores and Improvements

    • d30e9c0 Release 3.7.8
    • a6112dc (deps) Update dependencies in /example
    Source code(tar.gz)
    Source code(zip)
  • v3.7.7(Jun 1, 2021)

    :bug: Bug fixes

    • 6e72c3f Apply the Open API's style & explode keywords to explicit examples

    :wrench: Chores and Improvements

    • 8411718 Release 3.7.7

    :package: Other

    • 5e46a7d Simplify examples serialization
    Source code(tar.gz)
    Source code(zip)
  • v3.7.6(May 31, 2021)

    :bug: Bug fixes

    • 586ba77 Disable filtering optimization for headers when there are keywords other than type

    :wrench: Chores and Improvements

    • 0daefff Release 3.7.6
    Source code(tar.gz)
    Source code(zip)
  • v3.7.5(May 31, 2021)

    :bug: Bug fixes

    • 31d322f Too much filtering in headers that have schemas with the pattern keyword

    :wrench: Chores and Improvements

    • 909205d Release 3.7.5
    Source code(tar.gz)
    Source code(zip)
  • v3.7.4(May 28, 2021)

    :wrench: Chores and Improvements

    • 996bc8e Release 3.7.4
    • bf894c8 Make SerializedCase.path_template return path templates as they are in the schema, without base path
    Source code(tar.gz)
    Source code(zip)
  • v3.7.3(May 28, 2021)

    :bug: Bug fixes

    • 288bf17 Invalid multipart payload generated for unusual schemas for the multipart/form-data media type

    :wrench: Chores and Improvements

    • 6dcce2e Release 3.7.3

    :package: Other

    • 00d3796 Improve schemas to avoid unneeded filtering
    • bd0cc79 Reduce the amount of filtering needed to generate valid headers and cookies
    Source code(tar.gz)
    Source code(zip)
  • v3.7.2(May 27, 2021)

    :bug: Bug fixes

    • 1b8da90 Ignored explicit Content-Type override in Case.as_requests_kwargs
    • bb6fe04 Internal error on unusual schemas for the multipart/form-data media type

    :wrench: Chores and Improvements

    • 04b24b4 Release 3.7.2
    • bd066f0 Add SerializedCase.media_type attribute
    Source code(tar.gz)
    Source code(zip)
  • v3.7.1(May 23, 2021)

  • v3.7.0(May 23, 2021)

    :rocket: Features

    • 1cb291b Additional context for each failure coming from the runner.

    :wrench: Chores and Improvements

    • bc3c08b Release 3.7.0
    • 06ecf49 Extend the default test duration deadline to 15 seconds
    • 6dd1586 Set default timeout for network requests to 10 seconds.
    • 7f26e1f Use different exception classes for not_a_server_error and status_code_conformance checks.

    :package: Other

    • c5e5770 Clarify scope for the add_case hook
    Source code(tar.gz)
    Source code(zip)
  • v3.6.11(May 20, 2021)

    :rocket: Features

    • d3e1766 (runner) Add BeforeExecution.verbose_name & SerializedCase.verbose_name that reflect specification-specific API operation name

    :wrench: Chores and Improvements

    • af7cde3 Release 3.6.11

    :package: Other

    • bb7a8a3 Remove unused fixtures
    Source code(tar.gz)
    Source code(zip)
  • v3.6.10(May 17, 2021)

    :wrench: Chores and Improvements

    • c3339d7 Release 3.6.10
    • d7a0e9e Bump hypothesis-jsonschema version
    • 844655a Explicitly add colorama to dependencies
    • b1d45b8 Fix hypothesis-jsonschema version constraint

    :package: Other

    • efd1969 Fix corpus tests
    Source code(tar.gz)
    Source code(zip)
  • v3.6.9(May 14, 2021)

    :bug: Bug fixes

    • acd2ac4 Ignored Open API specific keywords & types in schemas with deeply nested references
    • dbd4cfd Ignored $ref keyword in schemas with deeply nested references.

    :wrench: Chores and Improvements

    • 86b396b Release 3.6.9
    Source code(tar.gz)
    Source code(zip)
  • v3.6.8(May 13, 2021)

    :wrench: Chores and Improvements

    • ce92e34 Release 3.6.8
    • 0779b97 Relax dependency on starlette to >=0.13,<1

    :package: Other

    • 42a2113 Fix insufficient validation in test apps
    Source code(tar.gz)
    Source code(zip)
  • v3.6.7(May 12, 2021)

    :bug: Bug fixes

    • 46ec17d Missing support for date string format

    :wrench: Chores and Improvements

    • 9297654 Release 3.6.7
    • 8ca014c Update pre-commit hooks
    • 59840e1 Call shutdown on all handlers before CLI exit

    :package: Other

    • a371c3b Update notes on stateful testing in CLI
    Source code(tar.gz)
    Source code(zip)
  • v3.6.6(May 7, 2021)

    :wrench: Chores and Improvements

    • a7d2bf1 Release 3.6.6
    • 667d8b7 Improve error message for failing Hypothesis deadline healthcheck in CLI

    :package: Other

    • 88818f6 Disable deadline in flaky tests
    Source code(tar.gz)
    Source code(zip)
  • v3.6.5(May 7, 2021)

    :rocket: Features

    • b78a164 Support for disabling ANSI color escape codes via the NO_COLOR environment variable or the --no-color CLI option

    :wrench: Chores and Improvements

    • b8bed68 Release 3.6.5
    • e362292 Generate valid header values for Bearer auth by construction rather than by filtering
    • d414477 (deps) bump urllib3 from 1.25.6 to 1.25.8 in /example

    :package: Other

    • 4e2114c Avoid flakiness in test_path_parameters_encoding
    Source code(tar.gz)
    Source code(zip)
  • v3.6.4(Apr 30, 2021)

    :wrench: Chores and Improvements

    • 9d9255b Release 3.6.4
    • e85a13d Bump minimum hypothesis-graphql version to 0.5.0
    • aa632df Remove obsolete pylint and mypy tox envs
    Source code(tar.gz)
    Source code(zip)
  • v3.6.3(Apr 20, 2021)

    :bug: Bug fixes

    • 5841dda UnicodeEncodeError when sending application/octet-stream payloads that has no format: binary in their schemas
    • 5098a1d Bump minimum hypothesis-graphql version to 0.4.1

    :wrench: Chores and Improvements

    • 1050b81 Release 3.6.3
    • 95ec3ef (deps) bump asyncpg from 0.19.0 to 0.21.0 in /example
    Source code(tar.gz)
    Source code(zip)
  • v3.6.2(Apr 15, 2021)

    :bug: Bug fixes

    • f8f571b (windows) UnicodeDecodeError during schema loading via the from_path loader if it contains certain Unicode symbols

    :wrench: Chores and Improvements

    • d786d33 Release 3.6.2
    • 6d7ddad Remove unused imports

    :package: Other

    • 9382d71 Pin the sphinx_rtd_theme version
    • a2b0de3 Fix argument order in hook examples
    Source code(tar.gz)
    Source code(zip)
Owner
Schemathesis.io
A tool that generates and runs test cases for Open API / GraphQL based apps
Schemathesis.io
A modern API testing tool for web applications built with Open API and GraphQL specifications.

Schemathesis Schemathesis is a modern API testing tool for web applications built with Open API and GraphQL specifications. It reads the application s

Schemathesis.io 1k Oct 18, 2021
βœ… Python web automation and testing. πŸš€ Fast, easy, reliable. πŸ’ 

Build fast, reliable, end-to-end tests. SeleniumBase is a Python framework for web automation, end-to-end testing, and more. Tests are run with "pytes

SeleniumBase 2.4k Oct 24, 2021
Language-agnostic HTTP API Testing Tool

Dredd β€” HTTP API Testing Framework Dredd is a language-agnostic command-line tool for validating API description document against backend implementati

Apiary 3.7k Oct 17, 2021
splinter - python test framework for web applications

splinter - python tool for testing web applications splinter is an open source tool for testing web applications using Python. It lets you automate br

Cobra Team 2.4k Oct 17, 2021
splinter - python test framework for web applications

splinter - python tool for testing web applications splinter is an open source tool for testing web applications using Python. It lets you automate br

Cobra Team 2.3k Feb 5, 2021
WEB PENETRATION TESTING TOOL πŸ’₯

N-WEB ADVANCE WEB PENETRATION TESTING TOOL Features ?? Admin Panel Finder Admin Scanner Dork Generator Advance Dork Finder Extract Links No Redirect H

null 41 Sep 24, 2021
pytest plugin that let you automate actions and assertions with test metrics reporting executing plain YAML files

pytest-play pytest-play is a codeless, generic, pluggable and extensible automation tool, not necessarily test automation only, based on the fantastic

pytest-dev 64 Sep 13, 2021
A single module to link Python ecosystem to the Web

A single module to link Python ecosystem to the Web. Have a quick look at the Gallery first to get convinced ! FAQ For any questions, please use Stack

null 56 Oct 23, 2021
A testing system for catching visual regressions in Web applications.

Huxley Watches you browse, takes screenshots, tells you when they change Huxley is a test-like system for catching visual regressions in Web applicati

Facebook Archive 4.1k Oct 14, 2021
A command-line tool and Python library and Pytest plugin for automated testing of RESTful APIs, with a simple, concise and flexible YAML-based syntax

1.0 Release See here for details about breaking changes with the upcoming 1.0 release: https://github.com/taverntesting/tavern/issues/495 Easier API t

null 820 Oct 3, 2021
pytest plugin for distributed testing and loop-on-failures testing modes.

xdist: pytest distributed testing plugin The pytest-xdist plugin extends pytest with some unique test execution modes: test run parallelization: if yo

pytest-dev 778 Oct 23, 2021
Sixpack is a language-agnostic a/b-testing framework

Sixpack Sixpack is a framework to enable A/B testing across multiple programming languages. It does this by exposing a simple API for client libraries

null 1.7k Oct 11, 2021
Sixpack is a language-agnostic a/b-testing framework

Sixpack Sixpack is a framework to enable A/B testing across multiple programming languages. It does this by exposing a simple API for client libraries

null 1.7k Oct 11, 2021
An improbable web debugger through WebSockets

wdb - Web Debugger Description wdb is a full featured web debugger based on a client-server architecture. The wdb server which is responsible of manag

Kozea 1.5k Oct 12, 2021
Green is a clean, colorful, fast python test runner.

Green -- A clean, colorful, fast python test runner. Features Clean - Low redundancy in output. Result statistics for each test is vertically aligned.

Nathan Stocks 709 Oct 20, 2021
Test django schema and data migrations, including migrations' order and best practices.

django-test-migrations Features Allows to test django schema and data migrations Allows to test both forward and rollback migrations Allows to test th

wemake.services 247 Oct 15, 2021
Useful additions to Django's default TestCase

django-test-plus Useful additions to Django's default TestCase from REVSYS Rationale Let's face it, writing tests isn't always fun. Part of the reason

REVSYS 485 Oct 14, 2021
Useful additions to Django's default TestCase

django-test-plus Useful additions to Django's default TestCase from REVSYS Rationale Let's face it, writing tests isn't always fun. Part of the reason

REVSYS 485 Oct 14, 2021
Python 3 wrapper of Microsoft UIAutomation. Support UIAutomation for MFC, WindowsForm, WPF, Modern UI(Metro UI), Qt, IE, Firefox, Chrome ...

Python 3 wrapper of Microsoft UIAutomation. Support UIAutomation for MFC, WindowsForm, WPF, Modern UI(Metro UI), Qt, IE, Firefox, Chrome ...

yin kaisheng 1.1k Oct 15, 2021