A command-line tool and Python library and Pytest plugin for automated testing of RESTful APIs, with a simple, concise and flexible YAML-based syntax

Hello gang!

I believe that there might be an issue reading integers from one file when they've been included in another.

For instance: foo.yaml: owner-id: 1001

bar.yaml: !include: foo.yaml owner-id: {owner-id} --or owner-id: "{owner-id:d}"

So far, I haven't been able to get this to return an int to the calling document. If explicitly casting to an int:

bar.yaml: !include: foo.yaml owner-id: !!int "{owner-id:d}"

The following error is produced: ValueError: invalid literal for int() with base 10: '{testid' because the cast is happening before the variable is returned.

If I have an incorrect implementation somewhere, please let me know! Thanks!

Fix for #753 where I bound all the available MQTT callbacks that would be useful to a developer facing connection issues. There are still a few callbacks that could be bound, but I haven't ever needed them.

Ran the code formatter and smoke tests on the code. Unit tests seem to have unrelated failures, and there is only calls to the logger in this. If there is a good way you wanted to unit test the new callbacks, let me know.

I am trying to create a parameterized test but don't want the combinations. for example:

-parametrize
    key: p1
    vals:
        - a
        - b
-parametrize
    key: p2
        -x
        -y

This gives 4 tests, (a,x), (a,y), (b,x), (b,y)but I only want 2 tests (a,x), (b,y)

Another use case is that the pass criteria changes with each parameter. For example:

key: input
vals:
    - 1
    - 3
key: double
vals:
    - 2
    - 6

Is that possible? thanks.

There are a few things that need changing and a few issues open that are hard to fix without breaking backwards compatibility. To make sure we know what is going to break in advance, I'm going to try and lay it all out here. This is almost all just related to tighter integration with pytest.

pytest integration

• tavern-ci will either be removed or just become an alias for py.test

• A 'base' REST/MQTT request/response object to make it easier for plugins to verify responses

• delay_after and delay_before will become built-in fixtures or hooks. This is a breaking API change.

• parametrization with pytest - running the same test, but with a different formatted value (or possibly a different dict/list/value)

• marking tests - this will just be for use with the -m flag. A list of tags applied to each test which can be used to run a certain subset of tests.

• This is sort of covered by parametrization and fixtures, but we want to be able to add 'plugin' blocks depending on a fixture. eg https://github.com/taverntesting/tavern/issues/113 . This would probably be a breaking API change, but I doubt many people are using plugins at the moment so it shouldn't be too bad.

Either/both:

• Some sort of integration with pytest fixtures - unsure how this will work now, but it will allow you to do some arbitrary python code and return a dictionary/list/value for the request/response/etc. This will probably be implemented such that we can send a value to the fixture if it's a generator, so fixtures can be written like this:

@pytest.fixture(name="tavern_complicated_request_val")
def test_complicated_return_value():
    val = calculate_something_complicated()

    response = (yield val)

    expected = calculate_complicated_response()
    assert response.body["value"] == expected    

request:
  json:
    sent_value: !fromfixture "tavern_complicated_request_val"

• extra hooks to allow people to run things before/after/during tests. For example, a hook called tavern_before_test which is run before every test and is called with the associated request and YamlItem.

Either one of these should cover all use cases, but both of them being available would definitely be the best case scenario.

Possible

• xfailing tests - this is probably going to be hidden from external use but it would be helpful to be able to write a deliberately broken test for tavern to make sure that it can be caught and reported to the user appropriately.
• skipif with pytest - eg, skip a test if running against a certain server. This will provide similar but different functionality to marking tests. I'm not sure of a nice way to implement this other than using exec(), but that might just have to happen.

problems

• Pytest only has the concept of a test, not the concept of a stage like Tavern does. We don't want to have to reimplement all the possible pytest behaviour for stages as well as tests, so adding parametrization or marking to individual stages is probably not going to happen. skip/skipif should be possible. Fixtures will almost always have to be run per-stage, so there may be some rudimentary 'decoration' of stages and tests to allow for this.

Note going to happen as a part of 1.0

• Better test reporting - this is quite a lot of work, but adding a better error reporting which actually prints the stage that failed and why it failed with arrows next to it like pytest would help a lot.

Note that none of this is set in stone, and no schema for the above changes has been decided on. This probably isn't going to be pushed out for at least a month or two. If there are any other (minor) breaking API changes that would greatly improve the user experience, feel free to suggest them here.

I'm using Tavern tests to test my REST API here.

On my Windows machine, I can run all tests without problem.

When running my tests on a Ubuntu host or inside an Ubuntu container, I get the following error:

Errors:
E   tavern.util.exceptions.PluginLoadError: Error loading plugin paho-mqtt = tavern._plugins.mqtt.tavernhook - [Errno 20] Not a directory: '/home/stefan/venv-tng/lib/python3.6/site-packages/tavern-0.26.1-py3.6.egg/tavern/_plugins/mqtt/schema.yaml'

In don't get this. I have installed tavern along with all other dependencies in a virtualenv. The error happens both when using python setup.py install and python setup.py develop.

When trying to install this package with

pip3 install -i https://devops.myURL/artifactory/api/pypi/pypi-all/simple -r requirements.txt

with requirements.txt

pytest
tavern[pytest]

one gets the following error:

Collecting pytest (from -r requirements.txt (line 1))
  Downloading https://devops.myURL/artifactory/api/pypi/pypi-all/packages/77/64/3a76f6fbb0f392d60c5960f2b2fbad8c2b802dada87ca6d1b99c0083a929/pytest-3.6.3-py2.py3-none-any.whl (195kB)
Collecting tavern[pytest] (from -r requirements.txt (line 2))
  Downloading https://devops.myURL/artifactory/api/pypi/pypi-all/packages/83/9f/419752827e66422a144597a40f4d53b1805f6af57ee64006f547bf1f89a0/tavern-0.14.4.tar.gz (42kB)
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.org/simple/pytest-runner/: [Errno 99] Cannot assign requested address -- Some packages may not be found!
    Couldn't find index page for 'pytest-runner' (maybe misspelled?)
    Download error on https://pypi.org/simple/: [Errno 99] Cannot assign requested address -- Some packages may not be found!
    No local packages or working download links found for pytest-runner
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-_4ygw1zi/tavern/setup.py", line 87, in <module>
        "tests": TESTS_REQUIRE
      File "/home/one/.local/lib/python3.5/site-packages/setuptools/__init__.py", line 130, in setup
        _install_setup_requires(attrs)
      File "/home/one/.local/lib/python3.5/site-packages/setuptools/__init__.py", line 125, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/home/one/.local/lib/python3.5/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
        replace_conflicting=True,
      File "/home/one/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 773, in resolve
        replace_conflicting=replace_conflicting
      File "/home/one/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1056, in best_match
        return self.obtain(req, installer)
      File "/home/one/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1068, in obtain
        return installer(requirement)
      File "/home/one/.local/lib/python3.5/site-packages/setuptools/dist.py", line 581, in fetch_build_egg
        return cmd.easy_install(req)
      File "/home/one/.local/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 670, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pytest-runner')
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-_4ygw1zi/tavern/

the Installation fails as this (ci) machine has only connectivity to this registry and not to the internet..

cc @max-wittig

Two dictionaries are loaded from fixtures in conftest.py: tmp_author_alias_a and tmp_author_alias_b.

They are same structure with differing values:

@pytest.fixture
def tmp_author_alias_a():
    return {
               "first_name": "Patrick",
               "last_name": "Neve",
# . . .
               "author_id": "dcfab002-d02f-4895-9557-b55f304af92d",
               "id": "ea36f093-1cae-403b-9c6e-3fe18b617221"
           }

They appear in the tavern script in - usefixtures:

  - usefixtures:
      - encode_web_token
      - conjure_uuid
      - conjure_uuid_2
      - tmp_author_alias_a
      - tmp_author_alias_b

They need to be in the request json as list of dictionaries.

Entered this way:

          authors: 
            - {tmp_author_alias_a}
            - {tmp_author_alias_b}

throws BadSchemaError: cannot define an empty value in test . . .

This way:

          authors: 
            - "{tmp_author_alias_a}"
            - "{tmp_author_alias_b}"

In line 39 dict_util.py format_keys the list is pulled in as a node_class:

formatted = val  # node_class: ['temp_author_alias_a', 'temp_author_alias_b']
box_vars = Box(variables)

That is as a list of strings.
Each is then sent back through format_keys() (in line 48)

    formatted = [format_keys(item, box_vars) for item in val]

 as a string into which the dict is interpolated (line 51-53)

elif isinstance(val, (ustr, str)):
    try:
        formatted = val.format(**box_vars)

Any way to get the list of dictionaries into the request json?  

A nice poser for your afternoon diversion! ;-)

The problem I want to solve is running multiple tests with almost the same data - without creating YAML for each possible test. An example might be "list users" API and then validate that each user has a login directory (for example) in a subsequent stage.

There are two variations of this that come to mind:

• specify values to set in a JSON (or CSV) list in a file
• perform a query in a previous step which provides a saved JSON list to be used in subsequent tests.

And then walk through these values, and provide them to all the "stages" which would be nested below this for this particular subtest.

After this "stage" was completed, test execution would continue with the next stage at the outer level.

I hope this is clear...

I would imagine a "foreach" tag to be used below the stages level - or below the name level. And below it would be more stages: tags and stages which are to be repeated using values specified either from a file or in previous saved JSON saved from a prior stage.

I'm currently doing this by doing textual substitution in a template file, and generating many YAML files - one for each iteration. I'd like to just do this as part of the test - without this extra python program, and then a separate test to run this program against the template - and so on...

Hi All,

Is there any plan in pipeline to add the report generation(html or some other standard) in the near future.It would be good if we add the reporting part. It would be good if we add detailed reports capturing in the format like, TestName Stages steps/name status with success/failure indicators,validation part and also request or response passed

As of now we are using pytest junitxml generation and pytest-html plugin for reporting .These options are giving only high-level reports.It would be good if we provide details in granular level. This feature would be very useful for Tavern wide usage for API testing.

Hi. I want to run tavern integration tests on my test database created in my test django environment. Also I want to run django server before run testing. (without typing and starting server manually) Tox manage all tests. Is there some possibility to do this stuff? If not is there some solution to enhance tavern? Thank you.

PS: also I use pytest-django library.

Hi all

I would like to check for a non-json body response. In my case the response is a simple: True or False.

I can't find any documentation stating how to check a text body instead of a json body.

Is possible to match only few parameters in returned json? I have big json, and I don't want checking all fields. I would like to check only key elements.

Currently mqtt_response with the unexpected key set to true fails if no message is received. This is strange, was we don't expect a message, and should only fail if a message is received.

Formatted stage:
  mqtt_publish:
    payload: 10.10.10.10
    topic: inet6/add
  mqtt_response:
    payload: !anything ''
    timeout: 5
    topic: vallumd/will
    unexpected: true

Errors:
E   tavern.util.exceptions.TestFailError: Test 'add IPv4 IP to IPv6 ipset' failed:
    - Expected '<Tavern YAML sentinel for anything>' on topic 'vallumd/will' but no such message received

Fix this by suppressing the error MQTTResponse._await_response() and returning an empty dict if the unexpected key is set.

Signed-off-by: Stijn Tintel [email protected]

I originally submitted this as a pytest issue but they suggested that I report it here instead.

In each of two directories, I have one Tavern test file that will run with pytest:

C02VF3KDHTD8:api schooler$ ls 2-disruptive/test_hardware_negative.tavern.yaml 3-destructive/test_components.tavern.yaml
2-disruptive/test_hardware_negative.tavern.yaml	3-destructive/test_components.tavern.yaml

When I execute them both with one invocation, the summary output shows an incorrect test file path.

C02VF3KDHTD8:api schooler$ python3 -m pytest 2-disruptive/ 3-destructive/
========================================================================================================== test session starts ===========================================================================================================
platform darwin -- Python 3.9.13, pytest-7.1.2, pluggy-1.0.0
rootdir: <REDACTED>, configfile: pytest.ini
plugins: tavern-1.23.3
collected 2 items

2-disruptive/test_hardware_negative.tavern.yaml .                                                                                                                                                                                  [ 50%]
2-disruptive/test_components.tavern.yaml .                                                                                                                                                                                         [100%]

=========================================================================================================== 2 passed in 5.74s ============================================================================================================

The summary output should show 3-destructive/test_components.tavern.yaml as the last test and not 2-disruptive/test_components.tavern.yaml which doesn't exist.

Additional comment from the pytest thread that was closed: "I suggest to report this in the tavern repository, as I suspect there might be an issue on how they are creating the test items."

Seeing as the pypy tests take a long time to run, and I doubt many people use pypy to run Tavern as it goes multiple times slower than just normal python for some reason

Chaining stages in 2 separate files, for example: test_setup.tavern.yaml - to be used by many other tests in different/separate yaml files, saves some variables test_execution.tavern.yaml - depends on test_setup.tavern.yaml to be executed first, and be able to use the saved variables in test_setup.tavern.yaml

Read about reusable code, but would like to have more stages in the reusable file, that is included.