• Add Python 3.9
• Merge deps and requirements from tox.ini into setup.py, and remove from tox.ini.
• Set minimum coverage percent to 100. WebTest's coverage has actually been failing for over 7 years. Oops! https://travis-ci.org/github/Pylons/webtest/jobs/23066821
• Bump docs tox env to Python 3.9
• Sort reqs and extras in setup.py
• Use consistent indentation in tox.ini

See also #230.

On my Gentoo ~amd64 machine I see the following error, when doing a form submission with a file upload.

    self.response = form.submit(button)
  File "/usr/lib64/python3.4/site-packages/webtest/forms.py", line 606, in submit
    params=fields, **args)
  File "/usr/lib64/python3.4/site-packages/webtest/response.py", line 294, in goto
    return method(href, **args)
  File "/usr/lib64/python3.4/site-packages/webtest/app.py", line 370, in post
    content_type=content_type)
  File "/usr/lib64/python3.4/site-packages/webtest/app.py", line 712, in _gen_request
    params, upload_files or ())
  File "/usr/lib64/python3.4/site-packages/webtest/app.py", line 525, in encode_multipart
    body = b'\r\n'.join(lines)
TypeError: sequence item 8: expected a bytes-like object, NoneType found

I could fix this by adding the following change to encode_multipart() in app.py. However I'm not sure this is the correct fix.

            if value.content is not None:
                    file_info.append(value.content)
                    if value.content_type is not None:
                file_info.append(value.content_type)
        _append_file(file_info)
-            else:
+            elif value is not None:
        if isinstance(value, text_type):
                    value = value.encode('utf8')
        lines.extend([
                    b'--' + boundary,
                    b'Content-Disposition: form-data; name="' + key + b'"',

Here is what I do: I have a form with one file upload and one submit button as follows.

    <form action="/event/event/1/minorform/change" method="POST" enctype="multipart/form-data" id="changeminorformform">
            <input type="file" name="minor_form" accept="application/pdf" class="form-control" >
            <button type="submit" name="submitform" class="btn btn-primary">Hochladen</button>
    </form>

I add a webtest.Upload for the file and submit the form with the button name as parameter. Then the variable params of encode_multipart has the value

[('minor_form', ), ('submitform', None)]

where the second one causes the problems.

Feature request. We're trying out a new feature of GitHub: issue templates.

See examples: https://github.com/Pylons/docs-style-guide/issues/new/choose

Source: https://github.com/Pylons/docs-style-guide/tree/master/.github/ISSUE_TEMPLATE

Hopefully they direct requests for support to the proper channel instead of the mere 20 people watching this repo. Please let me know your thoughts.

i get this warning: /home/vagrant/django-webtest/.tox/py34-django110-std/lib/python3.4/site-packages/bs4/init.py:146: UserWarning: You provided Unicode markup but also provided a value for from_encoding. Your from_encoding will be ignored.``

i believe this is caused by https://github.com/Pylons/webtest/blob/9ca03a04927744fbea1b6a3fd209623981204bf1/webtest/forms.py#L404 (there may be other occurrences, haven't checked)

cheers :)

While running nose2 tests in our application we are getting the following error

Exception AssertionError: AssertionError(u'Iterator garbage collected without being closed',) in <bound method IteratorWrapper.del of <webtest.lint.IteratorWrapper object at 0x110f15690>> ignored

Though the error is ignored for a particular test; running a series layered tests this error may affect the outcome of the test run

Carlos

Mikhail Korobov:

Hi Ian,
Sometimes there is no need to strictly require a redirect or to
prohibit it (e.g. if some legacy urls are migrated to new scheme).

Would you accept a patch with TestResponse.auto_follow method?
This method will follow the redirect chain and won't complain if
there is no redirect.

Gael Pasgrimaud:

Feel free to provide a patch if you still need this feature

URL: https://bitbucket.org/ianb/webtest/issue/13/support-redirect-auto-following

This have been discussed before, and I really wish this would be possible.

I have a form which looks like this:

	  <form class="form-horizontal" action="/create_bundle/">
        <div class="table-responsive">
          <table class="table table-bordered table-striped table-highlight">
            <thead>
             <tr>
               <th>Serial</th><th>Site</th>
             </tr>
            </thead>
            <tbody>
							 
                <tr>
                    <td>abcdef123</td>
                    <td>
                        <select class="form-control" name="site">
                            <option>lünen</option>
                        </select>
                    </td>
		      <td title="click to configure this device..." align="center">
                      <button type="submit" class="btn btn-success" name="serial" value="abcdef123">
                       <i class="fa fa-wrench" aria-hidden="true"></i>
                      </button>
                    </td>
	                     <td title="click to remove this device from the pending list..." align="center">
                             <button type="submit" formaction="/delete/" class="btn btn-danger" id="del1" name="serial" value="abcdef123">
                        <i class="fa fa-trash" aria-hidden="true"></i>
                      </button>
                    </td>
                </tr>
               
            </tbody>
          </table>
        </div>
      <input type="hidden" name="csrf_token" value="...asd.a...asdasd...">
</form>

I can't click this button ... but it is found with BeatifulSoup:

-> deleted = res.clickbutton(buttonid="del1")
(Pdb++) res.clickbutton(buttonid="del1")
*** IndexError: No matching elements found (from 0 possible)
(Pdb++) res.html.find(id="del1")
<button class="btn btn-danger" formaction="/delete/" id="del1" name="serial" type="submit" value="abcdef123">
<i aria-hidden="true" class="fa fa-trash"></i>
</button>

Is there any possible work around this?

webtest/app.py:550: WSGIWarning: You are not supposed to send a body in a DELETE request. Most web servers will ignore it

This is caused by a test that checks if params is not NoDefault, but the default value is u'' instead of NoDefault.

(Side note: using the stacklevel param of warnings.warn will cause the traceback to show the line that called the function, not the line of the function that calls warn).

The zope.testbrowser tests are affected by this, see https://github.com/zopefoundation/zope.testbrowser/commit/8e3c130778320650076b57f956fe55907cc1c784 and https://travis-ci.org/zopefoundation/zope.testbrowser/jobs/226375649.

The commit b35063f1bc6f6b0cd92b68c56838f385b56e8d00 added a shortcut that breaks strict cookie policies. If one uses a trailing dot in HTTP_HOST when the domain is not a FQDN, then it works without the shortcut and the strict policy works as well.

@gawel since you made the commit, could you take a look at this?

I get multiple test failures on Python 2.6.2 related to forms processing with a typical traceback:

....

======================================================================
ERROR: test_single_select_forced_value (tests.test_forms.TestSelect)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/users/is/skolesnik/webtest/WebTest-v2.0.6/WebTest-2.0.6/tests/test_forms.py", line 502, in test_single_select_forced_value
    single_form = res.forms["single_select_form"]
  File "/users/is/skolesnik/webtest/WebTest-v2.0.6/WebTest-2.0.6/webtest/response.py", line 42, in forms
    self._parse_forms()
  File "/users/is/skolesnik/webtest/WebTest-v2.0.6/WebTest-2.0.6/webtest/response.py", line 77, in _parse_forms
    form = forms.Form(self, text)
  File "/users/is/skolesnik/webtest/WebTest-v2.0.6/WebTest-2.0.6/webtest/forms.py", line 367, in __init__
    self._parse_fields()
  File "/users/is/skolesnik/webtest/WebTest-v2.0.6/WebTest-2.0.6/webtest/forms.py", line 414, in _parse_fields
    field = FieldClass(self, tag, name, pos, **attrs)
TypeError: __init__() keywords must be strings

...
----------------------------------------------------------------------
Ran 162 tests in 8.353s

FAILED (errors=37)

It is happening because BeautifulSoup returns form attributes as a dict where both keys and values are unicode strings. This dict is then attempted to be exploded as kwargs to a function, which fails on Python<3. Is it a known incompatibility or a bug?

Test environment is RHEL5 plus Python 2.6.2. pip freeze returns:

distribute==0.6.36
beautifulsoup4==4.2.1
coverage==3.5.3
cssselect==0.7.1
lxml==2.3.6
mock==1.0.1
nose=1.3.0
ordereddict==1.1
PasteDeploy==1.5.0
pyquery==1.2.4
six==1.3.0
unittest2==0.5.1
waitress==0.8.1
WebOb==1.2.3
WSGIProxy2==0.2

Any ideas?

Test suite with Python 3.3 triggers 4 errors. (All tests pass with Python 2.6, 2.7 and 3.2.)

$ nosetests-3.3
...
======================================================================
ERROR: test_expires_cookies (tests.test_app.TestCookies)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.3/site-packages/mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "/tmp/WebTest-2.0.5/tests/test_app.py", line 241, in test_expires_cookies
    app.get('/')
  File "/tmp/WebTest-2.0.5/webtest/app.py", line 199, in get
    expect_errors=expect_errors)
  File "/tmp/WebTest-2.0.5/webtest/app.py", line 494, in do_request
    utils._RequestCookieAdapter(req))
  File "/usr/lib64/python3.3/http/cookiejar.py", line 1647, in extract_cookies
    if self._policy.set_ok(cookie, request):
  File "/usr/lib64/python3.3/http/cookiejar.py", line 931, in set_ok
    if not fn(cookie, request):
  File "/usr/lib64/python3.3/http/cookiejar.py", line 952, in set_ok_verifiability
    if request.unverifiable and is_third_party(request):
  File "/usr/lib64/python3.3/http/cookiejar.py", line 707, in is_third_party
    if not domain_match(req_host, reach(request.origin_req_host)):
AttributeError: '_RequestCookieAdapter' object has no attribute 'origin_req_host'

======================================================================
ERROR: test_preserves_cookies (tests.test_app.TestCookies)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/WebTest-2.0.5/tests/test_app.py", line 172, in test_preserves_cookies
    res = app.get('/')
  File "/tmp/WebTest-2.0.5/webtest/app.py", line 199, in get
    expect_errors=expect_errors)
  File "/tmp/WebTest-2.0.5/webtest/app.py", line 494, in do_request
    utils._RequestCookieAdapter(req))
  File "/usr/lib64/python3.3/http/cookiejar.py", line 1647, in extract_cookies
    if self._policy.set_ok(cookie, request):
  File "/usr/lib64/python3.3/http/cookiejar.py", line 931, in set_ok
    if not fn(cookie, request):
  File "/usr/lib64/python3.3/http/cookiejar.py", line 952, in set_ok_verifiability
    if request.unverifiable and is_third_party(request):
  File "/usr/lib64/python3.3/http/cookiejar.py", line 707, in is_third_party
    if not domain_match(req_host, reach(request.origin_req_host)):
AttributeError: '_RequestCookieAdapter' object has no attribute 'origin_req_host'

======================================================================
ERROR: test_secure_cookies (tests.test_app.TestCookies)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/WebTest-2.0.5/tests/test_app.py", line 207, in test_secure_cookies
    res = app.get('https://localhost/')
  File "/tmp/WebTest-2.0.5/webtest/app.py", line 199, in get
    expect_errors=expect_errors)
  File "/tmp/WebTest-2.0.5/webtest/app.py", line 494, in do_request
    utils._RequestCookieAdapter(req))
  File "/usr/lib64/python3.3/http/cookiejar.py", line 1647, in extract_cookies
    if self._policy.set_ok(cookie, request):
  File "/usr/lib64/python3.3/http/cookiejar.py", line 931, in set_ok
    if not fn(cookie, request):
  File "/usr/lib64/python3.3/http/cookiejar.py", line 952, in set_ok_verifiability
    if request.unverifiable and is_third_party(request):
  File "/usr/lib64/python3.3/http/cookiejar.py", line 707, in is_third_party
    if not domain_match(req_host, reach(request.origin_req_host)):
AttributeError: '_RequestCookieAdapter' object has no attribute 'origin_req_host'

======================================================================
ERROR: test_follow_with_cookie (tests.test_response.TestFollow)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/WebTest-2.0.5/tests/test_response.py", line 382, in test_follow_with_cookie
    app.get('/?header-set-cookie=foo=bar')
  File "/tmp/WebTest-2.0.5/webtest/app.py", line 199, in get
    expect_errors=expect_errors)
  File "/tmp/WebTest-2.0.5/webtest/app.py", line 494, in do_request
    utils._RequestCookieAdapter(req))
  File "/usr/lib64/python3.3/http/cookiejar.py", line 1647, in extract_cookies
    if self._policy.set_ok(cookie, request):
  File "/usr/lib64/python3.3/http/cookiejar.py", line 931, in set_ok
    if not fn(cookie, request):
  File "/usr/lib64/python3.3/http/cookiejar.py", line 952, in set_ok_verifiability
    if request.unverifiable and is_third_party(request):
  File "/usr/lib64/python3.3/http/cookiejar.py", line 707, in is_third_party
    if not domain_match(req_host, reach(request.origin_req_host)):
AttributeError: '_RequestCookieAdapter' object has no attribute 'origin_req_host'

Name               Stmts   Miss  Cover   Missing
------------------------------------------------
webtest               24      0   100%   
webtest.app          263      1    99%   146
webtest.compat        18      2    89%   34-35
webtest.debugapp      63      1    98%   35
webtest.ext            3      0   100%   
webtest.forms        288      0   100%   
webtest.http          87      3    97%   89-91
webtest.lint         203      1    99%   251
webtest.response     256     14    95%   270-286, 382, 389, 539
webtest.sel            6      0   100%   
webtest.utils         91     10    89%   88, 107, 118, 121, 124, 127, 130, 135, 138, 153
------------------------------------------------
TOTAL               1302     32    98%   
----------------------------------------------------------------------
Ran 166 tests in 11.006s

FAILED (errors=4)

Documentation Suggestion

https://docs.pylonsproject.org/projects/webtest/en/latest/api.html#webtest.app.TestApp.delete

The doc says that Do a DELETE request. Similar to get().

But actually, the parameter params of get() create a querystring.

And params in DELETE create an url encode version of the parameter in a body.

so it behave more like post() than get()

Describe the improvement

As an improvement, actually, a breaking change should be made, where params should act like get()

Eg, a form like:

<form method="post">....

would get parsed out as a form with "/" as action url but instead the current url should be the action url. as per the html5 spec at https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#form-submission-algorithm

11. If action is the empty string, let action be the URL of the form document.

I've a test helper that populates a form that I'm porting from mechanize to webtest. During this I've noticed that webtest ignores the disabled attribute on individual radio buttons. Given the following HTML:

<html>
  <form name="my_form" action="test.html"
        enctype="multipart/form-data" method="post">
    <input type="radio" name="radio-4" value="1" />
    <input type="radio" name="radio-4" value="2" />
    <input type="radio" name="radio-4" value="3" />
    <input type="radio" name="radio-4" value="4" disabled="disabled" />
    <input type="submit" name="submit" value="Submit" />
  </form>
</html>

if I feed it to webtest (via a very simple wsgi app) and access the form, the following things stand out, form.fields['radio-4'].options lists all four options, including the disabled one.

Some interesting details that may be relevant:

• print(form.html) doesn't even show "disabled" attributes
• print(form.text) shows the disabled="disabled", but the HTML has been reformatted and doesn't match my actual HTML source

In this use case, we are implementing a checkbox/chevron kind of widget, for this we're adding new fields dynamically (with javascript) if they don't yet exist.

While testing this field, we had to add the checkboxes manually in order to be able to submit it. However, we have not found any mention of this in the docs; did we miss it, or is it not documented?

A bit of googling brought us to this SO answer (quoted below), and it seemed to work well for us, except we also had to add the pos argument. Is this the suggested way to do this? Would you appreciate a PR to document this use case?

from webtest.forms import Checkbox

def add_dynamic_checkbox_field(form, name, value):
    """ Add an extra checkbox field to a form. """
    field = Checkbox(form, 'input', None, 1, value)
    form.fields[name] = [field]
    form.field_order.append((name, field))

# usage:
add_dynamic_checkbox_field(resp.form, 'new_field_name', 'some_value')

If the server sends multiple Set-Cookie headers, both the default and the requests client conflate those into one header-line, comma-separating the values -- which cookiejar cannot parse. Luckily, the urllib3 client keeps the headers as-is, so that it works in this case.

I've been wrestling with this for a while and I still can't figure out who the culprit is, however I suppose WebTest should be responsible since it is the package pulling the various components together, and may be Web Test's handling of the URL. Still unsure... In any case, take the following unit tests:

class WebTestUnicodeUrlTests(WebTest):

    def setUp(self):
        self.event_ascii = EventFactory(url=u'eee')
        self.event_unicode = EventFactory(url=u'ééé')

    def test_ascii_url(self):
        url = reverse('vanity_url_catcher', args=[self.event_ascii.url])
        print url  # /eee
        self.app.get(url, status=200)

    def test_unicode_url(self):
        url = reverse('vanity_url_catcher', args=[self.event_unicode.url])
        print url  # /%C3%A9%C3%A9%C3%A9
        self.app.get(url, status=200)

    def test_ascii_url_via_baserequest(self):
        url = reverse('vanity_url_catcher', args=[self.event_ascii.url])
        print url  # /eee
        req = BaseRequest.blank(url)
        self.assertEqual(req.url, url)

    def test_unicode_url_via_baserequest(self):
        url = reverse('vanity_url_catcher', args=[self.event_unicode.url])
        print url  # /%C3%A9%C3%A9%C3%A9
        req = BaseRequest.blank(url)
        self.assertEqual(req.url, url)

The result of which is:

ERROR: test_unicode_url (frontend.generic.tests.test_views.WebTestUnicodeUrlTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/frontend/generic/tests/test_views.py", line 58, in test_unicode_url
    self.app.get(url, status=200)
  File "/virtualenv/frontend/local/lib/python2.7/site-packages/django_webtest/__init__.py", line 124, in get
    response = super(DjangoTestApp, self).get(url, **all_kwargs_but_url)
  File "/virtualenv/frontend/local/lib/python2.7/site-packages/webtest/app.py", line 330, in get
    expect_errors=expect_errors)
  File "/virtualenv/frontend/local/lib/python2.7/site-packages/django_webtest/__init__.py", line 86, in do_request
    expect_errors)
  File "/virtualenv/frontend/local/lib/python2.7/site-packages/webtest/app.py", line 648, in do_request
    utils._RequestCookieAdapter(req))
  File "/usr/lib/python2.7/cookielib.py", line 1659, in extract_cookies
    if self._policy.set_ok(cookie, request):
  File "/usr/lib/python2.7/cookielib.py", line 941, in set_ok
    if not fn(cookie, request):
  File "/usr/lib/python2.7/cookielib.py", line 962, in set_ok_verifiability
    if request.is_unverifiable() and is_third_party(request):
  File "/usr/lib/python2.7/cookielib.py", line 716, in is_third_party
    req_host = request_host(request)
  File "/usr/lib/python2.7/cookielib.py", line 605, in request_host
    url = request.get_full_url()
  File "/virtualenv/frontend/local/lib/python2.7/site-packages/webtest/utils.py", line 111, in get_full_url
    return self._request.url
  File "/virtualenv/frontend/local/lib/python2.7/site-packages/webob/request.py", line 504, in url
    url = self.path_url
  File "/virtualenv/frontend/local/lib/python2.7/site-packages/webob/request.py", line 476, in path_url
    bpath_info = bytes_(self.path_info, self.url_encoding)
  File "/virtualenv/frontend/local/lib/python2.7/site-packages/webob/descriptors.py", line 68, in fget
    return req.encget(key, encattr=encattr)
  File "/virtualenv/frontend/local/lib/python2.7/site-packages/webob/request.py", line 175, in encget
    return val.decode(encoding)
  File "/virtualenv/frontend/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-3: ordinal not in range(128)

----------------------------------------------------------------------
Ran 4 tests in 4.683s

FAILED (errors=1)

Is there a way around this? I believe I've tried all combinations I can think of to convert the url to different formats but nothing makes a difference, or simply falls foul of the immediate str(url) call in webtest.app:

  File "/virtualenv/frontend/local/lib/python2.7/site-packages/webtest/app.py", line 310, in get
    url = str(url)