This was originally in #96, and removed to make review easier. This undoes that remove.

Part of addressing #27.

Again, planning on writing docs in another PR, as this PR is fairly large already and may get slightly larger depending on review comments.

By default, use the same Agent treq always provides, but allow using a custom Agent. Unless/until treq grows addtional features, this would allow you to customize some of the behavior.

I have been experimenting with this by adding the ability to add internal CA certs to the certificate store. This lets treq make requests to sites with public certs and sites on an internal network. I find this more desirable than adjusting the OS certificate bundle, as treq may be used in different contexts on the same server. E.g. process "A" should really only be dealing with public sites, but process "B" might need to make requests from internal sites.

https://en.wikipedia.org/wiki/Internationalized_domain_name

We really want to use treq with Twisted for making requests, but we are afraid that our code will not work if/when we encounter internationalized urls.

Thanks to @glyph for the suggestion of using the request traversal agent from https://github.com/rackerlabs/mimic, which means that treq actually operates and produces a real response, so we don't have to make a verified fake response, and for help debugging HTTPS and request issues.

Part of addressing #27.

Was going to write the docs and tutorial next if this looks good.

The auth keyword argument now takes a callable accepts an IAgent and returns an IAgent. The returned IAgent can then implement whatever authorization scheme the human mind can imagine.

This seems pretty bad:

>>> import treq
>>> treq.post('https://httpbin.org/post', json={'here is': 'some json'}).addCallback(treq.content).addCallback(print)
<Deferred at 0x1074450e0>
Waiting on Deferred #0...
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Connection": "close", 
    "Content-Length": "0", 
    "Host": "httpbin.org"
  }, 
  "json": null, 
  "origin": "lol", 
  "url": "https://httpbin.org/post"
}
Deferred #0 Fired:

The merge of #166 broke the RTD build, where using a requirements file (so that the install goes through pip instead of setuptools) was the only way I could get it to correctly install pyasn1 (python setup.py install got confused by pyasn1-modules).

This commit changes the name 'Treq' in _version.py to 'treq'. This makes the Version capitalization consistent with the caps throughout the package, and makes it possible to use __version__.local() to generate a local version number with incremental.

Based on a conversation with Glyph in PR #76, I reverted the changes from that PR and instead updated the examples and documentation to emphasize that client code may interact with treq.client.HTTPClient directly. In fact this should be the way to customize the behavior of the client. The various treq request functions are just shortcuts.

• Remove setup.cfg [metadata] section, which didn't match setup.py and is deprecated anyway.
• Instead of depending on Twisted's various TLS dependencies directly, depend on Twisted[tls] which has been present since at least Twisted 16.0.0 (but retain a higher PyOpenSSL dep for Python 3, since Twisted[tls] requires a lesser version).

Several changes to bring the tox/Travis setup up-to-date:

• Test against lowest/latest and trunk Twisted/pyOpenSSL rather than specific versions (this means we test against newer Twisted/pyOpenSSL than we were)
• Make Python 3.3 - 3.5 part of the main tests in Travis
• Install a newer PyPy for the PyPy tests
• Set the Travis Python version to match the pyenv one
• Upgrade Travis pip and enable caching

Things I haven't managed to fix yet:

• Coverage is broken with Twisted >= 16.4.0. This was a problem in Treq before my changes. Coverage only seems to work when --source is set to {envsitepackagesdir}/treq with Twisted >= 16.4.0 but needs to be set to treq with older Twisted.

import os

from twisted.internet.task import react

import treq


async def main(reactor):
    print("Without files:")
    response = await treq.post(
        "https://httpbin.org/post",
        data={"form": ["1", "2"]},
    )
    print(await response.text())

    print("With files:")
    response = await treq.post(
        "https://httpbin.org/post",
        data={"form": ["1", "2"]},
        files={"foo": (os.devnull, open(os.devnull, "rb"))},
    )
    print(await response.text())


react(main, [])

When run:

Without files:
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "form": [
      "1", 
      "2"
    ]
  }, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Content-Length": "13", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "X-Amzn-Trace-Id": "Root=1-63a26dee-580ab8e77fdd228c1eb84266"
  }, 
  "json": null, 
  "origin": "99.187.228.202", 
  "url": "https://httpbin.org/post"
}

With files:
main function encountered error
Traceback (most recent call last):
  File ".../twisted/internet/defer.py", line 696, in callback
    self._startRunCallbacks(result)
  File ".../twisted/internet/defer.py", line 798, in _startRunCallbacks
    self._runCallbacks()
  File ".../twisted/internet/defer.py", line 892, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File ".../twisted/internet/defer.py", line 1792, in gotResult
    _inlineCallbacks(r, gen, status, context)
--- <exception caught here> ---
  File ".../twisted/internet/defer.py", line 1697, in _inlineCallbacks
    result = context.run(gen.send, result)
  File ".../demo.py", line 17, in main
    response = await treq.post(
  File ".../src/treq/api.py", line 32, in post
    return _client(kwargs).post(url, data=data, _stacklevel=4, **kwargs)
  File ".../src/treq/client.py", line 182, in post
    return self.request('POST', url, data=data, **kwargs)
  File ".../src/treq/client.py", line 244, in request
    bodyProducer, contentType = self._request_body(data, files, json,
  File ".../src/treq/client.py", line 379, in _request_body
    multipart.MultiPartProducer(data + files, boundary=boundary),
  File ".../src/treq/multipart.py", line 53, in __init__
    self._fields = list(_sorted_by_type(_converted(fields)))
  File ".../src/treq/multipart.py", line 350, in _sorted_by_type
    return sorted(fields, key=key)
  File ".../src/treq/multipart.py", line 260, in _converted
    raise ValueError(
builtins.ValueError: Expected tuple: (filename, content type, producer)

When only data is passed we URL-encode the request body with urllencode(..., doseq=True), whereas when both data and files are passed we combine them and pass the result to MultiPartProducer without unboxing sequences of values.

There are two uses of the cgi module in treq:

1. One use of cgi.parse_header() in treq.content — suggested replacement is email.message.Message
2. One use of cgi.parse_multipart() in treq.test.test_multipart — same suggested replacement

httpbin, which is used by treq's test suite (but not elsewhere), hasn't seen a release since 2018. The ecosystem of libraries it relies on is accumulating changes. Without maintenance it will eventually break.

This first became a problem in #352, which relates to deprecated Werkzeug symbols. #353 pinned httpbin and werkzeug (which also fixed compatibility with markupsafe).

Eventually we will have to vendor or replace httpbin. At that point we should also reconsider having so many integration tests — they are slow relative to the coverage they provide.

Hey there!

I belong to an open source security research community, and a member (@ranjit-git) has found an issue, but doesn’t know the best way to disclose it.

If not a hassle, might you kindly add a SECURITY.md file with an email, or another contact method? GitHub recommends this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.

Thank you for your consideration, and I look forward to hearing from you!

(cc @huntr-helper)

If a DecodedURL with a username/password in it is passed to a treq API, an error results about encoding a colon character (and the password is removed). (I can followup with the traceback but I don't have it immediately handy).

To make such a URL work, one can simply pass auth=url.userinfo to the treq reqeuest (and .replace(userinfo=()) to remove the username/password from URL) so it might be nice to have that as "a feature" in treq itself. That is, take out the userinfo from the DecodedURL and use it for authentication.

The treq dependency on requests seems to be causing issues for Klein and other packages that depend on idna

https://twistedmatrix.com/pipermail/twisted-python/2021-February/065444.html