Python requests like API built on top of Twisted's HTTP client.

Overview

treq: High-level Twisted HTTP Client API

PyPI calver: YY.MM.MICRO Coverage Documentation

treq is an HTTP library inspired by requests but written on top of Twisted's Agents.

It provides a simple, higher level API for making HTTP requests when using Twisted.

>>> import treq

>>> def done(response):
...     print(response.code)
...     reactor.stop()

>>> treq.get("https://github.com").addCallback(done)

>>> from twisted.internet import reactor
>>> reactor.run()
200

For more info read the docs.

Contributing

treq development is hosted on GitHub.

We welcome contributions: feel to fork and send contributions over. See CONTRIBUTING.rst for more info.

Code of Conduct

Refer to the Twisted code of conduct.

Copyright and License

treq is made available under the MIT license. See LICENSE for legal details and copyright notices.

Comments
  • Utility for providing a sequence of requests/responses to StubTreq

    Utility for providing a sequence of requests/responses to StubTreq

    This was originally in #96, and removed to make review easier. This undoes that remove.

    Part of addressing #27.

    Again, planning on writing docs in another PR, as this PR is fairly large already and may get slightly larger depending on review comments.

    good to land (with minor changes) 
    opened by cyli 43
  • Allow the client to specify a custom Agent.

    Allow the client to specify a custom Agent.

    By default, use the same Agent treq always provides, but allow using a custom Agent. Unless/until treq grows addtional features, this would allow you to customize some of the behavior.

    I have been experimenting with this by adding the ability to add internal CA certs to the certificate store. This lets treq make requests to sites with public certs and sites on an internal network. I find this more desirable than adjusting the OS certificate bundle, as treq may be used in different contexts on the same server. E.g. process "A" should really only be dealing with public sites, but process "B" might need to make requests from internal sites.

    opened by cwaldbieser 17
  • Handle Internationalized Domain Names

    Handle Internationalized Domain Names

    https://en.wikipedia.org/wiki/Internationalized_domain_name

    We really want to use treq with Twisted for making requests, but we are afraid that our code will not work if/when we encounter internationalized urls.

    opened by offero 15
  • Adding an in-memory stub version of treq

    Adding an in-memory stub version of treq

    Thanks to @glyph for the suggestion of using the request traversal agent from https://github.com/rackerlabs/mimic, which means that treq actually operates and produces a real response, so we don't have to make a verified fake response, and for help debugging HTTPS and request issues.

    Part of addressing #27.

    Was going to write the docs and tutorial next if this looks good.

    opened by cyli 15
  • Allow custom authorization mechanisms.

    Allow custom authorization mechanisms.

    The auth keyword argument now takes a callable accepts an IAgent and returns an IAgent. The returned IAgent can then implement whatever authorization scheme the human mind can imagine.

    opened by markrwilliams 13
  • treq.post(..., json=) doesn't send any request body

    treq.post(..., json=) doesn't send any request body

    This seems pretty bad:

    >>> import treq
    >>> treq.post('https://httpbin.org/post', json={'here is': 'some json'}).addCallback(treq.content).addCallback(print)
    <Deferred at 0x1074450e0>
    Waiting on Deferred #0...
    {
      "args": {}, 
      "data": "", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept-Encoding": "gzip", 
        "Connection": "close", 
        "Content-Length": "0", 
        "Host": "httpbin.org"
      }, 
      "json": null, 
      "origin": "lol", 
      "url": "https://httpbin.org/post"
    }
    Deferred #0 Fired:
    
    opened by glyph 12
  • Fix the ReadTheDocs build

    Fix the ReadTheDocs build

    The merge of #166 broke the RTD build, where using a requirements file (so that the install goes through pip instead of setuptools) was the only way I could get it to correctly install pyasn1 (python setup.py install got confused by pyasn1-modules).

    opened by twm 10
  • Use lower-case package name in Version

    Use lower-case package name in Version

    This commit changes the name 'Treq' in _version.py to 'treq'. This makes the Version capitalization consistent with the caps throughout the package, and makes it possible to use __version__.local() to generate a local version number with incremental.

    opened by dmurvihill 10
  • Emphasize client

    Emphasize client

    Based on a conversation with Glyph in PR #76, I reverted the changes from that PR and instead updated the examples and documentation to emphasize that client code may interact with treq.client.HTTPClient directly. In fact this should be the way to customize the behavior of the client. The various treq request functions are just shortcuts.

    opened by cwaldbieser 10
  • Dependency cleanup

    Dependency cleanup

    • Remove setup.cfg [metadata] section, which didn't match setup.py and is deprecated anyway.
    • Instead of depending on Twisted's various TLS dependencies directly, depend on Twisted[tls] which has been present since at least Twisted 16.0.0 (but retain a higher PyOpenSSL dep for Python 3, since Twisted[tls] requires a lesser version).
    opened by twm 9
  • tox/Travis improvements

    tox/Travis improvements

    Several changes to bring the tox/Travis setup up-to-date:

    • Test against lowest/latest and trunk Twisted/pyOpenSSL rather than specific versions (this means we test against newer Twisted/pyOpenSSL than we were)
    • Make Python 3.3 - 3.5 part of the main tests in Travis
    • Install a newer PyPy for the PyPy tests
    • Set the Travis Python version to match the pyenv one
    • Upgrade Travis pip and enable caching

    Things I haven't managed to fix yet:

    • Coverage is broken with Twisted >= 16.4.0. This was a problem in Treq before my changes. Coverage only seems to work when --source is set to {envsitepackagesdir}/treq with Twisted >= 16.4.0 but needs to be set to treq with older Twisted.
    opened by JayH5 9
  • Inconsistent handling of data sequence values with/without files

    Inconsistent handling of data sequence values with/without files

    import os
    
    from twisted.internet.task import react
    
    import treq
    
    
    async def main(reactor):
        print("Without files:")
        response = await treq.post(
            "https://httpbin.org/post",
            data={"form": ["1", "2"]},
        )
        print(await response.text())
    
        print("With files:")
        response = await treq.post(
            "https://httpbin.org/post",
            data={"form": ["1", "2"]},
            files={"foo": (os.devnull, open(os.devnull, "rb"))},
        )
        print(await response.text())
    
    
    react(main, [])
    

    When run:

    Without files:
    {
      "args": {}, 
      "data": "", 
      "files": {}, 
      "form": {
        "form": [
          "1", 
          "2"
        ]
      }, 
      "headers": {
        "Accept-Encoding": "gzip", 
        "Content-Length": "13", 
        "Content-Type": "application/x-www-form-urlencoded", 
        "Host": "httpbin.org", 
        "X-Amzn-Trace-Id": "Root=1-63a26dee-580ab8e77fdd228c1eb84266"
      }, 
      "json": null, 
      "origin": "99.187.228.202", 
      "url": "https://httpbin.org/post"
    }
    
    With files:
    main function encountered error
    Traceback (most recent call last):
      File ".../twisted/internet/defer.py", line 696, in callback
        self._startRunCallbacks(result)
      File ".../twisted/internet/defer.py", line 798, in _startRunCallbacks
        self._runCallbacks()
      File ".../twisted/internet/defer.py", line 892, in _runCallbacks
        current.result = callback(  # type: ignore[misc]
      File ".../twisted/internet/defer.py", line 1792, in gotResult
        _inlineCallbacks(r, gen, status, context)
    --- <exception caught here> ---
      File ".../twisted/internet/defer.py", line 1697, in _inlineCallbacks
        result = context.run(gen.send, result)
      File ".../demo.py", line 17, in main
        response = await treq.post(
      File ".../src/treq/api.py", line 32, in post
        return _client(kwargs).post(url, data=data, _stacklevel=4, **kwargs)
      File ".../src/treq/client.py", line 182, in post
        return self.request('POST', url, data=data, **kwargs)
      File ".../src/treq/client.py", line 244, in request
        bodyProducer, contentType = self._request_body(data, files, json,
      File ".../src/treq/client.py", line 379, in _request_body
        multipart.MultiPartProducer(data + files, boundary=boundary),
      File ".../src/treq/multipart.py", line 53, in __init__
        self._fields = list(_sorted_by_type(_converted(fields)))
      File ".../src/treq/multipart.py", line 350, in _sorted_by_type
        return sorted(fields, key=key)
      File ".../src/treq/multipart.py", line 260, in _converted
        raise ValueError(
    builtins.ValueError: Expected tuple: (filename, content type, producer)
    

    When only data is passed we URL-encode the request body with urllencode(..., doseq=True), whereas when both data and files are passed we combine them and pass the result to MultiPartProducer without unboxing sequences of values.

    bug 
    opened by twm 0
  • `cgi` is deprecated as of Python 3.11

    `cgi` is deprecated as of Python 3.11

    There are two uses of the cgi module in treq:

    1. One use of cgi.parse_header() in treq.contentsuggested replacement is email.message.Message
    2. One use of cgi.parse_multipart() in treq.test.test_multipart — same suggested replacement
    opened by twm 1
  • httpbin test dependency is unmaintained

    httpbin test dependency is unmaintained

    httpbin, which is used by treq's test suite (but not elsewhere), hasn't seen a release since 2018. The ecosystem of libraries it relies on is accumulating changes. Without maintenance it will eventually break.

    This first became a problem in #352, which relates to deprecated Werkzeug symbols. #353 pinned httpbin and werkzeug (which also fixed compatibility with markupsafe).

    Eventually we will have to vendor or replace httpbin. At that point we should also reconsider having so many integration tests — they are slow relative to the coverage they provide.

    opened by twm 0
  • How to get in touch regarding a security concern

    How to get in touch regarding a security concern

    Hey there!

    I belong to an open source security research community, and a member (@ranjit-git) has found an issue, but doesn’t know the best way to disclose it.

    If not a hassle, might you kindly add a SECURITY.md file with an email, or another contact method? GitHub recommends this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.

    Thank you for your consideration, and I look forward to hearing from you!

    (cc @huntr-helper)

    question 
    opened by JamieSlome 8
  • username/password in URLs

    username/password in URLs

    If a DecodedURL with a username/password in it is passed to a treq API, an error results about encoding a colon character (and the password is removed). (I can followup with the traceback but I don't have it immediately handy).

    To make such a URL work, one can simply pass auth=url.userinfo to the treq reqeuest (and .replace(userinfo=()) to remove the username/password from URL) so it might be nice to have that as "a feature" in treq itself. That is, take out the userinfo from the DecodedURL and use it for authentication.

    bug 
    opened by meejah 1
  • Eliminate dependency on requests

    Eliminate dependency on requests

    The treq dependency on requests seems to be causing issues for Klein and other packages that depend on idna

    https://twistedmatrix.com/pipermail/twisted-python/2021-February/065444.html

    bug 
    opened by graingert 1
Releases(release-22.2.0)
  • release-22.2.0(Feb 9, 2022)

  • release-22.1.0(Jan 30, 2022)

  • release-21.5.0(May 24, 2021)

    Features

    • PEP 517/518 build-system metadata is now provided in pyproject.toml. (#329)

    Bugfixes

    • treq.testing.StubTreq now persists twisted.web.server.Session instances between requests. (#327)

    Improved Documentation

    • The dependency on Sphinx required to build the documentation has been moved from the dev extra to the new docs extra. (#296)

    Deprecations and Removals

    • Support for Python 2.7 and 3.5 has been dropped. treq no longer depends on six or mock. (#318)
    Source code(tar.gz)
    Source code(zip)
    treq-21.5.0-py3-none-any.whl(61.35 KB)
    treq-21.5.0.tar.gz(69.65 KB)
  • release-21.1.0(Jan 15, 2021)

    Features

    • Support for Python 3.9: treq is now tested with CPython 3.9. (#305)
    • The auth parameter now accepts arbitrary text and bytes for usernames and passwords. Text is encoded as UTF-8, per RFC 7617. Previously only ASCII was allowed. (#268)
    • treq produces a more helpful exception when passed a tuple of the wrong size in the files parameter. (#299)

    Bugfixes

    • The params argument once more accepts non-ASCII bytes, fixing a regression first introduced in treq 20.4.1. (#303)
    • treq request APIs no longer mutates a http_headers.Headers passed as the headers parameter when the auth parameter is also passed. (#314)
    • The agent returned by treq.auth.add_auth() and treq.auth.add_basic_auth() is now marked to provide twisted.web.iweb.IAgent. (#312)
    • treq's package metadata has been updated to require six >= 1.13, noting a dependency introduced in treq 20.9.0. (#295)

    Improved Documentation

    • The documentation of the params argument has been updated to more accurately describe its type-coercion behavior. (#281)
    • The treq.auth module has been documented. (#313)

    Deprecations and Removals

    • Support for Python 2.7, which has reached end of support, is deprecated. This is the last release with support for Python 2.7. (#309)
    • Support for Python 3.5, which has reached end of support, is deprecated. This is the last release with support for Python 3.5. (#306)
    • Deprecate tolerance of non-string values when passing headers as a dict. They have historically been silently dropped, but will raise TypeError in the next treq release. Also deprecate passing headers other than dict, twisted.web.http_headers.Headers, or None. Historically falsy values like [] or () were accepted. (#294)
    • treq request functions and methods like treq.get() and HTTPClient.post() now issue a DeprecationWarning when passed unknown keyword arguments, rather than ignoring them. Mixing the json argument with files or data is also deprecated. These warnings will change to a TypeError in the next treq release. (#297)
    • The minimum supported Twisted version has increased to 18.7.0. Older versions are no longer tested in CI. (#307)
    Source code(tar.gz)
    Source code(zip)
    treq-21.1.0-py2.py3-none-any.whl(62.74 KB)
    treq-21.1.0.tar.gz(69.35 KB)
  • release-20.9.0(Sep 27, 2020)

  • release-20.9.0rc1(Sep 27, 2020)

    Features

    • The url parameter of :meth:HTTPClient.request() (and shortcuts like :meth:~HTTPClient.get()) now accept :class:hyperlink.DecodedURL and :class:hyperlink.URL in addition to :class:str and :class:bytes. (#212 <https://github.com/twisted/treq/issues/212>__)

    Improved Documentation

    • An example of sending and receiving JSON has been added. (#278 <https://github.com/twisted/treq/issues/278>__)
    Source code(tar.gz)
    Source code(zip)
  • release-20.4.1(Apr 17, 2020)

  • release-20.4.0(Apr 17, 2020)

    Features

    • Support for Python 3.8 and PyPy3: treq is now tested with these interpreters. (#271)

    Bugfixes

    • treq.client.HTTPClient.request() and its aliases no longer raise UnicodeEncodeError when passed a Unicode url and non-empty params. Now the URL and query parameters are concatenated as documented. (#264)
    • In treq 20.3.0 the params argument didn't accept parameter names or values that contain the characters & or #. Now these characters are properly escaped. (#282)

    Improved Documentation

    • The treq documentation has been revised to emphasize use of treq.client.HTTPClient over the module-level convenience functions in the treq module. (#276)
    Source code(tar.gz)
    Source code(zip)
    treq-20.4.0-py2.py3-none-any.whl(54.68 KB)
    treq-20.4.0.tar.gz(57.67 KB)
  • release-16.12.0(Dec 14, 2016)

Owner
Twisted Matrix Labs
Twisted Matrix Labs
Asynchronous Python HTTP Requests for Humans using Futures

Asynchronous Python HTTP Requests for Humans Small add-on for the python requests http library. Makes use of python 3.2's concurrent.futures or the ba

Ross McFarland 2k Dec 30, 2022
🔄 🌐 Handle thousands of HTTP requests, disk writes, and other I/O-bound tasks simultaneously with Python's quintessential async libraries.

?? ?? Handle thousands of HTTP requests, disk writes, and other I/O-bound tasks simultaneously with Python's quintessential async libraries.

Hackers and Slackers 15 Dec 12, 2022
A Python obfuscator using HTTP Requests and Hastebin.

?? Jawbreaker ?? Jawbreaker is a Python obfuscator written in Python3, using double encoding in base16, base32, base64, HTTP requests and a Hastebin-l

Billy 50 Sep 28, 2022
EasyRequests is a minimalistic HTTP-Request Library that wraps aiohttp and asyncio in a small package that allows for sequential, parallel or even single requests

EasyRequests EasyRequests is a minimalistic HTTP-Request Library that wraps aiohttp and asyncio in a small package that allows for sequential, paralle

Avi 1 Jan 27, 2022
Screaming-fast Python 3.5+ HTTP toolkit integrated with pipelining HTTP server based on uvloop and picohttpparser.

Screaming-fast Python 3.5+ HTTP toolkit integrated with pipelining HTTP server based on uvloop and picohttpparser.

Paweł Piotr Przeradowski 8.6k Jan 4, 2023
As easy as /aitch-tee-tee-pie/ 🥧 Modern, user-friendly command-line HTTP client for the API era. JSON support, colors, sessions, downloads, plugins & more. https://twitter.com/httpie

HTTPie: human-friendly CLI HTTP client for the API era HTTPie (pronounced aitch-tee-tee-pie) is a command-line HTTP client. Its goal is to make CLI in

HTTPie 25.4k Jan 1, 2023
A modern/fast python SOAP client based on lxml / requests

Zeep: Python SOAP client A fast and modern Python SOAP client Highlights: Compatible with Python 3.6, 3.7, 3.8 and PyPy Build on top of lxml and reque

Michael van Tellingen 1.7k Jan 1, 2023
Small, fast HTTP client library for Python. Features persistent connections, cache, and Google App Engine support. Originally written by Joe Gregorio, now supported by community.

Introduction httplib2 is a comprehensive HTTP client library, httplib2.py supports many features left out of other HTTP libraries. HTTP and HTTPS HTTP

null 457 Dec 10, 2022
A next generation HTTP client for Python. 🦋

HTTPX - A next-generation HTTP client for Python. HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support

Encode 9.8k Jan 5, 2023
Asynchronous HTTP client/server framework for asyncio and Python

Async http client/server framework Key Features Supports both client and server side of HTTP protocol. Supports both client and server Web-Sockets out

aio-libs 13.1k Jan 1, 2023
Aiosonic - lightweight Python asyncio http client

aiosonic - lightweight Python asyncio http client Very fast, lightweight Python asyncio http client Here is some documentation. There is a performance

Johanderson Mogollon 93 Jan 6, 2023
A minimal HTTP client. ⚙️

HTTP Core Do one thing, and do it well. The HTTP Core package provides a minimal low-level HTTP client, which does one thing only. Sending HTTP reques

Encode 306 Dec 27, 2022
A toolbelt of useful classes and functions to be used with python-requests

The Requests Toolbelt This is just a collection of utilities for python-requests, but don't really belong in requests proper. The minimum tested reque

null 892 Jan 6, 2023
Single-file replacement for python-requests

mureq mureq is a single-file, zero-dependency replacement for python-requests, intended to be vendored in-tree by Linux systems software and other lig

Shivaram Lingamneni 267 Dec 28, 2022
r - a small subset of Python Requests

r a small subset of Python Requests a few years ago, when I was first learning Python and looking for http functionality, i found the batteries-includ

Gabriel Sroka 4 Dec 15, 2022
Requests + Gevent = <3

GRequests: Asynchronous Requests GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily. Note: You should probably

Spencer Phillip Young 4.2k Dec 30, 2022
Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more.

urllib3 is a powerful, user-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. urllib3 brings many

urllib3 3.2k Dec 29, 2022
Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more.

urllib3 is a powerful, user-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. urllib3 brings many

urllib3 3.2k Jan 2, 2023
HTTP/2 for Python.

Hyper: HTTP/2 Client for Python This project is no longer maintained! Please use an alternative, such as HTTPX or others. We will not publish further

Hyper 1k Dec 23, 2022