Asynchronous Python HTTP Requests for Humans using twisted

Pierre Tardy

Last update: Oct 27, 2021

Related tags

HTTP Clients txrequests

Overview

Asynchronous Python HTTP Requests for Humans

https://travis-ci.org/tardyp/txrequests.png?branch=master

Small add-on for the python requests http library. Makes use twisted's ThreadPool, so that the requests'API returns deferred

The additional API and changes are minimal and strives to avoid surprises.

The following synchronous code:

from requests import Session

session = Session()
# first requests starts and blocks until finished
response_one = session.get('http://httpbin.org/get')
# second request starts once first is finished
response_two = session.get('http://httpbin.org/get?foo=bar')
# both requests are complete
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)

Can be translated to make use of futures, and thus be asynchronous by creating a FuturesSession and catching the returned Future in place of Response. The Response can be retrieved by calling the result method on the Future:

from txrequests import Session
from twisted.internet import defer

@defer.inlineCallbacks
def main():
    # use with statement to cleanup session's threadpool, and connectionpool after use
    # you can also use session.close() if want to use session for long term use
    with Session() as session:
        # first request is started in background
        d1 = session.get('http://httpbin.org/get')
        # second requests is started immediately
        d2 = session.get('http://httpbin.org/get?foo=bar')
        # wait for the first request to complete, if it hasn't already
        response_one = yield d1
        print('response one status: {0}'.format(response_one.status_code))
        print(response_one.content)
        # wait for the second request to complete, if it hasn't already
        response_two = yield d2
        print('response two status: {0}'.format(response_two.status_code))
        print(response_two.content)

By default a ThreadPool is created with 4 max workers. If you would like to adjust that value or share a threadpool across multiple sessions you can provide one to the Session constructor.

from twisted.python.threadpool import ThreadPool
from txrequests import Session

session = FuturesSession(pool=ThreadPool(maxthreads=10))
# ...

As a shortcut in case of just increasing workers number you can pass minthreads and/or maxthreads straight to the Session constructor:

from txrequests import Session
session = Session(maxthreads=10)

That's it. The api of requests.Session is preserved without any modifications beyond returning a Deferred rather than Response. As with all futures exceptions are shifted to the deferred errback.

Working in the Background

There is one additional parameter to the various request functions, background_callback, which allows you to work with the Response objects in the background thread. This can be useful for shifting work out of the foreground, for a simple example take json parsing.

from pprint import pprint
from txrequests import Session
from twisted.internet import defer

@defer.inlineCallbacks
def main():
    with Session() as session:

        def bg_cb(sess, resp):
            # parse the json storing the result on the response object
            resp.data = resp.json()
            return resp

        d = session.get('http://httpbin.org/get', background_callback=bg_cb)
        # do some other stuff, send some more requests while this one works
        response = yield d
        print('response status {0}'.format(response.status_code))
        # data will have been attached to the response object in the background
        pprint(response.data)

Installation

pip install txrequests

Credits

txrequests is based on requests_future, from Ross McFarland

Comments

Readme examples deadlock

Python 3.6, Ubuntu 16.04

I've just added main() at the bottom of both code samples in the README and neither of them print anything: on keyboard interrupt it seems they're waiting at a lock.acquire. The serial version works fine.

opened by clbarnes 1
sessions.py: pep8 + put deferred as argument

This commit transfers the deferred used in request using an argument of the function, instead of relying on the dynamic state of the scope of the function.

opened by yetanotherion 0
sessions.py: pep8 + put deferred as argument

This commit transfers the deferred used in request using an argument of the function, instead of relying on the dynamic state of the scope of the functions.

opened by yetanotherion 0
example hangs

using python3 (3.7.0) on a mac. I tried the example in the Readme file and it hangs while executing the first yield operation: I put some print statements to confirm. It doesn't print line 13

` from txrequests import Session from twisted.internet import defer

@defer.inlineCallbacks def main(): with Session() as session: print ('AAA') d1 = session.get('http://httpbin.org/get') print ('BBB') d2 = session.get('http://httpbin.org/get?foo=bar') print ('CCC') response_one = yield d1 print('response one status: {0}'.format(response_one.status_code)) print(response_one.content) # wait for the second request to complete, if it hasn't already response_two = yield d2 print('response two status: {0}'.format(response_two.status_code)) print(response_two.content)

main()`

opened by jcastill0 0
Stop pool only after shutdown

People often add their cleanup code via addSystemTrigger('before', 'shutdown', callback). Sometimes this cleanup code contains additional requests. We should therefore only close the thread pool at a later stage.

Thanks for this package :)

opened by jdemaeyer 0