Requests + Gevent = <3

Overview

GRequests: Asynchronous Requests

GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily.

version pyversions

Note: You should probably use requests-threads or requests-futures instead.

Usage

Usage is simple:

import grequests

urls = [
    'http://www.heroku.com',
    'http://python-tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://fakedomain/',
    'http://kennethreitz.com'
]

Create a set of unsent Requests:

>>> rs = (grequests.get(u) for u in urls)

Send them all at the same time:

>>> grequests.map(rs)
[<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, None, <Response [200]>]

Optionally, in the event of a timeout or any other exception during the connection of the request, you can add an exception handler that will be called with the request and exception inside the main thread:

>>> def exception_handler(request, exception):
...    print("Request failed")

>>> reqs = [
...    grequests.get('http://httpbin.org/delay/1', timeout=0.001),
...    grequests.get('http://fakedomain/'),
...    grequests.get('http://httpbin.org/status/500')]
>>> grequests.map(reqs, exception_handler=exception_handler)
Request failed
Request failed
[None, None, <Response [500]>]

For some speed/performance gains, you may also want to use imap instead of map. imap returns a generator of responses. Order of these responses does not map to the order of the requests you send out. The API for imap is equivalent to the API for map.

Installation

Installation is easy with pip:

$ pip install grequests
✨🍰✨
Comments
  • Requests v1.x.x compatibility. Fixes #28 #29

    Requests v1.x.x compatibility. Fixes #28 #29

    Makes grequests fully compatible with Requests v1.1.0

    • Adds callback parameter to get, post, etc, works as shortcut for hooks={'response': callback}.
    • Restores ability to pass Session object.
    • Adds basic tests.
    opened by reclosedev 28
  • gevent patch_all call in grequests breaks pymysql, etc.

    gevent patch_all call in grequests breaks pymysql, etc.

    I was looking forward to using this awesome package with my app and gevent until I noticed that it accomplishes gevent-compatibility via gevent.monkey.patch_all():

    try:
        import gevent
        from gevent import monkey as curious_george
        from gevent.pool import Pool
    except ImportError:
        raise RuntimeError('Gevent is required for grequests.')
    
    # Monkey-patch.
    curious_george.patch_all(thread=False, select=False)
    

    Unfortunately, patch_all() impacts all other packages/modules used by the app, breaking some of them that were not designed for reentrancy. For example, gevent.monkey.patch_all() broke the combination of pymysql/pooled_db (with sporadic, hard-to-reproduce failures); my app has several greenlets that on occasion make pymysql calls; I isolated the problem to gevent's monkey-patching of the socket module in that case.

    A local solution involving monkey-patching of just the requests package to use gevent's variants of blocking API's would greatly enhance compatibility of grequests with apps and eliminate unexpected/undesirable side-effects on other packages. For example, I recently monkey-patched HBase/Thrift via the following code snippet with the desired effect on Thrift and no harm to the rest of the app (pymysql, etc.):

    # Monkey-patch Thrift's TSocket to use gevent.socket to make our HBase interface
    # gevent-friendly
    import gevent.socket
    from thrift.transport import TSocket
    TSocket.socket = gevent.socket
    

    gevent-zeromq is another example of local monkey-patching, albeit a more complex one: https://github.com/traviscline/gevent-zeromq/blob/master/gevent_zeromq/init.py

    Thank you, Vitaly

    opened by vitaly-krugl 24
  • Improved Error Handling

    Improved Error Handling

    Resubmitting this due to a mistake I made in the branching. This builds off of this pull request here, so you will not want to merge both:

    https://github.com/kennethreitz/grequests/pull/15

    opened by brendon-16389 20
  • Grequests use of `monkey.patch_all(...)` breaks my application

    Grequests use of `monkey.patch_all(...)` breaks my application

    The aggressive python stdlib monkey patching originating from https://github.com/kennethreitz/grequests/blob/master/grequests.py#L21 causes a lot of breakage throughout my app. The degree of monkey patching causes the redis client to error out which in turn then breaks all async celery processing.

    I really like the grequests lib, but cannot use it if it's going to make things that used to work fine not work anymore.

      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/billiard/pool.py", line 1426, in safe_apply_callback
        fun(*args)
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/celery/worker/job.py", line 347, in on_success
        self.acknowledge()
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/celery/worker/job.py", line 453, in acknowledge
        self.on_ack(logger, self.connection_errors)
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/base.py", line 100, in ack_log_error
        self.ack()
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/base.py", line 95, in ack
        self.channel.basic_ack(self.delivery_tag)
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 511, in basic_ack
        self.qos.ack(delivery_tag)
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 127, in ack
        self._remove_from_indices(delivery_tag).execute()
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1919, in execute
        return execute(conn, stack, raise_on_error)
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1811, in _execute_transaction
        self.parse_response(connection, '_')
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1882, in parse_response
        self, connection, command_name, **options)
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 387, in parse_response
        response = connection.read_response()
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 307, in read_response
        response = self._parser.read_response()
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 105, in read_response
        response = self.read()
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 90, in read
        return self._fp.readline()[:-2]
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 447, in readline
        data = self._sock.recv(self._rbufsize)
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/socket.py", line 392, in recv
        self._wait(self._read_event)
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/socket.py", line 298, in _wait
        self.hub.wait(watcher)
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 341, in wait
        result = waiter.get()
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 568, in get
        return self.hub.switch()
      File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch
        return greenlet.switch(self)
    LoopExit: This operation would block forever
    

    Additional references pertaining to gevent monkey patching issues:

    • http://stackoverflow.com/questions/13040458/why-is-gevent-spawn-different-than-a-monkeypatched-threading-thread
    • https://github.com/jrief/django-websocket-redis/issues/3
    • http://bighow.net/5332403-Why_is__gevent_spawn__different_than_a_monkeypatched__threading_Thread____.html
    • http://stackoverflow.com/questions/9192539/using-gevent-monkey-patching-with-threading-makes-thread-work-serially
    • http://pastebin.com/TPYMvuVr
    :hear_no_evil::see_no_evil::speak_no_evil: gevent 
    opened by jaytaylor 14
  • suggestion: use concurrent.futures instead of gevent

    suggestion: use concurrent.futures instead of gevent

    Hi Kenneth,

    May I suggest that you consider using this instead of gevent:

    http://docs.python.org/dev/library/concurrent.futures.html

    It has a backport to previous versions

    http://pypi.python.org/pypi/futures

    It has a good interface IMO, it's in the standard library so will be supported indefinitely, it doesn't have external native dependencies like gevent does, and it can help avoid the problems stemming from gevent monkey patching as in issue 1. To be fair, gevent has awesome performance but it's just not as easy a dependency to work with as I'd like (I had to update to 1.0b2 to get pool.imap to work, but that version is not in pypi, etc.)

    opened by kislyuk 12
  • Cut a new 0.3.0 release :)

    Cut a new 0.3.0 release :)

    test fix for requests hook kwargs [1] is handy for packagers / porters (QA) , along with the test_suite and tests_require changes [2]

    [1] https://github.com/kennethreitz/grequests/commit/f50782ad63607b85e5e009f4a0a4ce0a8a6aef0d [2] https://github.com/kennethreitz/grequests/pull/47

    opened by koobs 10
  • Importing grequests after requests breaks requests

    Importing grequests after requests breaks requests

    >>> import requests
    >>> import grequests
    >>> requests.get('https://google.com')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/lib/python3.6/site-packages/requests/api.py", line 70, in get
        return request('get', url, params=params, **kwargs)
      File "/usr/lib/python3.6/site-packages/requests/api.py", line 56, in request
        return session.request(method=method, url=url, **kwargs)
      File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
        r = adapter.send(request, **kwargs)
      File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 423, in send
        timeout=timeout
      File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
        chunked=chunked)
      File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
        self._validate_conn(conn)
      File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 844, in _validate_conn
        conn.connect()
      File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 314, in connect
        cert_reqs=resolve_cert_reqs(self.cert_reqs),
      File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 264, in create_urllib3_context
        context.options |= options
      File "/usr/lib/python3.6/ssl.py", line 459, in options
        super(SSLContext, SSLContext).options.__set__(self, value)
      File "/usr/lib/python3.6/ssl.py", line 459, in options
        super(SSLContext, SSLContext).options.__set__(self, value)
      File "/usr/lib/python3.6/ssl.py", line 459, in options
        super(SSLContext, SSLContext).options.__set__(self, value)
      [Previous line repeated 323 more times]
    RecursionError: maximum recursion depth exceeded
    

    If the order of imports is reversed, the problem no longer occurs.

    :hear_no_evil::see_no_evil::speak_no_evil: gevent 
    opened by cauebs 9
  • Gevent failed during installation on MAC OS

    Gevent failed during installation on MAC OS

    Can't find anything that solves my problem, I don't know what else to do.

    Installing collected packages: gevent, grequests
      Running setup.py install for gevent
        Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-09c1Gh/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-0hQM3Z-record/install-record.txt --single-version-externally-managed --compile:
        running install
        running build
        running build_py
        creating build
        creating build/lib.macosx-10.10-intel-2.7
        creating build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/__init__.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/_ssl2.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/_sslgte279.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/_threading.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/backdoor.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/baseserver.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/coros.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/event.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/fileobject.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/greenlet.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/hub.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/local.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/lock.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/monkey.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/os.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/pool.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/pywsgi.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/queue.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/resolver_ares.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/resolver_thread.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/select.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/server.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/socket.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/ssl.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/subprocess.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/thread.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/threading.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/threadpool.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/timeout.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/util.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/win32util.py -> build/lib.macosx-10.10-intel-2.7/gevent
        copying gevent/wsgi.py -> build/lib.macosx-10.10-intel-2.7/gevent
        running build_ext
        Running '/bin/sh /private/tmp/pip-build-09c1Gh/gevent/libev/configure > configure-output.txt' in /private/tmp/pip-build-09c1Gh/gevent/build/temp.macosx-10.10-intel-2.7/libev
        building 'gevent.core' extension
        creating build/temp.macosx-10.10-intel-2.7/gevent
        cc -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -U__llvm__ -arch x86_64 -arch i386 -pipe -DLIBEV_EMBED=1 -DEV_COMMON= -DEV_CLEANUP_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_PERIODIC_ENABLE=0 -Ibuild/temp.macosx-10.10-intel-2.7/libev -Ilibev -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c gevent/gevent.core.c -o build/temp.macosx-10.10-intel-2.7/gevent/gevent.core.o
        In file included from gevent/gevent.core.c:249:
        In file included from gevent/libev.h:2:
        libev/ev.c:483:48: warning: '/*' within block comment [-Wcomment]
        /*#define MIN_INTERVAL  0.00000095367431640625 /* 1/2**20, good till 2200 */
                                                       ^
        libev/ev.c:1029:42: error: '_Noreturn' keyword must precede function declarator
          ecb_inline void ecb_unreachable (void) ecb_noreturn;
                                                 ^~~~~~~~~~~~
          _Noreturn
        libev/ev.c:832:26: note: expanded from macro 'ecb_noreturn'
          #define ecb_noreturn   _Noreturn
                                 ^
        libev/ev.c:1625:31: warning: 'extern' variable has an initializer [-Wextern-initializer]
          EV_API_DECL struct ev_loop *ev_default_loop_ptr = 0; /* needs to be initialised to make it a definition despite extern */
                                      ^
        libev/ev.c:1796:7: warning: unused variable 'ocur_' [-Wunused-variable]
              array_needsize (ANPENDING, pendings [pri], pendingmax [pri], w_->pending, EMPTY2);
              ^
        libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
              int ecb_unused ocur_ = (cur);                                     \
                             ^
        libev/ev.c:1807:3: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (W, rfeeds, rfeedmax, rfeedcnt + 1, EMPTY2);
          ^
        libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
              int ecb_unused ocur_ = (cur);                                     \
                             ^
        libev/ev.c:1934:7: warning: unused variable 'ocur_' [-Wunused-variable]
              array_needsize (int, fdchanges, fdchangemax, fdchangecnt, EMPTY2);
              ^
        libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
              int ecb_unused ocur_ = (cur);                                     \
                             ^
        In file included from gevent/gevent.core.c:249:
        In file included from gevent/libev.h:2:
        In file included from libev/ev.c:2484:
        libev/ev_kqueue.c:50:3: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (struct kevent, kqueue_changes, kqueue_changemax, kqueue_changecnt, EMPTY2);
          ^
        libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
              int ecb_unused ocur_ = (cur);                                     \
                             ^
        In file included from gevent/gevent.core.c:249:
        In file included from gevent/libev.h:2:
        In file included from libev/ev.c:2490:
        libev/ev_poll.c:66:7: warning: unused variable 'ocur_' [-Wunused-variable]
              array_needsize (struct pollfd, polls, pollmax, pollcnt, EMPTY2);
              ^
        libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
              int ecb_unused ocur_ = (cur);                                     \
                             ^
        libev/ev.c:3648:34: warning: '&' within '|' [-Wbitwise-op-parentheses]
          fd_change (EV_A_ fd, w->events & EV__IOFDSET | EV_ANFD_REIFY);
                               ~~~~~~~~~~^~~~~~~~~~~~~ ~
        libev/ev.c:3648:34: note: place parentheses around the '&' expression to silence this warning
          fd_change (EV_A_ fd, w->events & EV__IOFDSET | EV_ANFD_REIFY);
                                         ^
                               (                      )
        libev/ev.c:3687:3: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (ANHE, timers, timermax, ev_active (w) + 1, EMPTY2);
          ^
        libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
              int ecb_unused ocur_ = (cur);                                     \
                             ^
        libev/ev.c:4367:5: warning: unused variable 'ocur_' [-Wunused-variable]
            array_needsize (ev_idle *, idles [ABSPRI (w)], idlemax [ABSPRI (w)], active, EMPTY2);
            ^
        libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
              int ecb_unused ocur_ = (cur);                                     \
                             ^
        libev/ev.c:4407:3: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (ev_prepare *, prepares, preparemax, preparecnt, EMPTY2);
          ^
        libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
              int ecb_unused ocur_ = (cur);                                     \
                             ^
        libev/ev.c:4445:3: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (ev_check *, checks, checkmax, checkcnt, EMPTY2);
          ^
        libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
              int ecb_unused ocur_ = (cur);                                     \
                             ^
        libev/ev.c:4592:3: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (ev_fork *, forks, forkmax, forkcnt, EMPTY2);
          ^
        libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
              int ecb_unused ocur_ = (cur);                                     \
                             ^
        libev/ev.c:4675:3: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (ev_async *, asyncs, asyncmax, asynccnt, EMPTY2);
          ^
        libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
              int ecb_unused ocur_ = (cur);                                     \
                             ^
        14 warnings and 1 error generated.
        error: command 'cc' failed with exit status 1
    
        ----------------------------------------
    Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-09c1Gh/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-0hQM3Z-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-09c1Gh/gevent```
    
    :hear_no_evil::see_no_evil::speak_no_evil: gevent 
    opened by caioremedio 9
  • gevent's monkeypatching breaks timeouts on linux

    gevent's monkeypatching breaks timeouts on linux

    See kennethreitz/requests#500 for context, but basically the timeout is no longer respected after importing the async module that monkeypatches socket.

    >>> import requests
    >>> import time
    >>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "requests/api.py", line 52, in get
        return request('get', url, **kwargs)
      File "requests/api.py", line 40, in request
        return s.request(method=method, url=url, **kwargs)
      File "requests/sessions.py", line 229, in request
        r.send(prefetch=prefetch)
      File "requests/models.py", line 604, in send
        raise Timeout('Request timed out.')
    requests.exceptions.Timeout: Request timed out.
    >>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "requests/api.py", line 52, in get
        return request('get', url, **kwargs)
      File "requests/api.py", line 40, in request
        return s.request(method=method, url=url, **kwargs)
      File "requests/sessions.py", line 229, in request
        r.send(prefetch=prefetch)
      File "requests/models.py", line 604, in send
        raise Timeout('Request timed out.')
    requests.exceptions.Timeout: Request timed out.
    >>> import requests.async
    >>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
    4.38320612907 200
    >>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
    4.67993688583 200
    
    opened by slingamn 9
  • Ascync non-blocking?

    Ascync non-blocking?

    Is it possible to initialize a pool to return the result but continue the code execution like so?

    import grequests
    
    def print_res(res, *args, **kwargs):
        from pprint import pprint
        pprint (vars(res))
    
    print 'starting grequest'
    
    req = grequests.post('http://www.cricket.com.au/', hooks=dict(response=print_res))
    job = grequests.send(req, grequests.Pool(1))
    
    print 'request made'
    
    for i in range(10):
        print i
    

    So that would show the request was made but still print the range while the request was in process and then finally print out the results of the send?

    I'd really love to use grequests but the examples/documentation are fairly sparse so I'm struggling.

    Thanks for any help.

    opened by jakehilton 8
  • "Too many open files" error, regardless of size parameter

    I am using grequests to test the performance of my service. Both the service and the machine from which I start the requests are inside Amazon EC2. It seems that I cannot start more requests than ulimit -n, even though I explicitly specify the size parameter for map() / imap()

    My setup is roughly like this:

    headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
    params = [json.dumps({'url': urls[i]}) for i in urls]
    reqs = [grequests.post(SERVICE_URL, data=params[i % len(params)], headers=headers)
                for i in xrange(no_requests)]
    res = grequests.imap(reqs, size=200)
    for r in res:
        ...
        r.close()
    

    Doesn't the size parameter here tell grequests that it should issue no_requests, 200 at a time?

    opened by david-cliqz 8
  • add imap_enumerated

    add imap_enumerated

    Adds the imap_enumerated function.

    Like imap, but yields tuple of original request index (as it was provided in the requests parameter sequence) and response object.

    Unlike imap, failed results and responses from exception handlers that return None are not ignored. Instead, a tuple of (index, None) is yielded. Additionally, the requests parameter must be a sequence of Request objects (generators will be exhausted, but this may cause issues if the generator is infinite or otherwise very large).

    The index is merely the original index of the original request in the requests list and does NOT provide any indication of the order in which requests or responses are sent or received. Responses are still yielded in arbitrary order.

    Resolves #159

    opened by spyoungtech 0
  • Add index in imap

    Add index in imap

    Hi!, thx for this lib!, I was looking long time ago how to do this in a fast way.

    The actual imap function does not keep the order of map, is not necessary bad, but I think would be great if it can return the index. At least in my case, the data usually need more things than the request it self, and the index help to identify and relate information.

    Thx!

    enhancement 
    opened by latot 3
  • unexpected behavior of stream=True

    unexpected behavior of stream=True

    Python 3.8.3 (default, May 19 2020, 06:50:17) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32

    pip list
    Package                Version
    ---------------------- -------------------
    alembic                1.4.3
    apache-airflow         1.10.12
    apispec                1.3.3
    argcomplete            1.12.1
    attrs                  19.3.0
    Babel                  2.8.0
    cached-property        1.5.2
    cattrs                 1.0.0
    certifi                2020.4.5.1
    cffi                   1.14.0
    chardet                3.0.4
    click                  7.1.2
    colorama               0.4.4
    colorlog               4.0.2
    conda                  4.8.3
    conda-package-handling 1.7.0
    configparser           3.5.3
    croniter               0.3.35
    cryptography           2.9.2
    defusedxml             0.6.0
    dill                   0.3.2
    dnspython              2.0.0
    docutils               0.16
    email-validator        1.1.1
    Flask                  1.1.2
    Flask-Admin            1.5.4
    Flask-AppBuilder       2.3.4
    Flask-Babel            1.0.0
    Flask-Caching          1.3.3
    Flask-JWT-Extended     3.24.1
    Flask-Login            0.4.1
    Flask-OpenID           1.2.5
    Flask-SQLAlchemy       2.4.4
    flask-swagger          0.2.14
    Flask-WTF              0.14.3
    funcsigs               1.0.2
    future                 0.18.2
    gevent                 20.9.0
    graphviz               0.14.2
    greenlet               0.4.17
    grequests              0.6.0
    gunicorn               20.0.4
    idna                   2.9
    iso8601                0.1.13
    itsdangerous           1.1.0
    Jinja2                 2.11.2
    json-merge-patch       0.2
    jsonschema             3.2.0
    lazy-object-proxy      1.5.1
    lockfile               0.12.2
    Mako                   1.1.3
    Markdown               2.6.11
    MarkupSafe             1.1.1
    marshmallow            2.21.0
    marshmallow-enum       1.5.1
    marshmallow-sqlalchemy 0.24.0
    menuinst               1.4.16
    natsort                7.0.1
    Nuitka                 0.6.9.4
    numpy                  1.19.2
    pandas                 1.1.3
    pendulum               1.4.4
    pip                    20.0.2
    prison                 0.1.3
    psutil                 5.7.3
    pycosat                0.6.3
    pycparser              2.20
    Pygments               2.7.2
    PyJWT                  1.7.1
    pyOpenSSL              19.1.0
    pyotp                  2.4.1
    pyrsistent             0.17.3
    PySocks                1.7.1
    python-daemon          2.2.4
    python-dateutil        2.8.1
    python-editor          1.0.4
    python-nvd3            0.15.0
    python-slugify         4.0.1
    python3-openid         3.2.0
    pytz                   2020.1
    pytzdata               2020.1
    pywin32                227
    PyYAML                 5.3.1
    requests               2.23.0
    requests-toolbelt      0.9.1
    ruamel-yaml            0.15.87
    setproctitle           1.1.10
    setuptools             46.4.0.post20200518
    six                    1.14.0
    SQLAlchemy             1.3.20
    SQLAlchemy-JSONField   0.9.0
    SQLAlchemy-Utils       0.36.8
    tabulate               0.8.7
    tenacity               4.12.0
    text-unidecode         1.3
    thrift                 0.13.0
    tornado                6.0.4
    tqdm                   4.46.0
    tzlocal                1.5.1
    unicodecsv             0.14.1
    urllib3                1.25.8
    Werkzeug               0.16.1
    wheel                  0.34.2
    win-inet-pton          1.1.0
    wincertstore           0.2
    WTForms                2.3.3
    zope.deprecation       4.4.0
    zope.event             4.5.0
    zope.interface         5.1.2
    

    i'm trying to use grequests to check live stream ,i just expect a status_code 200,so i use stream = True and close it my code like :

    class ExInfoAdapter(HTTPAdapter): 
        def send(self, request, **kwargs):
            request.raw_info = request.headers.pop("data", None)
            return super(ExInfoAdapter, self).send(request, **kwargs)
    
    cdnsession = requests.session()
    cdnsession.headers.update(self.header)
    retries = Retry(total=2, backoff_factor=0, status_forcelist=[500, 502, 503, 504, 404], raise_on_redirect=False)
    cdnsession.mount('http://', ExInfoAdapter(max_retries=retries, pool_connections=200, pool_maxsize=400))
    cdnsession.mount('https://', ExInfoAdapter(max_retries=retries, pool_connections=200, pool_maxsize=400))
    
    for ...
    	greqs.append(grequests.get(
                        "http://{rip}/live/{zuid}.flv".format(rip=ip, zuid=zuid),
                        headers={"Host": host,
                                 "data": json.dumps( # this will pop out and never send to server
                                     {"rip": ip, "zuid": zuid, "domain": domain, "type": ctype})},
                        timeout=15,
                        session=cdnsession
                    ))
    def fqecp(request, exception):
        return [request, exception]
    
    resps = grequests.imap(greqs, stream=True, size=200, exception_handler=fqecp)
    
    for resp in resps:
        if isinstance(resp, list):  # handle err
            rd = json.loads(resp[0].kwargs["headers"].get("data", None))
            gerrs[rd['type']].append(rd)
        else:
            rd = json.loads(resp.request.raw_info)
            if resp.status_code != 200:
                gerrs[rd['type']].append(rd)
                print("non-200 : %s" % resp.status_code)
            else:
                pass
                # print("%s check ok" % resp.url)
            resp.close()
    
    

    it workd fine and prety faster than just requests,but by checking net usage,it seems like stream=True not effect image last 5 low: image it semms like keep download until close() . i tried requests in shell with stream=True,code like:

    import requests
    from requests.adapters import HTTPAdapter
    from requests.packages.urllib3.util.retry import Retry
    
    class ExInfoAdapter(HTTPAdapter):
        def send(self, request, **kwargs):
            request.raw_info = request.headers.pop("data", None)
            return super(ExInfoAdapter, self).send(request, **kwargs)
    
    s=requests.Session()
    retries = Retry(total=2, backoff_factor=0, status_forcelist=[500, 502, 503, 504, 404], raise_on_redirect=False)
    s.mount('http://', ExInfoAdapter(max_retries=retries, pool_connections=200, pool_maxsize=400))
    s.mount('https://', ExInfoAdapter(max_retries=retries, pool_connections=200, pool_maxsize=400))
    
    r=s.get("http://116.207.172.73/live/2358590143_1603765853.flv",headers={"Host": "v2.zb.marketing.i.mi.com"},timeout=15,stream=True)
    assert r.status_code == 200
    r.close()
    # this ip and url may unavilable when you try,please find another live stream or call me to get a new one
    

    here 's the result: image i tryedboth grequests and requests 2 times , grequests download avg 3MB(2.8and3.2) and requests download avg 322KB(321and323) 323KB is mach accessable but still a lot,i just want to chek http code ,it's still a lot where's my fault and how to resolve this issue?

    opened by 9268 0
  • RAM increase slowly

    RAM increase slowly

    first of all. thanks for your work. when I use the grequests the RAM will increase slowly. code such as :

    def exception_handler(request, exception):
        print("Request failed request:{} \n exception:{} ".format(request,exception))
    
    if __name__ == '__main__':
        task = []
        f_file= "./data_scp/3031_xiuxiu_coverImage_v1.dat"
    
        session = requests.session()
        with open(f_file,"r") as r_f:
            for i in r_f:
                tmp = i.strip("\n").split(",")
                url = tmp[-1]
                feed_id = tmp[0]
                rs = grequests.request("GET", url,session=session)
                task.append(rs)
    
        resp = grequests.imap(task, size=30,exception_handler=exception_handler)
    
        for i in resp:
            if i.status_code ==200:
                print(i.status_code)
    

    the 3031_xiuxiu_coverImage_v1.dat such as 6650058925696645684,http://***8.jpg 6650058925696645684,http://***8.jpg 6650058925696645684,http://***8.jpg 6650058925696645684,http://***8.jpg

    my grequest version is 0.4.0 . thanks in advance

    opened by Usernamezhx 6
Releases(v0.6.0)
Owner
Spencer Phillip Young
Software Engineer, consultant, Open Source enthusiast, Pythonista.
Spencer Phillip Young
A modern/fast python SOAP client based on lxml / requests

Zeep: Python SOAP client A fast and modern Python SOAP client Highlights: Compatible with Python 3.6, 3.7, 3.8 and PyPy Build on top of lxml and reque

Michael van Tellingen 1.7k Jan 1, 2023
A toolbelt of useful classes and functions to be used with python-requests

The Requests Toolbelt This is just a collection of utilities for python-requests, but don't really belong in requests proper. The minimum tested reque

null 892 Jan 6, 2023
Asynchronous Python HTTP Requests for Humans using Futures

Asynchronous Python HTTP Requests for Humans Small add-on for the python requests http library. Makes use of python 3.2's concurrent.futures or the ba

Ross McFarland 2k Dec 30, 2022
🔄 🌐 Handle thousands of HTTP requests, disk writes, and other I/O-bound tasks simultaneously with Python's quintessential async libraries.

?? ?? Handle thousands of HTTP requests, disk writes, and other I/O-bound tasks simultaneously with Python's quintessential async libraries.

Hackers and Slackers 15 Dec 12, 2022
A Python obfuscator using HTTP Requests and Hastebin.

?? Jawbreaker ?? Jawbreaker is a Python obfuscator written in Python3, using double encoding in base16, base32, base64, HTTP requests and a Hastebin-l

Billy 50 Sep 28, 2022
Single-file replacement for python-requests

mureq mureq is a single-file, zero-dependency replacement for python-requests, intended to be vendored in-tree by Linux systems software and other lig

Shivaram Lingamneni 267 Dec 28, 2022
EasyRequests is a minimalistic HTTP-Request Library that wraps aiohttp and asyncio in a small package that allows for sequential, parallel or even single requests

EasyRequests EasyRequests is a minimalistic HTTP-Request Library that wraps aiohttp and asyncio in a small package that allows for sequential, paralle

Avi 1 Jan 27, 2022
r - a small subset of Python Requests

r a small subset of Python Requests a few years ago, when I was first learning Python and looking for http functionality, i found the batteries-includ

Gabriel Sroka 4 Dec 15, 2022
The purpose of this bot is to take soundcloud track requests, that are posted in the stream-requests channel, and add them to a playlist, to make the process of scrolling through the requests easier for Root

Discord Song Collector Dont know if anyone is actually going to read this, but the purpose of this bot is to check every message in the stream-request

null 2 Mar 1, 2022
Mr. Queue - A distributed worker task queue in Python using Redis & gevent

MRQ MRQ is a distributed task queue for python built on top of mongo, redis and gevent. Full documentation is available on readthedocs Why? MRQ is an

Pricing Assistant 871 Dec 25, 2022
Official repository for gevent-socketio

Presentation gevent-socketio is a Python implementation of the Socket.IO protocol, developed originally for Node.js by LearnBoost and then ported to o

Alexandre Bourget 1.2k Dec 12, 2022
Simple reproduction of connection leak with celery/django/gevent

Redis connection leak with celery/django/gevent Reproduces celery issue at https://github.com/celery/celery/issues/6819 using gevented django web serv

null 2 Apr 3, 2022
Multiple-requests-poster - A tool to send multiple requests to a particular website written in Python

Multiple-requests-poster - A tool to send multiple requests to a particular website written in Python

RLX 2 Feb 14, 2022
Python requests like API built on top of Twisted's HTTP client.

treq: High-level Twisted HTTP Client API treq is an HTTP library inspired by requests but written on top of Twisted's Agents. It provides a simple, hi

Twisted Matrix Labs 553 Dec 18, 2022
A mocking library for requests

httmock A mocking library for requests for Python 2.7 and 3.4+. Installation pip install httmock Or, if you are a Gentoo user: emerge dev-python/httm

Patryk Zawadzki 452 Dec 28, 2022
A utility for mocking out the Python Requests library.

Responses A utility library for mocking out the requests Python library. Note Responses requires Python 2.7 or newer, and requests >= 2.0 Installing p

Sentry 3.8k Jan 3, 2023
A mocking library for requests

httmock A mocking library for requests for Python 2.7 and 3.4+. Installation pip install httmock Or, if you are a Gentoo user: emerge dev-python/httm

Patryk Zawadzki 452 Dec 28, 2022
A utility for mocking out the Python Requests library.

Responses A utility library for mocking out the requests Python library. Note Responses requires Python 2.7 or newer, and requests >= 2.0 Installing p

Sentry 3.8k Jan 2, 2023
A modern/fast python SOAP client based on lxml / requests

Zeep: Python SOAP client A fast and modern Python SOAP client Highlights: Compatible with Python 3.6, 3.7, 3.8 and PyPy Build on top of lxml and reque

Michael van Tellingen 1.7k Jan 1, 2023
A toolbelt of useful classes and functions to be used with python-requests

The Requests Toolbelt This is just a collection of utilities for python-requests, but don't really belong in requests proper. The minimum tested reque

null 892 Jan 6, 2023