See https://github.com/kennethreitz/requests/issues/2006

This currently breaks and i can't explain how it breaks since the same code is used in requests.

At the moment this will not work because the AppEngineManager does not have the same signature as urllib3's PoolManager.

Related to: https://github.com/kennethreitz/requests/issues/1905#issuecomment-159380704

cc @Lukasa @shazow @jonparrott @webmaven @jacobg

The goal here is to allow the following code to work:

from requests_toolbelt.adapters import appengine
appengine.monkeypatch()

...and then run all your libraries that have dependencies on requests and sessions buried deep inside, all working properly with URLFetch on Appengine.

The code doesn't work as-is, as it depends on the following change: https://github.com/shazow/urllib3/pull/763 ...which then would probably need to be released and ported into the requests library as well...not sure how long that takes or what the process is.

This also is tangentially related to the following issues relating to getting requests playing nicely with AppEngine's URLFetch: https://github.com/kennethreitz/requests/issues/1905 https://github.com/shazow/urllib3/issues/618

We have a use case for this in which, due to an issue with AppEngine's dev_appserver.py, certain certificates will fail validation; more on that issue here. The only real workaround when using the emulator is to disable certificate validation.

Because this is a large code base, in which requests is used both explicitly and within third-party libraries, we are using the monkeypatch; however, to my eye there's no way to set the validate_certificate kwarg in the constructor for AppEngineAdapter to True when monkeypatching; if we could, it would serve our use case by allowing global disabling of certificate validation when a local environment is detected.

This creates a new AppEngineInsecureAdapter in which cert validation is disabled by default and adds a kwarg to monkeypatch() to trigger its use. Initially I tried to simply make the monkeypatch set partial(AppEngineAdapter, validate_certificate=False) but that was a bit naive as the adapter is not just instantiated.

This just provides a basic way to wrap a file object so you can force a specific amount of data to be returned at a minimum.

So far, basic tests at the interactive console have shown it to work as expected. This still needs real tests, however.

Closes #75

Fixes #38

The usecase for this is unsized file-like objects like sys.stdin.

This functionality could've been implemented as a classmethod or helper function. However, when using StreamingIterator with file-like objects, it would've been quite inefficient to

1.) Read data from the underlying file at a fixed chunk size for the iterator 2.) Then maintain a buffer for the file-like interface provided on top of that iterator

An adapter to be used in cases where a user needs to authenticate to a service using a .p12/.pfx certificate and it is not desirable or feasible to first convert that certificate into a .pem format. Born out of a use case in which the existence of temporary files containing cert data were deemed too great a risk.

Per investigation in: https://github.com/mitsuhiko/flask/issues/1025

"MultipartEncoder.read() is completely broken - read(8192) (default blocksize) returns less than 8192 bytes even though there is more left."

Following on the suggestion in requests, I've implemented the suggestion. Please let me know if there's anything more you'd like to see to get this accepted. I use this technique all the time and find the code copy/pasted in several places and I'd like to consolidate it somewhere.

Thanks.

Closes #145.

Another plausibe alternative here would be to change this line in urllib3's App Engine adapter from return timeout.total to something like return timeout.total or timeout._read or timeout._connect. I'm not sure of the semantics or conventions for requests vs urllib3, so I'll defer to you all.

cc @mikelambert

Edit: I realize this may have nothing to do with requests-toolbelt, it's just the only lead I have at the moment.

This is an odd issue that I just discovered with PRAW's testing suite as travis recently started failing almost all requests. I have narrowed the issue down to it failing only when .eggs/requests_toolbelt-0.4.0-py3.4.egg does not previously exist. On subsequent runs, when that file already exists, the tests pass just fine.

Below is a sample run starting with (1) it working, (2) removing the egg and observing the failure, (3) running again and seeing the success:

(tmp)bboe@spock:praw (updates)$ python setup.py test -s tests.test_wiki_page.WikiPageTests.test_revision_by
running test
Searching for mock>=1.0.0
Best match: mock 1.0.1
Processing mock-1.0.1-py3.4.egg

Using /Users/bboe/src/praw-dev/praw/.eggs/mock-1.0.1-py3.4.egg
Searching for betamax-matchers>=0.2.0
Best match: betamax-matchers 0.2.0
Processing betamax_matchers-0.2.0-py3.4.egg

Using /Users/bboe/src/praw-dev/praw/.eggs/betamax_matchers-0.2.0-py3.4.egg
Searching for betamax>=0.4.2
Best match: betamax 0.4.2
Processing betamax-0.4.2-py3.4.egg

Using /Users/bboe/src/praw-dev/praw/.eggs/betamax-0.4.2-py3.4.egg
Searching for requests-toolbelt>=0.4.0
Best match: requests-toolbelt 0.4.0
Processing requests_toolbelt-0.4.0-py3.4.egg

Using /Users/bboe/src/praw-dev/praw/.eggs/requests_toolbelt-0.4.0-py3.4.egg
running egg_info
writing entry points to praw.egg-info/entry_points.txt
writing dependency_links to praw.egg-info/dependency_links.txt
writing requirements to praw.egg-info/requires.txt
writing praw.egg-info/PKG-INFO
writing top-level names to praw.egg-info/top_level.txt
reading manifest file 'praw.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'praw/tests/files'
writing manifest file 'praw.egg-info/SOURCES.txt'
running build_ext
test_revision_by (tests.test_wiki_page.WikiPageTests) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.009s

OK
(tmp)bboe@spock:praw (updates)$ rm .eggs/requests_toolbelt-0.4.0-py3.4.egg 
(tmp)bboe@spock:praw (updates)$ python setup.py test -s tests.test_wiki_page.WikiPageTests.test_revision_by
running test
Searching for mock>=1.0.0
Best match: mock 1.0.1
Processing mock-1.0.1-py3.4.egg

Using /Users/bboe/src/praw-dev/praw/.eggs/mock-1.0.1-py3.4.egg
Searching for betamax-matchers>=0.2.0
Best match: betamax-matchers 0.2.0
Processing betamax_matchers-0.2.0-py3.4.egg

Using /Users/bboe/src/praw-dev/praw/.eggs/betamax_matchers-0.2.0-py3.4.egg
Searching for betamax>=0.4.2
Best match: betamax 0.4.2
Processing betamax-0.4.2-py3.4.egg

Using /Users/bboe/src/praw-dev/praw/.eggs/betamax-0.4.2-py3.4.egg
Searching for requests-toolbelt>=0.4.0
Reading https://pypi.python.org/simple/requests-toolbelt/
Best match: requests-toolbelt 0.4.0
Downloading https://pypi.python.org/packages/source/r/requests-toolbelt/requests-toolbelt-0.4.0.tar.gz#md5=2278d650faadf181dd180682591e5926
Processing requests-toolbelt-0.4.0.tar.gz
Writing /var/folders/zq/1jxg9xbx211_syhlv6cl2jq40000gn/T/easy_install-kowpbdbs/requests-toolbelt-0.4.0/setup.cfg
Running requests-toolbelt-0.4.0/setup.py -q bdist_egg --dist-dir /var/folders/zq/1jxg9xbx211_syhlv6cl2jq40000gn/T/easy_install-kowpbdbs/requests-toolbelt-0.4.0/egg-dist-tmp-avhtypx4
no previously-included directories found matching '*.pyc'
warning: manifest_maker: MANIFEST.in, line 6: 'recursive-include' expects <dir> <pattern1> <pattern2> ...

warning: manifest_maker: MANIFEST.in, line 7: 'recursive-include' expects <dir> <pattern1> <pattern2> ...

no previously-included directories found matching 'docs/_build'
zip_safe flag not set; analyzing archive contents...
Copying requests_toolbelt-0.4.0-py3.4.egg to /Users/bboe/src/praw-dev/praw/.eggs

Installed /Users/bboe/src/praw-dev/praw/.eggs/requests_toolbelt-0.4.0-py3.4.egg
running egg_info
writing top-level names to praw.egg-info/top_level.txt
writing requirements to praw.egg-info/requires.txt
writing entry points to praw.egg-info/entry_points.txt
writing dependency_links to praw.egg-info/dependency_links.txt
writing praw.egg-info/PKG-INFO
reading manifest file 'praw.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'praw/tests/files'
writing manifest file 'praw.egg-info/SOURCES.txt'
running build_ext
test_revision_by (tests.test_wiki_page.WikiPageTests) ... ERROR

======================================================================
ERROR: test_revision_by (tests.test_wiki_page.WikiPageTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bboe/src/praw-dev/praw/tests/helper.py", line 120, in betamax_function
    return function(obj)
  File "/Users/bboe/src/praw-dev/praw/tests/test_wiki_page.py", line 64, in test_revision_by
    self.subreddit.get_wiki_pages()))
  File "/Users/bboe/src/praw-dev/praw/praw/decorators.py", line 60, in wrapped
    return function(self.reddit_session, self, *args, **kwargs)
  File "/Users/bboe/src/praw-dev/praw/praw/decorators.py", line 345, in wrapped
    return function(cls, *args, **kwargs)
  File "/Users/bboe/src/praw-dev/praw/praw/__init__.py", line 1052, in get_wiki_pages
    six.text_type(subreddit))
  File "/Users/bboe/src/praw-dev/praw/praw/decorators.py", line 170, in wrapped
    return_value = function(reddit_session, *args, **kwargs)
  File "/Users/bboe/src/praw-dev/praw/praw/__init__.py", line 569, in request_json
    retry_on_error=retry_on_error)
  File "/Users/bboe/src/praw-dev/praw/praw/__init__.py", line 413, in _request
    response = handle_redirect()
  File "/Users/bboe/src/praw-dev/praw/praw/__init__.py", line 383, in handle_redirect
    timeout=timeout, **kwargs)
  File "/Users/bboe/src/praw-dev/praw/praw/handlers.py", line 138, in wrapped
    if _cache_key in cls.cache:
TypeError: unhashable type: 'RequestsCookieJar'

----------------------------------------------------------------------
Ran 1 test in 0.007s

FAILED (errors=1)
(tmp)bboe@spock:praw (updates)$ python setup.py test -s tests.test_wiki_page.WikiPageTests.test_revision_by
running test
Searching for mock>=1.0.0
Best match: mock 1.0.1
Processing mock-1.0.1-py3.4.egg

Using /Users/bboe/src/praw-dev/praw/.eggs/mock-1.0.1-py3.4.egg
Searching for betamax-matchers>=0.2.0
Best match: betamax-matchers 0.2.0
Processing betamax_matchers-0.2.0-py3.4.egg

Using /Users/bboe/src/praw-dev/praw/.eggs/betamax_matchers-0.2.0-py3.4.egg
Searching for betamax>=0.4.2
Best match: betamax 0.4.2
Processing betamax-0.4.2-py3.4.egg

Using /Users/bboe/src/praw-dev/praw/.eggs/betamax-0.4.2-py3.4.egg
Searching for requests-toolbelt>=0.4.0
Best match: requests-toolbelt 0.4.0
Processing requests_toolbelt-0.4.0-py3.4.egg

Using /Users/bboe/src/praw-dev/praw/.eggs/requests_toolbelt-0.4.0-py3.4.egg
running egg_info
writing requirements to praw.egg-info/requires.txt
writing entry points to praw.egg-info/entry_points.txt
writing dependency_links to praw.egg-info/dependency_links.txt
writing top-level names to praw.egg-info/top_level.txt
writing praw.egg-info/PKG-INFO
reading manifest file 'praw.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'praw/tests/files'
writing manifest file 'praw.egg-info/SOURCES.txt'
running build_ext
test_revision_by (tests.test_wiki_page.WikiPageTests) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.009s

OK

I really have no idea what to make of this at the moment. Any thoughts? Thanks!

MultipartEncoder can't handle mmap.mmap object correctly.

MultipartEncoder can't handle mmap.mmap object correctly. For example,

from requests_toolbelt import MultipartEncoder
import mmap

m = MultipartEncoder(
    fields={
            'field': ('filename', mmap.mmap(-1, 10))}
    )

m.read()

it will fall into an infinity loop in m.read().

I review those source code, find that the coerce_data function did't consider about the mmap.mmap object. Then the total_len function work different from other io object (always return the origin size). Then the bytes_left_to_write function always return True, fall into the infinity loop.

In my opinion, judge the mmap.mmap object by isinstance(data, mmap.mmap) or hasattr(data, 'madvise'), or other method?

Now, I fix my code by manually wrapped with FileWrapper, as

from requests_toolbelt.multipart.encoder import MultipartEncoder, FileWrapper
import mmap

m = MultipartEncoder(
    fields={
            'field': ('filename', FileWrapper(mmap.mmap(-1, 10)))}
    )

m.read()

I have little experience to pull requests. Could someone fix it from this package?

If returning a list-like object, MarkLogic can return a multipart document, one part per returned value. For an empty list, it returns zero parts (i.e. no bytes). Requests Toolbelt crashes in this instance, because the assumption that a content type must exist at all is violated.

File /usr/local/lib/python3.9/site-packages/caselawclient/Client.py:29 in decode_multipart
multipart_data = decoder.MultipartDecoder.from_response(response)
File /usr/local/lib/python3.9/site-packages/requests_toolbelt/multipart/decoder.py:156 in from_response
return cls(content, content_type, encoding)
File /usr/local/lib/python3.9/site-packages/requests_toolbelt/multipart/decoder.py:110 in __init__
self._find_boundary()
File /usr/local/lib/python3.9/site-packages/requests_toolbelt/multipart/decoder.py:114 in _find_boundary
ct_info = tuple(x.strip() for x in self.content_type.split(';'))
AttributeError: 'NoneType' object has no attribute 'split'

We got around this by hardcoding that no content means zero items returned, but this feels like a thing worth solving at the library level. our workaround: https://github.com/nationalarchives/ds-caselaw-custom-api-client/pull/99/files

I added documentation for the multipart.decoder.MultipartDecoder class. Created a new page on the documentation called "Receiveing and Handling Multipart Data". I added detailed instruction on how to use the parameters for the base initialization of the class.

On the basic usage examples, I inverted it to show the base initialization first, since this is a more generalized way of using this class. The .from_response method is for a specific use case, so I put it in the end.

There isn't a documentation page for the multipart submodule. The MultipartEncoder class is mentioned at times, but I didn't find any mentions to the MultipartDecoder class.

I have the intentions of adding this documentation myself. Will add a new page called "multipart", with the usage guide for both classes mentioned above. Opening the issue to let maintainer know I'm working on this.

As far as I understand, if SourceAddressAdapter is used with an (IP, PORT) tuple and the port is other than 0, there is no way this adapter can work without passing

pool_connections=1,
pool_maxsize=1,
pool_block=True,

to the underlying HTTPAdapter.

Shouldn't this either

• be forced by the implementation
• raise a warning if this exact values are not specified by the user
• write more documentation about it

?