• Fixes new check appeared in mypy 0.620
• Also fixes usage of generic type aliases in some places (see below)

Things I've noticed:

• MultiDictProxy.copy and CIMultiDictProxy.copy had wrong return type (fixed)
• MultiDictProxy is inherited from a different class in .pyx and .py implementation
• Coundn't enable disallow_any_generics = True in mypy config because there are a lot of errors like No library stub file for module 'pytest' (see travis). Not sure what to do with them.
• Update: Using _S as a type name in MultiDict[_S, _T] confuses _S with a generic argument (TypeVar), while it's just an alias for Union. Probably better name it _Str or alike.

fixes: https://github.com/aio-libs/multidict/issues/78 fixes when sending multidicts across processes

testcase:

import asyncio
from concurrent.futures import ProcessPoolExecutor
from aiohttp.web import Response
from multidict import CIMultiDict


class JSONResponseException(Exception):
    def __init__(self, http_status_code:int, payload_object=None, response_headers: CIMultiDict = None):
        self.__http_status_code = http_status_code
        self.__payload_object = payload_object if payload_object is not None else {}
        self.__response_headers = response_headers if response_headers is not None else CIMultiDict()

    @property
    def http_status_code(self):
        return self.__http_status_code

    @property
    def payload_object(self):
        return self.__payload_object

    @property
    def response_headers(self):
        return self.__response_headers


def foo():
    err = JSONResponseException(404)
    raise err

async def main(loop):
    with ProcessPoolExecutor() as executor:
        try:
            await loop.run_in_executor(executor, foo)
        except JSONResponseException as e:
            return Response(body=b'',
                            headers=e.response_headers,
                            status=e.http_status_code,
                            content_type='application/json')

if __name__ == '__main__':
    _loop = asyncio.get_event_loop()
    _loop.run_until_complete(main(_loop))

Summary Installing multidict on aarch64 via pip using command "pip3 install multidict" tries to build wheel from source code

Problem description multidict don't have wheel for aarch64 on PyPI repository. So, while installing multidict via pip on aarch64, pip builds wheel for same resulting in it takes more time to install multidict. Making wheel available for aarch64 will benefit aarch64 users by minimizing multidict installation time.

Expected Output Pip should be able to download multidict wheel from PyPI repository rather than building it from source code.

@multidict-taem, please let me know if I can help you building wheel/uploading to PyPI repository. I am curious to make multidict wheel available for aarch64. It will be a great opportunity for me to work with you.

It seems like the recent version of multidict brakes aiohttp somehow.

Server:

from aiohttp import web

async def hello(request):
    return web.json_response(await request.json())

app = web.Application()
app.add_routes([web.post('/', hello)])
web.run_app(app)

Client:

import asyncio
import aiohttp

async def foo(times):
    data = {'foo': 1}
    async with aiohttp.ClientSession() as session:
        for x in range(times):
            resp = await session.post('http://localhost:8080', json=data)
            if not x % 100:
                print(await resp.json())

loop = asyncio.get_event_loop()
loop.run_until_complete(foo(100000))
loop.close()

Just after 100000 cycles this simple server takes 171M of ram and doesn't release it. 4.4.2 works like a charm.

python 3.6.4 aiohttp 3.4.4 but should affect other versions multidict>=4.0,<5.0

As per https://hynek.me/articles/testing-packaging/ when testing --editable (develop) install you are not testing the package installed same way as it will be installed in by users.

@asvetlov what do you think about such approach? Sounds reasonable to me.

Describe the bug

This will resolve #79 and #101

To Reproduce

script

rm -rf venv
virtualenv venv -ppython3.6
venv/bin/pip install multidict
venv/bin/python -Werror -c 'import multidict'

output

...
Successfully installed multidict-4.4.2
$ venv/bin/python -Werror -c 'import multidict'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/t/venv/lib/python3.6/site-packages/multidict/__init__.py", line 23, in <module>
    from ._multidict import (MultiDictProxy,
  File "multidict/_multidict.pyx", line 9, in init multidict._multidict
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

Expected behavior

(silence)

Logs/tracebacks

see above

Your version of the Python

$ ./venv/bin/python --version --version
Python 3.6.6 (default, Sep 12 2018, 18:26:19) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]]

Your version of the multidict distribution

see above

Additional context

The current wheel is built against 0.28.5:

$ grep Generated !$
grep Generated venv/lib/python3.6/site-packages/multidict/_multidict.c
/* Generated by Cython 0.28.5 */

Building from source against 0.28.5 produces the error:

rm -rf venv
virtualenv venv -ppython3.6
venv/bin/pip install cython==0.28.5
git -C multidict clean -fxfd
venv/bin/pip install ./multidict
venv/bin/python -Werror -c 'import multidict'

...
$ venv/bin/python -Werror -c 'import multidict'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/t/venv/lib/python3.6/site-packages/multidict/__init__.py", line 23, in <module>
    from ._multidict import (MultiDictProxy,
  File "multidict/_multidict.pyx", line 9, in init multidict._multidict
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

Building from source against 0.29.0:

rm -rf venv
virtualenv venv -ppython3.6
venv/bin/pip install cython==0.29.0
git -C multidict clean -fxfd
venv/bin/pip install ./multidict
venv/bin/python -Werror -c 'import multidict'

...
$ venv/bin/python -Werror -c 'import multidict'
$ 

proper silence!

Changes:

• merged test_instantiate__from_arg0 and test_instantiate__from_arg0_dict
• tests from _BaseMutableMultiDictTests are at TestMutableMultiDict
• TestNonProxyCIMultiDict#test_extend_with_istr was moved to TestMutableMultiDict
• _TestProxy/_TestCIProxy#test_copy was moved to test_proxy_copy

Questions:

• test_instantiate__with_kwargs: should we rely on order and drop sorted?
• test_instantiate__empty: is duplication done by purpose?
• test_items__contains: is duplication done by purpose?
• test_get duplicates test_getone
• _CIMultiDictTests was inherited from _Root and not _BaseTest
• test__iter__types was done only for non-ci dicts. Should I add CI classes?

I'm attempting to build a distribution package from the source on PyPi. However, the source contains the compiled shared library _istr.cpython-34m.so.

This causes problems if the arch and Python version don't match up with the library. Manually removing the file from the source solves the problem but it doesn't seem like this should exist in the source package?

I also noticed the source on PyPi does not match up with the source on Github. Github does not contain the _multidict.c file. Is this also a build artifact that was compiled from the pyx file?

The url I'm pulling source from is: https://files.pythonhosted.org/packages/source/m/multidict/multidict-%{version}.tar.gz

$ python -We -c "import multidict as x; print(x.__version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages/multidict/__init__.py", line 24, in <module>
    from ._multidict import (MultiDictProxy,
  File "multidict/_multidict.pyx", line 1, in init multidict._multidict (multidict/_multidict.c:14817)
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

Describe the bug

After upgrading to multidict 4.7.1 we have found that the applications we bundle using PyInstaller crash. Actually, I discovered this problem because we use aiohttp which depends on multidict. Version 4.6.1 does not have this problem.

To Reproduce

1. Create a new virtual environment
2. pip install multidict==4.7.1 pyinstaller
3. Create a simple test.py script with:

print("Hello!")
import multidict

4. Bundle it pyinstaller test.py
5. Run it:

./dist/test/test
Hello!
[1]    22548 segmentation fault (core dumped)  ./dist/test/test

Expected behavior

Application prints "Hello!" and quits nicely.

Logs/tracebacks

Provided above.

Your version of the Python

Observed with 3.6.9 and 3.7.5

Your version of the multidict distribution

4.7.1

Additional context

N/A

This attempts to make tox a single source of truth for all the test envs (mainly CI, but it wires it into GNU make as well.

I didn't test deployment flow yet, but current results look promising in terms of clearly separating different envs/tasks + there's a nice ~10% speedup in Travis.

Bumps sphinx from 5.3.0 to 6.1.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

What do these changes do?

Adds build-backend to pyproject.toml so that there's no warnings from pip when building from source (+ so that modern setuptools backend is used).

Are there changes in behavior for the user?

pip will no longer show warning:

  WARNING: Missing build requirements in pyproject.toml for multidict==6.0.4 from https://files.pythonhosted.org/packages/4a/15/bd620f7a6eb9aa5112c4ef93e7031bcd071e0611763d8e17706ef8ba65e0/multidict-6.0.4.tar.gz.
  WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'wheel'.

Related issue number

None

Checklist

• [x] I think the code is well written
• [ ] Unit tests for the changes exist
• [ ] Documentation reflects the changes
• [ ] Add a new news fragment into the CHANGES folder
  • name it <issue_id>.<type> (e.g. 588.bugfix)
  • if you don't have an issue_id change it to the pr id after creating the PR
  • ensure type is one of the following:
    • .feature: Signifying a new feature.
    • .bugfix: Signifying a bug fix.
    • .doc: Signifying a documentation improvement.
    • .removal: Signifying a deprecation or removal of public API.
    • .misc: A ticket has been closed, but it is not of interest to users.
  • Make sure to use full sentences with correct case and punctuation, for example: Fix issue with non-ascii contents in doctest text files.

Hi there,

Thanks for this library. I'd like to ask for a convenience method that output a dict of a multidict - with lists for multi-values. Currently I have this:

    def dict(self) -> Dict[str, List[Any]]:
        """

        Returns:
            A dict of lists
        """
        return {k: self.getall(k) for k in set(self.keys)}

While this works, its not very performant.

The idea is: statically cache the most common HTTP headers.

For example, aiohttp.hdrs can do the following:

CONTENT_LENGTH = istr("Content-Length")
multidict.add_static_cache(str(CONTENT_LENGTH), CONTENT_LENGTH)
multidict.add_static_cache(str(CONTENT_LENGTH).lower(), CONTENT_LENGTH)
multidict.add_static_cache(str(CONTENT_LENGTH).upper(), CONTENT_LENGTH)

Since Python 3.8 keys view is reversible (but Mapping/MutableMapping ABCs are not for the sake of backward compatibility).

It would be nice to support reversibility in multidict keys views as well.

Multidict is used mostly for storing primitive types as keys and values: str, int, float, None.

PyObject_GC_Track should be called only if a key or value doesn't belong to a primitive type.

BTW, now PyObject_GC_Track is not called for multidicts and proxies which is an error.