Multidict is dict-like collection of key-value pairs where key might be occurred more than once in the container.

Overview

multidict

GitHub status for master branch Coverage metrics PyPI Documentationb Python versions Chat on Gitter

Multidict is dict-like collection of key-value pairs where key might be occurred more than once in the container.

Introduction

HTTP Headers and URL query string require specific data structure: multidict. It behaves mostly like a regular dict but it may have several values for the same key and preserves insertion ordering.

The key is str (or istr for case-insensitive dictionaries).

multidict has four multidict classes: MultiDict, MultiDictProxy, CIMultiDict and CIMultiDictProxy.

Immutable proxies (MultiDictProxy and CIMultiDictProxy) provide a dynamic view for the proxied multidict, the view reflects underlying collection changes. They implement the collections.abc.Mapping interface.

Regular mutable (MultiDict and CIMultiDict) classes implement collections.abc.MutableMapping and allows to change their own content.

Case insensitive (CIMultiDict and CIMultiDictProxy) ones assume the keys are case insensitive, e.g.:

>>> dct = CIMultiDict(key='val')
>>> 'Key' in dct
True
>>> dct['Key']
'val'

Keys should be str or istr instances.

The library has optional C Extensions for sake of speed.

License

Apache 2

Library Installation

$ pip install multidict

The library is Python 3 only!

PyPI contains binary wheels for Linux, Windows and MacOS. If you want to install multidict on another operation system (or Alpine Linux inside a Docker) the Tarball will be used to compile the library from sources. It requires C compiler and Python headers installed.

To skip the compilation please use MULTIDICT_NO_EXTENSIONS environment variable, e.g.:

$ MULTIDICT_NO_EXTENSIONS=1 pip install multidict

Please note, Pure Python (uncompiled) version is about 20-50 times slower depending on the usage scenario!!!

Changelog

See RTD page.

Comments
  • Mypy 0.620 and generic type aliases

    Mypy 0.620 and generic type aliases

    • Fixes new check appeared in mypy 0.620
    • Also fixes usage of generic type aliases in some places (see below)

    Things I've noticed:

    • MultiDictProxy.copy and CIMultiDictProxy.copy had wrong return type (fixed)
    • MultiDictProxy is inherited from a different class in .pyx and .py implementation
    • Coundn't enable disallow_any_generics = True in mypy config because there are a lot of errors like No library stub file for module 'pytest' (see travis). Not sure what to do with them.
    • Update: Using _S as a type name in MultiDict[_S, _T] confuses _S with a generic argument (TypeVar), while it's just an alias for Union. Probably better name it _Str or alike.
    opened by tailhook 30
  • fix pickling of multidicts

    fix pickling of multidicts

    fixes: https://github.com/aio-libs/multidict/issues/78 fixes when sending multidicts across processes

    testcase:

    import asyncio
    from concurrent.futures import ProcessPoolExecutor
    from aiohttp.web import Response
    from multidict import CIMultiDict
    
    
    class JSONResponseException(Exception):
        def __init__(self, http_status_code:int, payload_object=None, response_headers: CIMultiDict = None):
            self.__http_status_code = http_status_code
            self.__payload_object = payload_object if payload_object is not None else {}
            self.__response_headers = response_headers if response_headers is not None else CIMultiDict()
    
        @property
        def http_status_code(self):
            return self.__http_status_code
    
        @property
        def payload_object(self):
            return self.__payload_object
    
        @property
        def response_headers(self):
            return self.__response_headers
    
    
    def foo():
        err = JSONResponseException(404)
        raise err
    
    async def main(loop):
        with ProcessPoolExecutor() as executor:
            try:
                await loop.run_in_executor(executor, foo)
            except JSONResponseException as e:
                return Response(body=b'',
                                headers=e.response_headers,
                                status=e.http_status_code,
                                content_type='application/json')
    
    if __name__ == '__main__':
        _loop = asyncio.get_event_loop()
        _loop.run_until_complete(main(_loop))
    
    opened by thehesiod 23
  • Wheel support for linux aarch64

    Wheel support for linux aarch64

    Summary Installing multidict on aarch64 via pip using command "pip3 install multidict" tries to build wheel from source code

    Problem description multidict don't have wheel for aarch64 on PyPI repository. So, while installing multidict via pip on aarch64, pip builds wheel for same resulting in it takes more time to install multidict. Making wheel available for aarch64 will benefit aarch64 users by minimizing multidict installation time.

    Expected Output Pip should be able to download multidict wheel from PyPI repository rather than building it from source code.

    @multidict-taem, please let me know if I can help you building wheel/uploading to PyPI repository. I am curious to make multidict wheel available for aarch64. It will be a great opportunity for me to work with you.

    enhancement 
    opened by odidev 18
  • multidict 4.5 memory leak

    multidict 4.5 memory leak

    It seems like the recent version of multidict brakes aiohttp somehow.

    Server:

    from aiohttp import web
    
    async def hello(request):
        return web.json_response(await request.json())
    
    app = web.Application()
    app.add_routes([web.post('/', hello)])
    web.run_app(app)
    

    Client:

    import asyncio
    import aiohttp
    
    async def foo(times):
        data = {'foo': 1}
        async with aiohttp.ClientSession() as session:
            for x in range(times):
                resp = await session.post('http://localhost:8080', json=data)
                if not x % 100:
                    print(await resp.json())
    
    loop = asyncio.get_event_loop()
    loop.run_until_complete(foo(100000))
    loop.close()
    

    Just after 100000 cycles this simple server takes 171M of ram and doesn't release it. 4.4.2 works like a charm.

    python 3.6.4 aiohttp 3.4.4 but should affect other versions multidict>=4.0,<5.0

    bug 
    opened by byashimov 15
  • Consider moving package to `src` subfolder

    Consider moving package to `src` subfolder

    As per https://hynek.me/articles/testing-packaging/ when testing --editable (develop) install you are not testing the package installed same way as it will be installed in by users.

    @asvetlov what do you think about such approach? Sounds reasonable to me.

    opened by webknjaz 14
  • Rebuild multidict using cython 0.29.0

    Rebuild multidict using cython 0.29.0

    Describe the bug

    This will resolve #79 and #101

    To Reproduce

    script

    rm -rf venv
    virtualenv venv -ppython3.6
    venv/bin/pip install multidict
    venv/bin/python -Werror -c 'import multidict'
    

    output

    ...
    Successfully installed multidict-4.4.2
    $ venv/bin/python -Werror -c 'import multidict'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/t/venv/lib/python3.6/site-packages/multidict/__init__.py", line 23, in <module>
        from ._multidict import (MultiDictProxy,
      File "multidict/_multidict.pyx", line 9, in init multidict._multidict
    ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
    

    Expected behavior

    (silence)

    Logs/tracebacks

    see above

    Your version of the Python

    $ ./venv/bin/python --version --version
    Python 3.6.6 (default, Sep 12 2018, 18:26:19) 
    [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]]
    

    Your version of the multidict distribution

    see above

    Additional context

    The current wheel is built against 0.28.5:

    $ grep Generated !$
    grep Generated venv/lib/python3.6/site-packages/multidict/_multidict.c
    /* Generated by Cython 0.28.5 */
    

    Building from source against 0.28.5 produces the error:

    rm -rf venv
    virtualenv venv -ppython3.6
    venv/bin/pip install cython==0.28.5
    git -C multidict clean -fxfd
    venv/bin/pip install ./multidict
    venv/bin/python -Werror -c 'import multidict'
    
    ...
    $ venv/bin/python -Werror -c 'import multidict'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/t/venv/lib/python3.6/site-packages/multidict/__init__.py", line 23, in <module>
        from ._multidict import (MultiDictProxy,
      File "multidict/_multidict.pyx", line 9, in init multidict._multidict
    ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
    

    Building from source against 0.29.0:

    rm -rf venv
    virtualenv venv -ppython3.6
    venv/bin/pip install cython==0.29.0
    git -C multidict clean -fxfd
    venv/bin/pip install ./multidict
    venv/bin/python -Werror -c 'import multidict'
    
    ...
    $ venv/bin/python -Werror -c 'import multidict'
    $ 
    

    proper silence!

    bug 
    opened by asottile 13
  • Refactor unit tests

    Refactor unit tests

    Changes:

    • merged test_instantiate__from_arg0 and test_instantiate__from_arg0_dict
    • tests from _BaseMutableMultiDictTests are at TestMutableMultiDict
    • TestNonProxyCIMultiDict#test_extend_with_istr was moved to TestMutableMultiDict
    • _TestProxy/_TestCIProxy#test_copy was moved to test_proxy_copy

    Questions:

    • test_instantiate__with_kwargs: should we rely on order and drop sorted?
    • test_instantiate__empty: is duplication done by purpose?
    • test_items__contains: is duplication done by purpose?
    • test_get duplicates test_getone
    • _CIMultiDictTests was inherited from _Root and not _BaseTest
    • test__iter__types was done only for non-ci dicts. Should I add CI classes?
    opened by akhomchenko 13
  • Pre-compiled shared libraries

    Pre-compiled shared libraries

    I'm attempting to build a distribution package from the source on PyPi. However, the source contains the compiled shared library _istr.cpython-34m.so.

    This causes problems if the arch and Python version don't match up with the library. Manually removing the file from the source solves the problem but it doesn't seem like this should exist in the source package?

    I also noticed the source on PyPi does not match up with the source on Github. Github does not contain the _multidict.c file. Is this also a build artifact that was compiled from the pyx file?

    The url I'm pulling source from is: https://files.pythonhosted.org/packages/source/m/multidict/multidict-%{version}.tar.gz

    opened by smarlowucf 12
  • Importing multidict raises ImportWarning on 3.6

    Importing multidict raises ImportWarning on 3.6

    $ python -We -c "import multidict as x; print(x.__version__)"
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages/multidict/__init__.py", line 24, in <module>
        from ._multidict import (MultiDictProxy,
      File "multidict/_multidict.pyx", line 1, in init multidict._multidict (multidict/_multidict.c:14817)
    ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
    
    opened by pitrou 12
  • multidict 4.7.1 causes segfault when bundled with PyInstaller

    multidict 4.7.1 causes segfault when bundled with PyInstaller

    Describe the bug

    After upgrading to multidict 4.7.1 we have found that the applications we bundle using PyInstaller crash. Actually, I discovered this problem because we use aiohttp which depends on multidict. Version 4.6.1 does not have this problem.

    To Reproduce

    1. Create a new virtual environment
    2. pip install multidict==4.7.1 pyinstaller
    3. Create a simple test.py script with:
    print("Hello!")
    import multidict
    
    1. Bundle it pyinstaller test.py
    2. Run it:
    ./dist/test/test
    Hello!
    [1]    22548 segmentation fault (core dumped)  ./dist/test/test
    

    Expected behavior

    Application prints "Hello!" and quits nicely.

    Logs/tracebacks

    Provided above.

    Your version of the Python

    Observed with 3.6.9 and 3.7.5

    Your version of the multidict distribution

    4.7.1

    Additional context

    N/A

    opened by gmarull 11
  • Tox based tests

    Tox based tests

    This attempts to make tox a single source of truth for all the test envs (mainly CI, but it wires it into GNU make as well.

    I didn't test deployment flow yet, but current results look promising in terms of clearly separating different envs/tasks + there's a nice ~10% speedup in Travis.

    opened by webknjaz 10
  • Bump sphinx from 5.3.0 to 6.1.1

    Bump sphinx from 5.3.0 to 6.1.1

    Bumps sphinx from 5.3.0 to 6.1.1.

    Release notes

    Sourced from sphinx's releases.

    v6.1.1

    Changelog: https://www.sphinx-doc.org/en/master/changes.html

    v6.1.0

    Changelog: https://www.sphinx-doc.org/en/master/changes.html

    v6.0.1

    Changelog: https://www.sphinx-doc.org/en/master/changes.html

    v6.0.0

    Changelog: https://www.sphinx-doc.org/en/master/changes.html

    v6.0.0b2

    Changelog: https://www.sphinx-doc.org/en/master/changes.html

    v6.0.0b1

    Changelog: https://www.sphinx-doc.org/en/master/changes.html

    Changelog

    Sourced from sphinx's changelog.

    Release 6.1.1 (released Jan 05, 2023)

    Bugs fixed

    • #11091: Fix util.nodes.apply_source_workaround for literal_block nodes with no source information in the node or the node's parents.

    Release 6.1.0 (released Jan 05, 2023)

    Dependencies

    Incompatible changes

    • #10979: gettext: Removed support for pluralisation in get_translation. This was unused and complicated other changes to sphinx.locale.

    Deprecated

    • sphinx.util functions:

      • Renamed sphinx.util.typing.stringify() to sphinx.util.typing.stringify_annotation()
      • Moved sphinx.util.xmlname_checker() to sphinx.builders.epub3._XML_NAME_PATTERN

      Moved to sphinx.util.display:

      • sphinx.util.status_iterator
      • sphinx.util.display_chunk
      • sphinx.util.SkipProgressMessage
      • sphinx.util.progress_message

      Moved to sphinx.util.http_date:

      • sphinx.util.epoch_to_rfc1123
      • sphinx.util.rfc1123_to_epoch

      Moved to sphinx.util.exceptions:

      • sphinx.util.save_traceback

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies autosquash 
    opened by dependabot[bot] 1
  • Specify the build system in pyproject.toml

    Specify the build system in pyproject.toml

    What do these changes do?

    Adds build-backend to pyproject.toml so that there's no warnings from pip when building from source (+ so that modern setuptools backend is used).

    Are there changes in behavior for the user?

    pip will no longer show warning:

      WARNING: Missing build requirements in pyproject.toml for multidict==6.0.4 from https://files.pythonhosted.org/packages/4a/15/bd620f7a6eb9aa5112c4ef93e7031bcd071e0611763d8e17706ef8ba65e0/multidict-6.0.4.tar.gz.
      WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'wheel'.
    

    Related issue number

    None

    Checklist

    • [x] I think the code is well written
    • [ ] Unit tests for the changes exist
    • [ ] Documentation reflects the changes
    • [ ] Add a new news fragment into the CHANGES folder
      • name it <issue_id>.<type> (e.g. 588.bugfix)
      • if you don't have an issue_id change it to the pr id after creating the PR
      • ensure type is one of the following:
        • .feature: Signifying a new feature.
        • .bugfix: Signifying a bug fix.
        • .doc: Signifying a documentation improvement.
        • .removal: Signifying a deprecation or removal of public API.
        • .misc: A ticket has been closed, but it is not of interest to users.
      • Make sure to use full sentences with correct case and punctuation, for example: Fix issue with non-ascii contents in doctest text files.
    bot:chronographer:provided 
    opened by Jackenmen 3
  • Enhancement: `to_dict` method

    Enhancement: `to_dict` method

    Hi there,

    Thanks for this library. I'd like to ask for a convenience method that output a dict of a multidict - with lists for multi-values. Currently I have this:

        def dict(self) -> Dict[str, List[Any]]:
            """
    
            Returns:
                A dict of lists
            """
            return {k: self.getall(k) for k in set(self.keys)}
    

    While this works, its not very performant.

    opened by Goldziher 1
  • Add a sketch for static keys cache

    Add a sketch for static keys cache

    The idea is: statically cache the most common HTTP headers.

    For example, aiohttp.hdrs can do the following:

    CONTENT_LENGTH = istr("Content-Length")
    multidict.add_static_cache(str(CONTENT_LENGTH), CONTENT_LENGTH)
    multidict.add_static_cache(str(CONTENT_LENGTH).lower(), CONTENT_LENGTH)
    multidict.add_static_cache(str(CONTENT_LENGTH).upper(), CONTENT_LENGTH)
    
    opened by asvetlov 0
  • Make keys view reversible

    Make keys view reversible

    Since Python 3.8 keys view is reversible (but Mapping/MutableMapping ABCs are not for the sake of backward compatibility).

    It would be nice to support reversibility in multidict keys views as well.

    opened by asvetlov 0
  • Don't track multidict if it contains untrackable objects only

    Don't track multidict if it contains untrackable objects only

    Multidict is used mostly for storing primitive types as keys and values: str, int, float, None.

    PyObject_GC_Track should be called only if a key or value doesn't belong to a primitive type.

    BTW, now PyObject_GC_Track is not called for multidicts and proxies which is an error.

    opened by asvetlov 1
Releases(v6.0.4)
Owner
aio-libs
The set of asyncio-based libraries built with high quality
aio-libs
Python library for doing things with Grid-like structures

gridthings Python library for doing things with Grid-like structures Development This project uses poetry for dependency management, pre-commit for li

Matt Kafonek 2 Dec 21, 2021
Python Sorted Container Types: Sorted List, Sorted Dict, and Sorted Set

Python Sorted Containers Sorted Containers is an Apache2 licensed sorted collections library, written in pure-Python, and fast as C-extensions. Python

Grant Jenks 2.8k Jan 4, 2023
A Python Covid-19 cases tracker that scrapes data off the web and presents the number of Cases, Recovered Cases, and Deaths that occurred because of the pandemic.

A Python Covid-19 cases tracker that scrapes data off the web and presents the number of Cases, Recovered Cases, and Deaths that occurred because of the pandemic.

Alex Papadopoulos 1 Nov 13, 2021
Key Cast - Cast your key presses and mouse clicks on the screen, while casting your favorite application on the screen. Better than the rest.

Key Cast Screen cast your keyboard and mouse clicks in style Project Homepage » View Demo · Report Bug · Request Feature Table of Contents Introductio

Mehul Singh Teya 13 Dec 23, 2022
A simple key-based text encryption process that encrypts a string based in a list of characteres pairs.

Simple Cipher Encrypter About | New Features | Exemple | How To Use | License ℹ️ About A simple key-based text encryption process that encrypts a stri

Guilherme Farrel 1 Oct 21, 2021
ClutterDB - Extremely simple JSON database made for infrequent changes which behaves like a dict

extremely simple JSON database made for infrequent changes which behaves like a dict this was made for ClutterBot

Clutter Development 1 Jan 12, 2022
Some Python scripts that fx(hash) users might find useful.

fx_hash_utils Some Python scripts that fx(hash) users might find useful. get_images This script downloads all the static images of the tokens generate

null 30 Oct 5, 2022
Words_And_Phrases - Just a repo for useful words and phrases that might come handy in some scenarios. Feel free to add yours

Words_And_Phrases Just a repo for useful words and phrases that might come handy in some scenarios. Feel free to add yours Abbreviations Abbreviation

Subhadeep Mandal 1 Feb 1, 2022
TarkovScrappy - A nifty little bot that lets you know if a queried item might be required for a quest at some point in the land of Tarkov!

TarkovScrappy A nifty little bot that lets you know if a queried item might be required for a quest at some point in the land of Tarkov! Hideout items

Joshua Smeda 2 Apr 11, 2022
Smaller, easier, more powerful, and more reliable than make. An implementation of djb's redo.

redo - a recursive build system Smaller, easier, more powerful, and more reliable than make. This is an implementation of Daniel J. Bernstein's redo b

null 1.7k Jan 4, 2023
Much faster than SORT(Simple Online and Realtime Tracking), a little worse than SORT

QSORT QSORT(Quick + Simple Online and Realtime Tracking) is a simple online and realtime tracking algorithm for 2D multiple object tracking in video s

Yonghye Kwon 8 Jul 27, 2022
If you are in allot of groups or channel and you would like to leave them at once use this

Telegram-auto-leave-groups If you are in allot of groups or channel and you would like to leave them at once use this USER GUIDE ?? Insert your telegr

Julius Njoroge 4 Jan 3, 2023
Waydroid is a container-based approach to boot a full Android system on a regular GNU/Linux system like Ubuntu.

Waydroid is a container-based approach to boot a full Android system on a regular GNU/Linux system like Ubuntu.

WayDroid 4.7k Jan 8, 2023
pickleDB is an open source key-value store using Python's json module.

pickleDB pickleDB is lightweight, fast, and simple database based on the json module. And it's BSD licensed! pickleDB is Fun >>> import pickledb >>>

Harrison Erd 738 Jan 4, 2023
Segcache: a memory-efficient and scalable in-memory key-value cache for small objects

Segcache: a memory-efficient and scalable in-memory key-value cache for small objects This repo contains the code of Segcache described in the followi

TheSys Group @ CMU CS 78 Jan 7, 2023
This is a package that allows you to create a key-value vault for storing variables in a global context

This is a package that allows you to create a key-value vault for storing variables in a global context. It allows you to set up a keyring with pre-defined constants which act as keys for the vault. These constants are then what is stored inside the vault. A key is just a string, but the value that the key is mapped to can be assigned to any type of object in Python. If the object is serializable (like a list or a dict), it can also be writen to a JSON file You can then use a decorator to annotate functions that you want to have use this vault to either store return variables in or to extract variables to be used as input for the function.

Data Ductus 2 Dec 14, 2022
AWS Tags As A Database is a Python library using AWS Tags as a Key-Value database.

AWS Tags As A Database is a Python library using AWS Tags as a Key-Value database. This database is completely free* ??

Oren Leung 42 Nov 25, 2022
Oh-My-PickleDB is an open source key-value store using Python's json module.

OH-MY-PICKLEDB oh-my-pickleDB is a lightweight, fast, and intuitive data manager written in python ?? Table of Contents About Getting Started Deployme

Adrián Toral 6 Feb 20, 2022