Reading the docks I saw that this is not multiprocess safe; would it be sufficient if a user passed in their own multiprocessing safe lock object to the chest? Either way, this could be a convenient feature.

Filenames are stored along with keys, making old chests more portable. _keys, which was a set, is replaced with a {key -> filename} dictionary. The dictionary is dumped/loaded as a list of tuples for json compatibility, but used as a dict internally.

cache_many loads a list of keys to the in-cache memory.

open_many take a list of filenames and returns a list of open files. This is mostly to support overloading of open.

c1.eat(c2) flushes c2, moves c2's files into c1's directory, and adds c2's keys to c1. By default, keys in c2 overwrite the same keys in c1, but that behavior can be controlled by overwrite=False.

Allows a user to set a maximum disk usage and a policy defining what to do when the max disk is used and a new entry must be added.

The two options are currently:

1. raise_: raise an OSError indicating you ran out of space.
2. pop_lru: rotate the least recently used element out of the chest.

Users may pass any callable here; however, these two are defined for them.

Example

    In [1]: from chest import Chest

    In [2]: c = Chest(path='foo')

    In [3]: c['one'] = 1

    In [4]: c['one', 'two'] = 12

    In [5]: c['one', 'three'] = 13

    In [6]: c.flush()

    In [7]: !tree foo
    foo
    ├── one
    └── _one
        ├── three
        └── two

        1 directory, 3 files

cc @maxhutch

When prefetching, the list of keys to load are chunked up to prevent open_many from opening too many files simultaneously.

On my system, the user file limit was 1024, so I set it to 512.

This should resolve #14.

For a reasonably large chest c, c.prefetch(list(c.keys())) will cause OSError: [Errno 24] Too many open files because open_many opens everything, consumes everything, and then closes everything.

Simple solution: chunk up prefetches into open_many's of some maximum size.

The memory_usage property is now a statefull field that we track during operation. This is error prone but faster for cases where we churn disk frequently.

This just checks if dump has been called (by seeing if self.path exists). It dump was called, then delete the contents of self.inmem but don't move them to disk or write keys.

This gets rid of the exceptions that were briefly mentioned in #4

Python 3.3 and above moved the abstract base classes to their own module under collections, collections.abc. Import from that location, falling back if required for Python 3.2.

Fixes #29

 /opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/chest/core.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
151
    from collections import MutableMapping

Would you accept a PR fix, and make a new release?

Hi,

I tried chest on a large file and on an ubuntu 16.04 system. Chest created millions of files on my hard disk and I was running out of inodes. I have plenty of hard disk avaiable but I can not use it because inodes usage is at the peak. What can I do about it? Can I reduce the millions of generated files to hundereds? Is there any option for this?

Thanks in advance

I am trying to modify the protocol used for dumping, however, it always complains that it does not understand the argument protocol in dump=partial(pickle.dump, protocol=pickle.HIGHEST_PROTOCOL). Could you put this as default?

Hello, this is an auto-generated Pull Request. (Feedback?)

Some time ago, pypip.in shut down. This broke the badges for a bunch of repositories, including chest. Thankfully, an equivalent service is run by shields.io. This pull request changes the badges to use shields.io instead.

Unfortunately, PyPI has removed download statistics from their API, which means that even the shields.io "download count" badges are broken (they display "no longer available". See this). So those badges should really be removed entirely. Since this is an automated process (and trying to automatically remove the badges from READMEs can be tricky), this pull request just replaces the URL with the shields.io syntax.

I am trying to save the resulting chest into a different file format than the one used for caching.

A practical example: I used pickle when I initialize my chest because I am using nested dicts and sets, which would be stripped off I guess if it was to be stored in json.

But in the end, I would like to dump my chest into json for export.

Is there a way to do that?

For example, this does not work past the first level:

import ujson as json
with open('db.json', 'w') as f:
    f.write(json.dumps(out, ensure_ascii=False, indent=4, sort_keys=True))