serialize all of python

Related tags

dill
Overview

dill

serialize all of python

About Dill

dill extends python's pickle module for serializing and de-serializing python objects to the majority of the built-in python types. Serialization is the process of converting an object to a byte stream, and the inverse of which is converting a byte stream back to a python object hierarchy.

dill provides the user the same interface as the pickle module, and also includes some additional features. In addition to pickling python objects, dill provides the ability to save the state of an interpreter session in a single command. Hence, it would be feasable to save a interpreter session, close the interpreter, ship the pickled file to another computer, open a new interpreter, unpickle the session and thus continue from the 'saved' state of the original interpreter session.

dill can be used to store python objects to a file, but the primary usage is to send python objects across the network as a byte stream. dill is quite flexible, and allows arbitrary user defined classes and functions to be serialized. Thus dill is not intended to be secure against erroneously or maliciously constructed data. It is left to the user to decide whether the data they unpickle is from a trustworthy source.

dill is part of pathos, a python framework for heterogeneous computing. dill is in active development, so any user feedback, bug reports, comments, or suggestions are highly appreciated. A list of issues is located at https://github.com/uqfoundation/dill/issues, with a legacy list maintained at https://uqfoundation.github.io/pathos-issues.html.

Major Features

dill can pickle the following standard types:

  • none, type, bool, int, long, float, complex, str, unicode,
  • tuple, list, dict, file, buffer, builtin,
  • both old and new style classes,
  • instances of old and new style classes,
  • set, frozenset, array, functions, exceptions

dill can also pickle more 'exotic' standard types:

  • functions with yields, nested functions, lambdas
  • cell, method, unboundmethod, module, code, methodwrapper,
  • dictproxy, methoddescriptor, getsetdescriptor, memberdescriptor,
  • wrapperdescriptor, xrange, slice,
  • notimplemented, ellipsis, quit

dill cannot yet pickle these standard types:

  • frame, generator, traceback

dill also provides the capability to:

  • save and load python interpreter sessions
  • save and extract the source code from functions and classes
  • interactively diagnose pickling errors

Current Release

The latest released version of dill is available from: https://pypi.org/project/dill

dill is distributed under a 3-clause BSD license.

Development Version Documentation Status Travis Build Status codecov Downloads

You can get the latest development version with all the shiny new features at: https://github.com/uqfoundation

If you have a new contribution, please submit a pull request.

More Information

Probably the best way to get started is to look at the documentation at http://dill.rtfd.io. Also see dill.tests for a set of scripts that demonstrate how dill can serialize different python objects. You can run the test suite with python -m dill.tests. The contents of any pickle file can be examined with undill. As dill conforms to the pickle interface, the examples and documentation found at http://docs.python.org/library/pickle.html also apply to dill if one will import dill as pickle. The source code is also generally well documented, so further questions may be resolved by inspecting the code itself. Please feel free to submit a ticket on github, or ask a question on stackoverflow (@Mike McKerns). If you would like to share how you use dill in your work, please send an email (to mmckerns at uqfoundation dot org).

Citation

If you use dill to do research that leads to publication, we ask that you acknowledge use of dill by citing the following in your publication::

M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis,
"Building a framework for predictive science", Proceedings of
the 10th Python in Science Conference, 2011;
http://arxiv.org/pdf/1202.1056

Michael McKerns and Michael Aivazis,
"pathos: a framework for heterogeneous computing", 2010- ;
https://uqfoundation.github.io/pathos.html

Please see https://uqfoundation.github.io/pathos.html or http://arxiv.org/pdf/1202.1056 for further information.

Issues
  • Implement changes in #57: improve behaviour of dumped files

    Implement changes in #57: improve behaviour of dumped files

    Here's a start on #57. Questions:

    1. When a file in read or append mode does not exists, the file currently points to os.devnull. Is this acceptable?
    2. Do you think the test file is enough?
    3. Should the IOErrors be replaced by UnpicklingErrors?
    4. When the position is off the end of the file, in safe mode, should an exception be raised, or the position moved to the end?
    enhancement 
    opened by matsjoyce 48
  • dump fails with RuntimeError (recursion depth) for class with super

    dump fails with RuntimeError (recursion depth) for class with super

    The following code fails for me using Python 3.4. (works under 2.7, and also with standard pickle module under 3.4)

    import dill
    
    class obj(object):
        def __init__(self):
            pass
    
    class obj2(obj):
        def __init__(self):
            super(obj2, self).__init__()
    
    dill.dumps(obj2())
    
    bug 
    opened by zentol 43
  • compatibility with pypy

    compatibility with pypy

    Looks like there are some compatibility issues with dill and pypy: see https://github.com/uqfoundation/pathos/issues/50. Apparently ctypes in pypy does not have a CellType… or at least this import fails:

    >>> import ctypes
    >>> ctypes.pythonapi.PyCell_New
    Traceback (most recent call last): File
    "<stdin>", line 1, in <module> AttributeError: 'module' object has no
    attribute 'pythonapi'
    

    Initial fix will be to disable the CellType for pypy

    compatibility 
    opened by mmckerns 33
  • handle failures for corner cases if recurse=True in dump

    handle failures for corner cases if recurse=True in dump

    If certain corner cases can be handled, then recurse=True could be the default setting for dump, which would be more robust.

    Failure with recurse=True noted in some tests: (test_classdef and others).

    bug 
    opened by mmckerns 28
  • Can't dump_session with inline plots

    Can't dump_session with inline plots

    dump_session() seems to fail when used with ipython notebook set up for inline plots. When the %matplotlib inline magic is used you get a traceback resulting in PicklingError: Can't pickle 'RendererAgg' object: <RendererAgg object at 0x42a2bb8> and when using the --pylab=inline option at start up you get a traceback resulting in PicklingError: Can't pickle <class 'matplotlib.axes.AxesSubplot'>: it's not found as matplotlib.axes.AxesSubplot

    Using Dill 0.2a.dev, ipython 1.1.0

    compatibility 
    opened by niallrobinson 23
  • would it be possible to remove dependancy on pyreadline ?

    would it be possible to remove dependancy on pyreadline ?

    indeed:

    • pyreadline for Windows is a problem.
    • the jupyter/ipython team replaced it by python-prompt, so there seems to be a way.
    compatibility 
    opened by stonebig 22
  • Saving a modules __dict__ attribute

    Saving a modules __dict__ attribute

    Possibly a follow up for #35, but saving a session doesn't save the state of the modules:

    Python 3.4.1 (default, May 19 2014, 17:23:49) 
    [GCC 4.9.0 20140507 (prerelease)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import dill
    >>> import test
    >>> test.a = 2
    >>> dill.dump_session()
    >>> 
    ==========================Restart==========================
    Python 3.4.1 (default, May 19 2014, 17:23:49) 
    [GCC 4.9.0 20140507 (prerelease)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import dill
    >>> dill.load_session()
    >>> test
    <module 'test' from 'test.py'>
    >>> test.a
    1
    >>>
    

    test.py:

    def f():
        print(x)
    a = 1
    

    I think this code does the job, but it may need a few adjustments, depending which modules need to be pickled.

    enhancement 
    opened by matsjoyce 22
  • pickles from 0.3.1.1 are not guaranteed to unpickle with 0.3.0

    pickles from 0.3.1.1 are not guaranteed to unpickle with 0.3.0

    We observed in https://issues.apache.org/jira/browse/BEAM-8324 that main sessions pickled with dill==0.3.1.1 cannot be unpickled with dill==0.3.0.

    ... return dill.load_session(file_path)
      File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 410, in load_session
        module = unpickler.load()
    TypeError: _create_function() takes from 2 to 6 positional arguments but 7 were given
    

    This is due to changes https://github.com/uqfoundation/dill/pull/336 that happened between 0.3.0 and 0.3.1.1.

    The changes https://github.com/uqfoundation/dill/pull/336 improve dill functionality, however they also open a question whether such changes should be implemented in a backwards-compatible way, when minor version is not increased.

    @mmckerns - does dill intend to have any backwards-compatibility guarantees between versions?

    cc:@lazylynx

    compatibility 
    opened by tvalentyn 21
  • Fix problem with pickling file instances

    Fix problem with pickling file instances

    I was trying to pickle files earlier and I think I've found a bug:

    Traceback (most recent call last):
      File "example.py", line 2, in <module>
        dill.dumps(open(__file__))
      File "/home/matthew/Programming/C++/Python/eggs/dill/dill.py", line 135, in dumps
        dump(obj, file, protocol, byref)
      File "/home/matthew/Programming/C++/Python/eggs/dill/dill.py", line 128, in dump
        pik.dump(obj)
      File "/usr/lib/python3.3/pickle.py", line 235, in dump
        self.save(obj)
      File "/usr/lib/python3.3/pickle.py", line 317, in save
        rv = reduce(self.proto)
    TypeError: cannot serialize '_io.TextIOWrapper' object
    

    It can be fixed by adding type for the file type definitions.

    bug 
    opened by matsjoyce 21
  • desired behavior for file handles?

    desired behavior for file handles?

    Load of a pickled file handle can create a file. Is that what is desired? If so, what should the file-pointer position in the file be? Right now, the state of the file is preserved, as well as the mode and the position. Here are the consequences...

    [email protected]>$ python
    Python 2.7.8 (default, Jul  3 2014, 05:59:29) 
    [GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import dill
    >>> f = open('test.txt', 'w+')
    >>> f.write('hello')
    >>> dill.dump_session('test.pkl')
    >>> 
    

    Deleting the file and then loading the session simulates going to another computer where test.txt does not exist.

    [email protected]>$ rm test.txt 
    remove test.txt? y
    [email protected]>$ python      
    Python 2.7.8 (default, Jul  3 2014, 05:59:29) 
    [GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import dill
    >>> dill.load_session('test.pkl')
    >>> f
    <open file 'test.txt', mode 'w+' at 0x1096fac90>
    >>> f.write('world')
    >>> f.seek(0)
    >>> f.read()
    '\x00\x00\x00\x00\x00world'
    >>> 
    

    Similarly, if 'test.txt' existed, but had the contents "goodbye" instead of "hello"… it gets even nastier.

    [email protected]>$ python
    Python 2.7.8 (default, Jul  3 2014, 05:59:29) 
    [GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import dill
    >>> g = open('test.txt', 'r')
    >>> g.read()
    'goodbye\n'
    >>> dill.load_session('test.pkl')
    >>> f
    <open file 'test.txt', mode 'w+' at 0x100e69d20>
    >>> f.write('world')
    >>> f.seek(0)
    >>> f.read()
    '\x00\x00\x00\x00\x00world'
    >>> g.seek(0)
    >>> g.read()
    '\x00\x00\x00\x00\x00world'
    >>> 
    

    This indicates that dill is actually serializing more than the file handle, it's in essence serializing the file, but with "filler" if it's creating a new file and the position is greater than zero. Is that the desired behavior? Or better if a new file is created, the position be reset with seek(0)? Or better that if the file doesn't exist, the file handle be closed?

    question 
    opened by mmckerns 20
  • Issue 332 abc data

    Issue 332 abc data

    Resolves #332

    Register ABCMeta to use StockPickler. If workable, this is preferable to the cloudpickle solution that depends heavily on the internals of the ABC implementation.

    This will remain broken on some versions of Python 3.7 that did not support pickling the new c implementation of ABC.

    Issues:

    • Registering ABCMeta seems to exclude other registered types, for instance the module/locality solution for ClassType (see broken test)
    • Need to confirm expected behavior for ABC registry once locality issue is solved
    opened by emfdavid 1
  • Custom pickling doc

    Custom pickling doc

    null

    opened by NickFillot 2
  • Coolprop

    Coolprop "TypeError: no default __reduce__ due to non-trivial __cinit__"

    import dill
    from CoolProp.Plots import PropertyPlot # conda install -c conda-forge coolprop
    
    plot = PropertyPlot("Air", 'Ph')
    plot.calc_isolines()
    plot.show()
    
    dill.dump_session("plot")
    

    Running the above code will result: Replaced unnecessary paths with ".."

    File "..\test2.py", line 8, in <module>
        dill.dump_session("plot")
      File "..\lib\site-packages\dill\_dill.py", line 403, in dump_session
        pickler.dump(main)
      File "..\lib\site-packages\dill\_dill.py", line 498, in dump
        StockPickler.dump(self, obj)
      File "..\lib\pickle.py", line 487, in dump
        self.save(obj)
      File "..\lib\pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "..\lib\site-packages\dill\_dill.py", line 1375, in save_module
        pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
      File "..\lib\pickle.py", line 717, in save_reduce
        save(state)
      File "..\lib\pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "..\lib\site-packages\dill\_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "..\lib\pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "..\lib\pickle.py", line 997, in _batch_setitems
        save(v)
      File "..\lib\pickle.py", line 603, in save
        self.save_reduce(obj=obj, *rv)
      File "..\lib\pickle.py", line 717, in save_reduce
        save(state)
      File "..\lib\pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "..\lib\site-packages\dill\_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "..\lib\pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "..\lib\pickle.py", line 997, in _batch_setitems
        save(v)
      File "..\lib\pickle.py", line 578, in save
        rv = reduce(self.proto)
      File "stringsource", line 2, in CoolProp.CoolProp.AbstractState.__reduce_cython__
    TypeError: no default __reduce__ due to non-trivial __cinit__
    
    Process finished with exit code 1
    

    issue #298 may address the same problem. is the problem caused by CoolProp, or dill ?

    opened by Shihabus-Sakib-Rad 0
  • Ability to force by-value serialization of classes?

    Ability to force by-value serialization of classes?

    @mmckerns, would you be up for adding a feature to dill that allows forcing a particular class to be serialized by value rather than by reference? Specifying byref=False only applies to a limited set of cases at the moment.

    The specific motivation is https://github.com/pulumi/pulumi/issues/7453, in which it is desired to have the pickled class always serialized by value, regardless of whether it is in __main__ or a different module.

    cloudpickle recently added a feature that allows this sort of control: https://github.com/cloudpipe/cloudpickle/pull/417. (But there are other reasons that cloudpickle doesn't work for Pulumi.)

    opened by benesch 2
  • [Question] Create_filehandle: reader position during mode HANDLE_FMODE

    [Question] Create_filehandle: reader position during mode HANDLE_FMODE

    [Edit to make my question clearer] I understand that dill currently has 3 file modes and understand their differences. However, is there an underlying reason why dill decides to ignore the reader position when using HANDLE_FMODE (default)? I couldn't find the document or comment anywhere.

    https://github.com/uqfoundation/dill/blob/master/dill/_dill.py#L782

    def _create_filehandle(name, mode, position, closed, open, strictio, fmode, fdata):
        ...
        ...
        if closed:
            f.close()
        elif position >= 0 and fmode != HANDLE_FMODE:
            f.seek(position)
        return f
    
    opened by Bankde 0
  • Pickle function attributes

    Pickle function attributes

    add annotations arg

    opened by NickFillot 0
  • [Enhancement] pickle function __annotations__

    [Enhancement] pickle function __annotations__

    Hello,

    Is it possible to implement function.annotations pickle on loads / dumps ?

    Br Nick

    enhancement 
    opened by NickFillot 3
  • dill.pickles returns True but pickling still creates an error

    dill.pickles returns True but pickling still creates an error

    I'm using discord.py and am trying to pickle a Message object.

    I extracted all the variables out but that doesn't work either, but I can't narrow down which ones have the issues.

    I tried using dill.pickles on all of the variables, and it always returns True for every single item, but pickling still raises TypeError: can't pickle _asyncio.Future objects

    opened by TheTechRobo 1
  • Issue serializing TensorFlow

    Issue serializing TensorFlow

    With the following blob of Python:

    # mlp for binary classification
    from pandas import read_csv
    from sklearn.model_selection import train_test_split
    from sklearn.preprocessing import LabelEncoder
    from tensorflow.keras import Sequential
    from tensorflow.keras.layers import Dense
    # load the dataset
    path = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/ionosphere.csv'
    df = read_csv(path, header=None)
    # split into input and output columns
    X, y = df.values[:, :-1], df.values[:, -1]
    # ensure all data are floating point values
    X = X.astype('float32')
    # encode strings to integer
    y = LabelEncoder().fit_transform(y)
    # split into train and test datasets
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)
    print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)
    # determine the number of input features
    n_features = X_train.shape[1]
    # define model
    model = Sequential()
    model.add(Dense(10, activation='relu', kernel_initializer='he_normal', input_shape=(n_features,)))
    model.add(Dense(8, activation='relu', kernel_initializer='he_normal'))
    model.add(Dense(1, activation='sigmoid'))
    # compile the model
    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
    # fit the model
    model.fit(X_train, y_train, epochs=150, batch_size=32, verbose=0)
    # evaluate the model
    loss, acc = model.evaluate(X_test, y_test, verbose=0)
    print('Test Accuracy: %.3f' % acc)
    # make a prediction
    row = [1,0,0.99539,-0.05889,0.85243,0.02306,0.83398,-0.37708,1,0.03760,0.85243,-0.17755,0.59755,-0.44945,0.60536,-0.38223,0.84356,-0.38542,0.58212,-0.32192,0.56971,-0.29674,0.36946,-0.47357,0.56811,-0.51171,0.41078,-0.46168,0.21266,-0.34090,0.42267,-0.54487,0.18641,-0.45300]
    yhat = model.predict([row])
    print('Predicted: %.3f' % yhat)
    
    __locals_keys = frozenset(locals().keys())
    __globals_keys = frozenset(globals().keys())
    __context_export = {}
    
    for val in __globals_keys:
    	if not val.startswith("_") and not isinstance(val, __ModuleType):
    		__context_export[val] = dill.dumps(globals()[val])
    
    for val in __locals_keys:
    	if not val.startswith("_") and not isinstance(val, __ModuleType):
    		__context_export[val] = dill.dumps(locals()[val])
    
    __b64_string = str(urlsafe_b64encode(dill.dumps(__context_export)), encoding="ascii")
    

    We get the following error:

    Traceback (most recent call last):
      File "/tmp/tmp.sHD1auXCoK", line 156, in <module>
        _outputs = main(**_parsed_args)
      File "/tmp/tmp.sHD1auXCoK", line 117, in main
        exec(__inner_code_to_execute, __variables_to_mount, __loc)
      File "<string>", line 74, in <module>
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 304, in dumps
        dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 276, in dump
        Pickler(file, protocol, **_kwds).dump(obj)
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 498, in dump
        StockPickler.dump(self, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 487, in dump
        self.save(obj)
      File "/usr/local/lib/python3.9/pickle.py", line 603, in save
        self.save_reduce(obj=obj, *rv)
      File "/usr/local/lib/python3.9/pickle.py", line 717, in save_reduce
        save(state)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 997, in _batch_setitems
        save(v)
      File "/usr/local/lib/python3.9/pickle.py", line 603, in save
        self.save_reduce(obj=obj, *rv)
      File "/usr/local/lib/python3.9/pickle.py", line 717, in save_reduce
        save(state)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/pickle.py", line 886, in save_tuple
        save(element)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 1002, in _batch_setitems
        save(v)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 996, in _batch_setitems
        save(k)
      File "/usr/local/lib/python3.9/pickle.py", line 603, in save
        self.save_reduce(obj=obj, *rv)
      File "/usr/local/lib/python3.9/pickle.py", line 717, in save_reduce
        save(state)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/pickle.py", line 886, in save_tuple
        save(element)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 1002, in _batch_setitems
        save(v)
      File "/usr/local/lib/python3.9/pickle.py", line 603, in save
        self.save_reduce(obj=obj, *rv)
      File "/usr/local/lib/python3.9/pickle.py", line 717, in save_reduce
        save(state)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 997, in _batch_setitems
        save(v)
      File "/usr/local/lib/python3.9/pickle.py", line 603, in save
        self.save_reduce(obj=obj, *rv)
      File "/usr/local/lib/python3.9/pickle.py", line 717, in save_reduce
        save(state)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 997, in _batch_setitems
        save(v)
      File "/usr/local/lib/python3.9/pickle.py", line 603, in save
        self.save_reduce(obj=obj, *rv)
      File "/usr/local/lib/python3.9/pickle.py", line 717, in save_reduce
        save(state)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/pickle.py", line 886, in save_tuple
        save(element)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 1002, in _batch_setitems
        save(v)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 996, in _batch_setitems
        save(k)
      File "/usr/local/lib/python3.9/pickle.py", line 603, in save
        self.save_reduce(obj=obj, *rv)
      File "/usr/local/lib/python3.9/pickle.py", line 717, in save_reduce
        save(state)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/pickle.py", line 886, in save_tuple
        save(element)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 1002, in _batch_setitems
        save(v)
      File "/usr/local/lib/python3.9/pickle.py", line 603, in save
        self.save_reduce(obj=obj, *rv)
      File "/usr/local/lib/python3.9/pickle.py", line 717, in save_reduce
        save(state)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 997, in _batch_setitems
        save(v)
      File "/usr/local/lib/python3.9/pickle.py", line 603, in save
        self.save_reduce(obj=obj, *rv)
      File "/usr/local/lib/python3.9/pickle.py", line 717, in save_reduce
        save(state)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 997, in _batch_setitems
        save(v)
      File "/usr/local/lib/python3.9/pickle.py", line 603, in save
        self.save_reduce(obj=obj, *rv)
      File "/usr/local/lib/python3.9/pickle.py", line 717, in save_reduce
        save(state)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 997, in _batch_setitems
        save(v)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 997, in _batch_setitems
        save(v)
      File "/usr/local/lib/python3.9/pickle.py", line 603, in save
        self.save_reduce(obj=obj, *rv)
      File "/usr/local/lib/python3.9/pickle.py", line 717, in save_reduce
        save(state)
      File "/usr/local/lib/python3.9/pickle.py", line 560, in save
        f(self, obj)  # Call unbound method with explicit self
      File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict
        StockPickler.save_dict(pickler, obj)
      File "/usr/local/lib/python3.9/pickle.py", line 971, in save_dict
        self._batch_setitems(obj.items())
      File "/usr/local/lib/python3.9/pickle.py", line 997, in _batch_setitems
        save(v)
      File "/usr/local/lib/python3.9/pickle.py", line 578, in save
        rv = reduce(self.proto)
    TypeError: cannot pickle 'tensorflow.python._pywrap_tf_session.TF_Operation' object
    

    Do you have any guidance on where we should go next and/or if there's a better pattern for us?

    opened by aronchick 1
  • Cannot dill.copy cv2.TrackerKCF object

    Cannot dill.copy cv2.TrackerKCF object

    Hi,

    Context

    I am developping a tracking algorithm from video stream. I have several people identified from an input video stream that I would like to track individually, frame to frame. So I use the module OpenCV-python to track one person from one frame to the next one. One tracker object is used for each person. Sometimes, I have dozens of people on a frame. Each tracking is done on the same CPU. I would like to do a multiprocess tracking.

    Example: I have 20 tracked people on the frame and I have 4 CPU. I would like to track 5 people per CPU instead of tracking 20 people on a single CPU (default behaviour). So I make a pool with 4 available proccesses that parrallelize my tracking task.

    I realized that cv2.TrackerKCF cannot be pickled. Although I can pickle my custom objects thanks to dill

    Error

    I would like to be able to pickle an cv2.TrackerKCF object. Step to reproduce:

    import dill, cv2
    
    dill.copy( cv2.TrackerKCF() )
    

    Logs:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/muffie/code/ksm/code/env/lib/python3.8/site-packages/dill/_dill.py", line 264, in copy
        return loads(dumps(obj, *args, **kwds), ignore=ignore)
      File "/home/muffie/code/ksm/code/env/lib/python3.8/site-packages/dill/_dill.py", line 304, in dumps
        dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
      File "/home/muffie/code/ksm/code/env/lib/python3.8/site-packages/dill/_dill.py", line 276, in dump
        Pickler(file, protocol, **_kwds).dump(obj)
      File "/home/muffie/code/ksm/code/env/lib/python3.8/site-packages/dill/_dill.py", line 498, in dump
        StockPickler.dump(self, obj)
      File "/usr/lib/python3.8/pickle.py", line 485, in dump
        self.save(obj)
      File "/usr/lib/python3.8/pickle.py", line 576, in save
        rv = reduce(self.proto)
    TypeError: cannot pickle 'cv2.TrackerKCF' object
    

    I think that it is caused by OpenCV-python written in C++ and its translation in Python. I would like to know if there is a work around for this case.

    Workaround attempt:

    I tried to wrap my cv2.TrackerKCF in another object, it doesn't work.

    import cv2, dill
    class Tracker:
      def __init__(self, tracker : cv2.TrackerKCF):
        self.tracker = tracker
    
    dill.copy( Tracker( cv2.TrackerKCF() )  )
    

    I'm not a multiprocessing expert at all, I hope I didn't miss any important point.

    Thanks for reading,

    Mathias.

    opened by ATrackerLearner 0
Releases(dill-0.3.4)
  • dill-0.3.4(Jun 13, 2021)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ pip install dill

    dill requires: - python, version == 2.7 *or* version >= 3.6 *or* pypy

    Optional requirements: - pyreadline, version >= 1.7.1 (install with $ pip install dill[readline]) - objgraph, version >= 1.7.2 (install with $ pip install dill[graph])

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.3.4-py2.py3-none-any.whl(84.88 KB)
    dill-0.3.4.tar.gz(154.71 KB)
    dill-0.3.4.zip(182.44 KB)
  • dill-0.3.3(Nov 1, 2020)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ pip install dill

    dill requires: - python, version == 2.7 *or* version >= 3.5 *or* pypy

    Optional requirements: - pyreadline, version >= 1.7.1 (install with $ pip install dill[readline]) - objgraph, version >= 1.7.2 (install with $ pip install dill[graph])

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.3.3-py2.py3-none-any.whl(79.40 KB)
    dill-0.3.3.tar.gz(149.73 KB)
    dill-0.3.3.zip(173.52 KB)
  • dill-0.3.2(Jun 15, 2020)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ pip install dill

    dill requires: - python, version == 2.7 *or* version >= 3.5 *or* pypy

    Optional requirements: - pyreadline, version >= 1.7.1 (install with $ pip install dill[readline]) - objgraph, version >= 1.7.2 (install with $ pip install dill[graph])

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.3.2.zip(173.40 KB)
    dill-0.3.2.tar.gz(149.59 KB)
  • dill-0.3.1.1(Sep 28, 2019)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ pip install dill

    dill requires: - python, version >=2.6 *or* version >= 3.1 *or* pypy

    Optional requirements: - pyreadline, version >= 1.7.1 (install with $ pip install dill[readline]) - objgraph, version >= 1.7.2 (install with $ pip install dill[graph])

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.3.1.1.tar.gz(148.42 KB)
    dill-0.3.1.1.zip(173.91 KB)
  • dill-0.3.0(Jun 26, 2019)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ pip install dill

    dill requires: - python, version >=2.6 *or* version >= 3.1 *or* pypy

    Optional requirements: - pyreadline, version >= 1.7.1 (install with $ pip install dill[readline]) - objgraph, version >= 1.7.2 (install with $ pip install dill[graph])

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.3.0.tar.gz(147.82 KB)
    dill-0.3.0.zip(172.60 KB)
  • dill-0.2.9(Jan 21, 2019)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ pip install dill

    dill requires: - python, version >=2.5 *or* version >= 3.1 *or* pypy

    Optional requirements: - pyreadline, version >= 1.7.1 (install with $ pip install dill[readline]) - objgraph, version >= 1.7.2 (install with $ pip install dill[graph])

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.9.tar.gz(147.20 KB)
    dill-0.2.9.zip(85.05 KB)
  • dill-0.2.8.2(Jun 22, 2018)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ pip install dill

    dill requires: - python, version >=2.5 *or* version >= 3.1 *or* pypy

    Optional requirements: - objgraph, version >= 1.7.2

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.8.2.tar.gz(146.56 KB)
    dill-0.2.8.2.zip(84.00 KB)
  • dill-0.2.8.1(Jun 20, 2018)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ pip install dill

    dill requires: - python, version >=2.5 *or* version >= 3.1 *or* pypy

    Optional requirements: - objgraph, version >= 1.7.2

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.8.1.tar.gz(146.56 KB)
    dill-0.2.8.1.zip(83.99 KB)
  • dill-0.2.8(Jun 20, 2018)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ pip install dill

    dill requires: - python, version >=2.5 *or* version >= 3.1 *or* pypy

    Optional requirements: - objgraph, version >= 1.7.2

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.8.tar.gz(579.87 KB)
    dill-0.2.8.zip(83.80 KB)
  • dill-0.2.7.1(Jul 18, 2017)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ easy_install dill

    dill requires: - python, version >=2.5 *or* version >= 3.1 *or* pypy

    Optional requirements: - objgraph, version >= 1.7.2

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.7.1.tar.gz(62.97 KB)
    dill-0.2.7.1.zip(83.04 KB)
  • dill-0.2.7(Jul 6, 2017)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ easy_install dill

    dill requires: - python, version >=2.5 *or* version >= 3.1 *or* pypy

    Optional requirements: - objgraph, version >= 1.7.2

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.7.tar.gz(62.56 KB)
    dill-0.2.7.zip(82.37 KB)
  • 0.2.6(Feb 1, 2017)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ easy_install dill

    dill requires: - python, version >=2.5 *or* version >= 3.1 *or* pypy

    Optional requirements: - objgraph, version >= 1.7.2

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.6.zip(81.41 KB)
    dill-0.2.6.tgz(60.94 KB)
  • dill-0.2.5(Feb 6, 2016)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ easy_install dill

    dill requires: - python, version >=2.5 *or* version >= 3.1

    Optional requirements: - objgraph, version >= 1.7.2

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.5.tgz(59.15 KB)
    dill-0.2.5.zip(79.79 KB)
  • dill-0.2.4(Aug 5, 2015)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    dill installs with easy_install or pip: $ easy_install dill

    dill requires: - python, version >=2.5 *or* version >= 3.1

    Optional requirements: - objgraph, version >= 1.7.2

    dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.4.tgz(57.23 KB)
    dill-0.2.4.zip(77.19 KB)
  • dill-0.2.3(Jun 22, 2015)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    Dill installs with easy_install or pip: $ easy_install dill

    Dill requires: - python, version >=2.5 *or* version >= 3.1

    Optional requirements: - objgraph, version >= 1.7.2

    Dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.3.zip(76.47 KB)
    dill-0.2.3.tgz(56.26 KB)
  • dill-0.2.2(Dec 26, 2014)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    Dill installs with easy_install or pip: $ easy_install dill

    Dill requires: - python, version >=2.5 *or* version >= 3.1

    Optional requirements: - objgraph, version >= 1.7.2

    Dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.2.tgz(53.57 KB)
    dill-0.2.2.zip(72.59 KB)
  • 0.2.1(Jun 1, 2014)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    Dill installs with easy_install or pip: $ easy_install dill

    Dill requires: - python, version >=2.5 *or* version >= 3.1

    Optional requirements: - objgraph, version >= 1.7.2

    Dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.1.tgz(44.72 KB)
    dill-0.2.1.zip(62.48 KB)
  • 0.2(May 17, 2014)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    Dill installs with easy_install or pip: $ easy_install dill

    Dill requires: - python, version >=2.5 *or* version >= 3.1

    Optional requirements: - objgraph, version >= 1.7.2

    Dill is licensed under 3-clause BSD:

    >>> import dill
    >>> print (dill.license())
    

    To cite dill:

    >>> import dill
    >>> print (dill.citation())
    
    Source code(tar.gz)
    Source code(zip)
    dill-0.2.tgz(44.20 KB)
    dill-0.2.zip(61.41 KB)
  • 0.2b1(Nov 27, 2013)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    Dill installs with easy_install or pip: $ easy_install dill

    Dill requires: - python, version >=2.5 *or* version >= 3.1

    Optional requirements: - objgraph, version >= 1.7.2

    Dill is licensed under 3-clause BSD:>>> import dill >>> print (dill.license()) To cite dill:>>> import dill >>> print (dill.citation())

    Source code(tar.gz)
    Source code(zip)
    dill-0.2b1.tgz(23.50 KB)
    dill-0.2b1.zip(35.89 KB)
    dill-0.2b1.win32.exe(90.52 KB)
  • 0.2a1(Oct 20, 2013)

    With dill, you can serialize almost anything in python, even an entire interpreter session. If you encounter any pickling failures, dill also has some good tools to help you discover why your object fails to pickle.

    Dill installs with easy_install or pip: $ easy_install dill

    Dill requires: - python, version >=2.5 *or* version >= 3.1

    Optional requirements: - objgraph, version >= 1.7.2

    Dill is licensed under 3-clause BSD:>>> import dill >>> print (dill.license()) To cite dill:>>> import dill >>> print (dill.citation())

    Source code(tar.gz)
    Source code(zip)
    dill-0.2a1.zip(32.91 KB)
    dill-0.2a1.tgz(23.66 KB)
Owner
The UQ Foundation
A nonprofit dedicated to the advancement of predictive science through research partnerships and education.
The UQ Foundation
serialize all of python

dill serialize all of python About Dill dill extends python's pickle module for serializing and de-serializing python objects to the majority of the b

The UQ Foundation 1.5k Sep 23, 2021
A lightweight library for converting complex objects to and from simple Python datatypes.

marshmallow: simplified object serialization marshmallow is an ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, t

marshmallow-code 5.7k Sep 24, 2021
Extended pickling support for Python objects

cloudpickle cloudpickle makes it possible to serialize Python constructs not supported by the default pickle module from the Python standard library.

null 1k Sep 23, 2021
Python bindings for the simdjson project.

pysimdjson Python bindings for the simdjson project, a SIMD-accelerated JSON parser. If SIMD instructions are unavailable a fallback parser is used, m

Tyler Kennedy 474 Sep 24, 2021
Generic ASN.1 library for Python

ASN.1 library for Python This is a free and open source implementation of ASN.1 types and codecs as a Python package. It has been first written to sup

Ilya Etingof 187 Sep 20, 2021
MessagePack serializer implementation for Python msgpack.org[Python]

MessagePack for Python What's this MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JS

MessagePack 1.5k Sep 22, 2021
Python library for serializing any arbitrary object graph into JSON. It can take almost any Python object and turn the object into JSON. Additionally, it can reconstitute the object back into Python.

jsonpickle jsonpickle is a library for the two-way conversion of complex Python objects and JSON. jsonpickle builds upon the existing JSON encoders, s

null 962 Sep 16, 2021
Ultra fast JSON decoder and encoder written in C with Python bindings

UltraJSON UltraJSON is an ultra fast JSON encoder and decoder written in pure C with bindings for Python 3.6+. Install with pip: $ python -m pip insta

null 3.4k Sep 24, 2021
Corset is a web-based data selection portal that helps you getting relevant data from massive amounts of parallel data.

Corset is a web-based data selection portal that helps you getting relevant data from massive amounts of parallel data. So, if you don't need the whole corpus, but just a suitable subset (indeed, a cor(pus sub)set, this is what Corset will do for you--and the reason of the name of the tool.

null 5 Jun 15, 2021
Python wrapper around rapidjson

python-rapidjson Python wrapper around RapidJSON Authors: Ken Robbins <[email protected]> Lele Gaifax <[email protected]> License: MIT License Sta

null 429 Sep 17, 2021
simplejson is a simple, fast, extensible JSON encoder/decoder for Python

simplejson simplejson is a simple, fast, complete, correct and extensible JSON <http://json.org> encoder and decoder for Python 3.3+ with legacy suppo

null 1.4k Sep 24, 2021
Protocol Buffers - Google's data interchange format

Protocol Buffers - Google's data interchange format Copyright 2008 Google Inc. https://developers.google.com/protocol-buffers/ Overview Protocol Buffe

Protocol Buffers 50.8k Sep 23, 2021
Crappy tool to convert .scw files to .json and and vice versa.

SCW-JSON-TOOL Crappy tool to convert .scw files to .json and vice versa. How to use Run main.py file with two arguments: python main.py <scw2json or j

Fred31 5 May 14, 2021
FlatBuffers: Memory Efficient Serialization Library

FlatBuffers FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serializ

Google 16.8k Sep 23, 2021