🦉 Modern high-performance serialization utilities for Python (JSON, MessagePack, Pickle)

Overview

srsly: Modern high-performance serialization utilities for Python

This package bundles some of the best Python serialization libraries into one standalone package, with a high-level API that makes it easy to write code that's correct across platforms and Pythons. This allows us to provide all the serialization utilities we need in a single binary wheel. Currently supports JSON, JSONL, MessagePack, Pickle and YAML.

Azure Pipelines PyPi conda GitHub Python wheels

Motivation

Serialization is hard, especially across Python versions and multiple platforms. After dealing with many subtle bugs over the years (encodings, locales, large files) our libraries like spaCy and Prodigy have steadily grown a number of utility functions to wrap the multiple serialization formats we need to support (especially json, msgpack and pickle). These wrapping functions ended up duplicated across our codebases, so we wanted to put them in one place.

At the same time, we noticed that having a lot of small dependencies was making maintenance harder, and making installation slower. To solve this, we've made srsly standalone, by including the component packages directly within it. This way we can provide all the serialization utilities we need in a single binary wheel.

srsly currently includes forks of the following packages:

Installation

⚠️ Note that v2.x is only compatible with Python 3.6+. For 2.7+ compatibility, use v1.x.

srsly can be installed from pip. Before installing, make sure that your pip, setuptools and wheel are up to date.

pip install -U pip setuptools wheel
pip install srsly

Or from conda via conda-forge:

conda install -c conda-forge srsly

Alternatively, you can also compile the library from source. You'll need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler (XCode command-line tools on macOS / OS X or Visual C++ build tools on Windows), pip, virtualenv and git installed.

pip install -r requirements.txt  # install development dependencies
python setup.py build_ext --inplace  # compile the library

API

JSON

📦 The underlying module is exposed via srsly.ujson. However, we normally interact with it via the utility functions only.

function srsly.json_dumps

Serialize an object to a JSON string. Falls back to json if sort_keys=True is used (until it's fixed in ujson).

data = {"foo": "bar", "baz": 123}
json_string = srsly.json_dumps(data)
Argument Type Description
data - The JSON-serializable data to output.
indent int Number of spaces used to indent JSON. Defaults to 0.
sort_keys bool Sort dictionary keys. Defaults to False.
RETURNS str The serialized string.

function srsly.json_loads

Deserialize unicode or bytes to a Python object.

data = '{"foo": "bar", "baz": 123}'
obj = srsly.json_loads(data)
Argument Type Description
data str / bytes The data to deserialize.
RETURNS - The deserialized Python object.

function srsly.write_json

Create a JSON file and dump contents or write to standard output.

data = {"foo": "bar", "baz": 123}
srsly.write_json("/path/to/file.json", data)
Argument Type Description
path str / Path The file path or "-" to write to stdout.
data - The JSON-serializable data to output.
indent int Number of spaces used to indent JSON. Defaults to 2.

function srsly.read_json

Load JSON from a file or standard input.

data = srsly.read_json("/path/to/file.json")
Argument Type Description
path str / Path The file path or "-" to read from stdin.
RETURNS dict / list The loaded JSON content.

function srsly.write_gzip_json

Create a gzipped JSON file and dump contents.

data = {"foo": "bar", "baz": 123}
srsly.write_gzip_json("/path/to/file.json.gz", data)
Argument Type Description
path str / Path The file path.
data - The JSON-serializable data to output.
indent int Number of spaces used to indent JSON. Defaults to 2.

function srsly.read_gzip_json

Load gzipped JSON from a file.

data = srsly.read_gzip_json("/path/to/file.json.gz")
Argument Type Description
path str / Path The file path.
RETURNS dict / list The loaded JSON content.

function srsly.write_jsonl

Create a JSONL file (newline-delimited JSON) and dump contents line by line, or write to standard output.

data = [{"foo": "bar"}, {"baz": 123}]
srsly.write_jsonl("/path/to/file.jsonl", data)
Argument Type Description
path str / Path The file path or "-" to write to stdout.
lines iterable The JSON-serializable lines.
append bool Append to an existing file. Will open it in "a" mode and insert a newline before writing lines. Defaults to False.
append_new_line bool Defines whether a new line should first be written when appending to an existing file. Defaults to True.

function srsly.read_jsonl

Read a JSONL file (newline-delimited JSON) or from JSONL data from standard input and yield contents line by line. Blank lines will always be skipped.

data = srsly.read_jsonl("/path/to/file.jsonl")
Argument Type Description
path str / Path The file path or "-" to read from stdin.
skip bool Skip broken lines and don't raise ValueError. Defaults to False.
YIELDS - The loaded JSON contents of each line.

function srsly.is_json_serializable

Check if a Python object is JSON-serializable.

assert srsly.is_json_serializable({"hello": "world"}) is True
assert srsly.is_json_serializable(lambda x: x) is False
Argument Type Description
obj - The object to check.
RETURNS bool Whether the object is JSON-serializable.

msgpack

📦 The underlying module is exposed via srsly.msgpack. However, we normally interact with it via the utility functions only.

function srsly.msgpack_dumps

Serialize an object to a msgpack byte string.

data = {"foo": "bar", "baz": 123}
msg = srsly.msgpack_dumps(data)
Argument Type Description
data - The data to serialize.
RETURNS bytes The serialized bytes.

function srsly.msgpack_loads

Deserialize msgpack bytes to a Python object.

msg = b"\x82\xa3foo\xa3bar\xa3baz{"
data = srsly.msgpack_loads(msg)
Argument Type Description
data bytes The data to deserialize.
use_list bool Don't use tuples instead of lists. Can make deserialization slower. Defaults to True.
RETURNS - The deserialized Python object.

function srsly.write_msgpack

Create a msgpack file and dump contents.

data = {"foo": "bar", "baz": 123}
srsly.write_msgpack("/path/to/file.msg", data)
Argument Type Description
path str / Path The file path.
data - The data to serialize.

function srsly.read_msgpack

Load a msgpack file.

data = srsly.read_msgpack("/path/to/file.msg")
Argument Type Description
path str / Path The file path.
use_list bool Don't use tuples instead of lists. Can make deserialization slower. Defaults to True.
RETURNS - The loaded and deserialized content.

pickle

📦 The underlying module is exposed via srsly.cloudpickle. However, we normally interact with it via the utility functions only.

function srsly.pickle_dumps

Serialize a Python object with pickle.

data = {"foo": "bar", "baz": 123}
pickled_data = srsly.pickle_dumps(data)
Argument Type Description
data - The object to serialize.
protocol int Protocol to use. -1 for highest. Defaults to None.
RETURNS bytes The serialized object.

function srsly.pickle_loads

Deserialize bytes with pickle.

pickled_data = b"\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00}\x94(\x8c\x03foo\x94\x8c\x03bar\x94\x8c\x03baz\x94K{u."
data = srsly.pickle_loads(pickled_data)
Argument Type Description
data bytes The data to deserialize.
RETURNS - The deserialized Python object.

YAML

📦 The underlying module is exposed via srsly.ruamel_yaml. However, we normally interact with it via the utility functions only.

function srsly.yaml_dumps

Serialize an object to a YAML string. See the ruamel.yaml docs for details on the indentation format.

data = {"foo": "bar", "baz": 123}
yaml_string = srsly.yaml_dumps(data)
Argument Type Description
data - The JSON-serializable data to output.
indent_mapping int Mapping indentation. Defaults to 2.
indent_sequence int Sequence indentation. Defaults to 4.
indent_offset int Indentation offset. Defaults to 2.
sort_keys bool Sort dictionary keys. Defaults to False.
RETURNS str The serialized string.

function srsly.yaml_loads

Deserialize unicode or a file object to a Python object.

data = 'foo: bar\nbaz: 123'
obj = srsly.yaml_loads(data)
Argument Type Description
data str / file The data to deserialize.
RETURNS - The deserialized Python object.

function srsly.write_yaml

Create a YAML file and dump contents or write to standard output.

data = {"foo": "bar", "baz": 123}
srsly.write_yaml("/path/to/file.yml", data)
Argument Type Description
path str / Path The file path or "-" to write to stdout.
data - The JSON-serializable data to output.
indent_mapping int Mapping indentation. Defaults to 2.
indent_sequence int Sequence indentation. Defaults to 4.
indent_offset int Indentation offset. Defaults to 2.
sort_keys bool Sort dictionary keys. Defaults to False.

function srsly.read_yaml

Load YAML from a file or standard input.

data = srsly.read_yaml("/path/to/file.yml")
Argument Type Description
path str / Path The file path or "-" to read from stdin.
RETURNS dict / list The loaded YAML content.

function srsly.is_yaml_serializable

Check if a Python object is YAML-serializable.

assert srsly.is_yaml_serializable({"hello": "world"}) is True
assert srsly.is_yaml_serializable(lambda x: x) is False
Argument Type Description
obj - The object to check.
RETURNS bool Whether the object is YAML-serializable.
Issues
  • No module named 'srsly.ujson.ujson'

    No module named 'srsly.ujson.ujson'

    I'm installing srsly on an AWS lambda but I keep getting an error : No module named 'srsly.ujson.ujson'

    It's install with srsly inside its package but can't be access. I tried to change all declaration to a global import ujson which i downloaded without any success ...

    Any ideas ?

    PS : I'm installing a Spacy lambda which require srsly

    opened by antoineGit 6
  • How to use `srsly.msgpack_dumps`  with my custom class?

    How to use `srsly.msgpack_dumps` with my custom class?

    I want to serialize my custom class with srsly.msgpack_dumps, because it is stored in spacy.Doc. In other words, doc.to_disk fails because my custom class cannotn be serialized with srsly.msgpack_dumps. How to make my custom class to be able to save?

    opened by tamuhey 3
  • Memory leaks in ujson

    Memory leaks in ujson

    We've replaced the usage of json with srsly.ujson a while ago for that free performance boost since we are doing lots of JSON encoding/decoding and we already have it installed as part of spacy, but now we had to move back because of some terrible memory leaks:

    import json
    import random
    import string
    import psutil
    from srsly import ujson as json
    
    sample = lambda x: ''.join(
      random.choice(string.ascii_uppercase + string.digits) for _ in range(x))
    
    process = psutil.Process()
    
    for i in range(10):
      data = json.dumps({sample(99): sample(100000) for k in range(50)})
      json.loads(data)
      print(process.memory_info())
    

    Output with ujson:

    pmem(rss=24203264, vms=4400664576, pfaults=19049, pageins=0)
    pmem(rss=29409280, vms=4414173184, pfaults=33898, pageins=0)
    pmem(rss=34557952, vms=4419309568, pfaults=47960, pageins=0)
    pmem(rss=39714816, vms=4424429568, pfaults=62855, pageins=0)
    pmem(rss=44838912, vms=4429549568, pfaults=77571, pageins=0)
    pmem(rss=50081792, vms=4434751488, pfaults=92307, pageins=0)
    pmem(rss=55312384, vms=4439973888, pfaults=107288, pageins=0)
    pmem(rss=60440576, vms=4445093888, pfaults=122711, pageins=0)
    pmem(rss=65806336, vms=4451422208, pfaults=137578, pageins=0)
    pmem(rss=70934528, vms=4456542208, pfaults=151382, pageins=0)
    

    Output with stdlib json:

    pmem(rss=17154048, vms=4385366016, pfaults=17692, pageins=0)
    pmem(rss=17317888, vms=4403191808, pfaults=32047, pageins=0)
    pmem(rss=17317888, vms=4403191808, pfaults=46541, pageins=0)
    pmem(rss=17317888, vms=4403191808, pfaults=61035, pageins=0)
    pmem(rss=17358848, vms=4403191808, pfaults=75539, pageins=0)
    pmem(rss=17383424, vms=4403191808, pfaults=90039, pageins=0)
    pmem(rss=17383424, vms=4403191808, pfaults=104533, pageins=0)
    pmem(rss=17420288, vms=4403191808, pfaults=119036, pageins=0)
    pmem(rss=17420288, vms=4403191808, pfaults=133530, pageins=0)
    pmem(rss=17420288, vms=4403191808, pfaults=148024, pageins=0)
    

    Benchmark ran on python3.7 on macos, but same leak exists on Debian with python27. You can increase the range from 10 and you will eventually run of our memory.

    Days were spent on this issue, because I would never suspect the JSON library to be at fault, but it is. I don't know if it affects Spacy in any way.

    Could be related: https://github.com/esnme/ultrajson/pull/270

    bug 
    opened by sadovnychyi 3
  • Wordwises works, X-Ray doesn't

    Wordwises works, X-Ray doesn't

    Wordwise seems to run fine, X-Ray errors :

    Traceback (most recent call last):
    File "calibre\gui2\threaded_jobs.py", line 83, in start_work
    File "calibre_plugins.worddumb.parse_job", line 37, in do_job
    File "C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\spacy_3.1.1_3.8\spacy\__init__.py", line 11, in <module>
    File "C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\thinc_8.0.8_3.8\thinc\__init__.py", line 5, in <module>
    File "C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\thinc_8.0.8_3.8\thinc\config.py", line 14, in <module>
    File "C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\srsly_2.4.1_3.8\srsly\__init__.py", line 1, in <module>
    File "C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\srsly_2.4.1_3.8\srsly\_json_api.py", line 6, in <module>
    File "C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\srsly_2.4.1_3.8\srsly\ujson\__init__.py", line 1, in <module>
    ModuleNotFoundError: No module named 'srsly.ujson.ujson'
    

    pip says srsly is installed correctly, and C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\srsly_2.4.1_3.8\srsly\ujson exists and has ujson.c inside among other files..

    (unrelated? : I had to manually add blis_0.7.4_3.8 to the worddumb-libs on Windows 10 to clear an other error)

    C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\srsly_2.4.1_3.8\srsly\ujson_init_.py line 1 is :

    from .ujson import decode, encode, dump, dumps, load, loads # noqa: F401

    Called with args: ((1587, 'MOBI', 'BBGL0Z779A', 'book.mobi', <calibre.ebooks.metadata.book.base.Metadata object at 0x082C44C0>, {'spacy': 'en_core_web_', 'wiki': 'en'}), False, True) {'notifications': <queue.Queue object at 0x082C46B8>, 'abort': <threading.Event object at 0x082C45F8>, 'log': <calibre.utils.logging.GUILog object at 0x082C45C8>}

    Windows 10 Python 3.9 Calibre - 5.24

    Installed the plugin today.

    opened by benninkcorien 2
  • Release v1.0.0 was removed from PyPI.org

    Release v1.0.0 was removed from PyPI.org

    The release package for 1.0.0 was removed from PyPI.org. Doing this causes problems for many automated build systems and pipelines. Removing packages from repos is frowned upon. Please replace the package in PyPI. It it is broken in some way, please release a v1.0.1 instead of removing 1.0.0.


    pip3 install srsly==1.0.0
    Collecting srsly==1.0.0
      Could not find a version that satisfies the requirement srsly==1.0.0 (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.1.0, 0.2.0, 2.0.0.dev0, 2.0.0.dev1, 2.0.0)
    No matching distribution found for srsly==1.0.0
    

    Releases_·_explosion_srsly srsly_·_PyPI
    opened by robbwagoner 2
  • How do I say “srsly”?

    How do I say “srsly”?

    It takes too long to verbally spell out s-r-s-l-y! How is it meant to be spoken?

    • seriously
    • serially
    • sirs-ly
    • …?
    opened by kinghuang 2
  • Rounding error in tests for x86 and aarch64

    Rounding error in tests for x86 and aarch64

    Compiling srsly 0.2.0 on ~~Arch~~ Alpine Linux I got following error in test for x86 and aarch64 architectures (https://cloud.drone.io/alpinelinux/aports/13161/1/1 , https://cloud.drone.io/alpinelinux/aports/13161/3/1):

    
    =================================== FAILURES ===================================
    --
    1770 | ____________ UltraJSONTests.test_decodeFloatingPointAdditionalTests ____________
    1771 |  
    1772 | self = <srsly.tests.ujson.test_ujson.UltraJSONTests testMethod=test_decodeFloatingPointAdditionalTests>
    1773 |  
    1774 | def test_decodeFloatingPointAdditionalTests(self):
    1775 | self.assertEqual(-1.1234567893, ujson.loads("-1.1234567893"))
    1776 | self.assertEqual(-1.234567893, ujson.loads("-1.234567893"))
    1777 | self.assertEqual(-1.34567893, ujson.loads("-1.34567893"))
    1778 | self.assertEqual(-1.4567893, ujson.loads("-1.4567893"))
    1779 | self.assertEqual(-1.567893, ujson.loads("-1.567893"))
    1780 | self.assertEqual(-1.67893, ujson.loads("-1.67893"))
    1781 | >       self.assertEqual(-1.7893, ujson.loads("-1.7893"))
    1782 | E       AssertionError: -1.7893 != -1.7893000000000001
    1783 |  
    1784 | srsly/tests/ujson/test_ujson.py:761: AssertionError
    
    
    opened by OTLabs 2
  • Add jsonl no new line append

    Add jsonl no new line append

    Hi,

    This is a PR for #13

    This should provide a way to specify whether or not you want a new line appended to the file when using write_jsonl. The current behaviour will be maintained as default.

    Please let me know if there are any further improvements that can be made.

    opened by nyejon 2
  • Bug: Cannot install if Cython is not already installed

    Bug: Cannot install if Cython is not already installed

    Summary

    The latest version 1 tag (v1.0.5) altered setup.py to directly import from Cython. This means that install fails with No module named 'Cython' unless you have manually installed it.

    Projects that use srsly as a dependency will need to be edited to manually install Cython now, whereas before setuptools handed this install.

    This also effects version 2 and the master branch.

    Edit

    I've looked into this and I think the expectation is that pyproject.toml will handle this cython pre-install. I am using python 3.6.5, I suspect this python version (or its setuptools) is not reading this file.

    Reproduce

    $ docker run -w /home/circleci circleci/python:3.6.5 bash -c "python3 -m venv venv; . venv/bin/activate; pip install srsly==1.0.5" 
    
    Collecting srsly==1.0.5
      Downloading https://files.pythonhosted.org/packages/c7/08/abe935f33b69a08d365b95e62b47ef48f93a69ab734e623248a8a4079ecb/srsly-1.0.5.tar.gz (86kB)
        Complete output from command python setup.py egg_info:
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-build-vjnjnimo/srsly/setup.py", line 11, in <module>
            from Cython.Build import cythonize
        ModuleNotFoundError: No module named 'Cython'
        
        ----------------------------------------
    Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-vjnjnimo/srsly/
    

    I give a docker route to reproduce to match the environment perfectly.

    Entities

    https://github.com/explosion/srsly/pull/44 https://github.com/explosion/srsly/blob/v1.0.5/setup.py#L11

    opened by DomHudson 2
  • Unable to import srsly.ujson.ujson for SpaCy -- AWS Lambda

    Unable to import srsly.ujson.ujson for SpaCy -- AWS Lambda

    I am trying to add SpaCy as a dependency to my Python Lambda. I am doing this by installing SpaCy as a standalone dependency inside a directory named dependencies using pip3 install spacy --no-deps -t . This is because I can't load the entire Spacy dependency inside the \tmp directory of my Lambda.

    I am able to successfully upload the folder to s3 and download it during the Lambda invocation. When I try to import spacy, I get this error: [ERROR] Runtime.ImportModuleError: Unable to import module : No module named 'srsly.ujson.ujson'.

    I manually installed srsly inside dependencies\ and I have all the files that are listed as per this link. This was referenced by this link. One of the responses says, "it seems like Python can't load it, because it's not compiled?". How would I compile a dependency which has a .c file in it?

    One other question which I found on SO is this question, but I have already manually installed srsly. How to I import the module? Thanks.


    I manually check in my code for the presence of ujson before importing spacy like this:

    if os.path.exists('/tmp/dependencies/srsly/ujson/ujson.c'):
        print('ujson exists')
    

    and the print statement gets printed.

    opened by nagarajanshanmuganthan 2
Releases(v2.4.1)
Owner
Explosion
A software company specializing in developer tools for Artificial Intelligence and Natural Language Processing
Explosion
MessagePack serializer implementation for Python msgpack.org[Python]

MessagePack for Python What's this MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JS

MessagePack 1.5k Oct 22, 2021
serialize all of python

dill serialize all of python About Dill dill extends python's pickle module for serializing and de-serializing python objects to the majority of the b

The UQ Foundation 1.5k Oct 21, 2021
Extended pickling support for Python objects

cloudpickle cloudpickle makes it possible to serialize Python constructs not supported by the default pickle module from the Python standard library.

null 1k Oct 21, 2021
Python wrapper around rapidjson

python-rapidjson Python wrapper around RapidJSON Authors: Ken Robbins <[email protected]> Lele Gaifax <[email protected]> License: MIT License Sta

null 430 Oct 20, 2021
Python bindings for the simdjson project.

pysimdjson Python bindings for the simdjson project, a SIMD-accelerated JSON parser. If SIMD instructions are unavailable a fallback parser is used, m

Tyler Kennedy 490 Oct 26, 2021
A lightweight library for converting complex objects to and from simple Python datatypes.

marshmallow: simplified object serialization marshmallow is an ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, t

marshmallow-code 5.8k Oct 16, 2021
FlatBuffers: Memory Efficient Serialization Library

FlatBuffers FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serializ

Google 16.9k Oct 24, 2021
simplejson is a simple, fast, extensible JSON encoder/decoder for Python

simplejson simplejson is a simple, fast, complete, correct and extensible JSON <http://json.org> encoder and decoder for Python 3.3+ with legacy suppo

null 1.4k Oct 16, 2021
Python library for serializing any arbitrary object graph into JSON. It can take almost any Python object and turn the object into JSON. Additionally, it can reconstitute the object back into Python.

jsonpickle jsonpickle is a library for the two-way conversion of complex Python objects and JSON. jsonpickle builds upon the existing JSON encoders, s

null 968 Oct 24, 2021
Crappy tool to convert .scw files to .json and and vice versa.

SCW-JSON-TOOL Crappy tool to convert .scw files to .json and vice versa. How to use Run main.py file with two arguments: python main.py <scw2json or j

Fred31 5 May 14, 2021
Protocol Buffers - Google's data interchange format

Protocol Buffers - Google's data interchange format Copyright 2008 Google Inc. https://developers.google.com/protocol-buffers/ Overview Protocol Buffe

Protocol Buffers 51.3k Oct 23, 2021
Ultra fast JSON decoder and encoder written in C with Python bindings

UltraJSON UltraJSON is an ultra fast JSON encoder and decoder written in pure C with bindings for Python 3.6+. Install with pip: $ python -m pip insta

null 3.4k Oct 21, 2021
Generic ASN.1 library for Python

ASN.1 library for Python This is a free and open source implementation of ASN.1 types and codecs as a Python package. It has been first written to sup

Ilya Etingof 189 Oct 22, 2021