Python wrapper around rapidjson

Last update: Jan 4, 2023

Related tags

Overview

python-rapidjson

Python wrapper around RapidJSON

Authors:	Ken Robbins <[email protected]> Lele Gaifax <[email protected]>
License:	MIT License
Status:

RapidJSON is an extremely fast C++ JSON parser and serialization library: this module wraps it into a Python 3 extension, exposing its serialization/deserialization (to/from either bytes, str or file-like instances) and JSON Schema validation capabilities.

Latest version documentation is automatically rendered by Read the Docs.

Getting Started

First install python-rapidjson:

$ pip install python-rapidjson

or, if you prefer Conda:

$ conda install -c conda-forge python-rapidjson

Basic usage looks like this:

>>> import rapidjson
>>> data = {'foo': 100, 'bar': 'baz'}
>>> rapidjson.dumps(data)
'{"bar":"baz","foo":100}'
>>> rapidjson.loads('{"bar":"baz","foo":100}')
{'bar': 'baz', 'foo': 100}
>>>
>>> class Stream:
...   def write(self, data):
...      print("Chunk:", data)
...
>>> rapidjson.dump(data, Stream(), chunk_size=5)
Chunk: b'{"foo'
Chunk: b'":100'
Chunk: b',"bar'
Chunk: b'":"ba'
Chunk: b'z"}'

Development

If you want to install the development version (maybe to contribute fixes or enhancements) you may clone the repository:

$ git clone --recursive https://github.com/python-rapidjson/python-rapidjson.git

Note

The --recursive option is needed because we use a submodule to include RapidJSON sources. Alternatively you can do a plain clone immediately followed by a git submodule update --init.

Alternatively, if you already have (a compatible version of) RapidJSON includes around, you can compile the module specifying their location with the option --rj-include-dir, for example:

$ python3 setup.py build --rj-include-dir=/usr/include/rapidjson

A set of makefiles implement most common operations, such as build, check and release; see make help output for a list of available targets.

Performance

python-rapidjson tries to be as performant as possible while staying compatible with the json module.

See the this section in the documentation for a comparison with other JSON libraries.

Incompatibility

Although we tried to implement an API similar to the standard library json, being a strict drop-in replacement in not our goal and we have decided to depart from there in some aspects. See this section in the documentation for further details.

Comments

unix timestamp

Rapidjson supports ISO8601 format for datetime object in dump. Possible to add support for unix timestamp? (can be handled with "default" parameter in dump but it is nice to have it built-in).

opened by alka2 39
Unable to compile MacOS wheel for Python 3.4

Under MacOS/clang, the compilation of the module with Python 3.4 runtime fails, see https://travis-ci.org/python-rapidjson/python-rapidjson/jobs/273856525

opened by lelit 22

Option for calling default method for dictionaries with non string keys possible ?

In order to allow serializejson to serialize dictionaries with no string keys, and enable serializejson to be a drop-in replacement for the Python pickle package, I wonder if it would be possible to add an option for automatically check if dictionaries contains only string keys, and call the "default" method if not, instead of raising a "TypeError: keys must be strings" .

In the following code, I'm able to serialize a dictionary containing non string keys, by simulating call to default method for this dictionary.

from serializejson import dumps,loads

def default(inst): 
    if isinstance(inst,dict):
        return {
              "__class__":"__main__.dict_json_keys",
              "__init__": {dumps(key,indent=None):value for key, value in inst.items()}
             } 
    return inst
        
def dict_json_keys(**args):
    return {loads(key):value for key,value in args.items()}
        
obj = {
        "string" : "str",
        b"bytes" : "bytes",
        True: "bool" , 
        2 : "int",
        3.4 : "float",
        (5,6) : "tuple",
        frozenset([7,8]) : "frozenset"
    }
print(obj)
"""{'string': 'str', b'bytes': 'bytes', True: 'bool', 2: 'int', 3.4: 'float', (5, 6): 'tuple', frozenset({8, 7}): 'frozenset'}"""

dumped = dumps(default(obj))

"""
{
    "__class__": "__main__.dict_json_keys",
    "__init__": {
            "\"string\"": "str",
            "2": "int",
            "3.4": "float",
            "true": "bool",
            "{\"__class__\":\"bytes\",\"__init__\":[\"bytes\",\"ascii\"]}": "bytes",
            "{\"__class__\":\"frozenset\",\"__init__\":[8,7]}": "frozenset",
            "{\"__class__\":\"tuple\",\"__init__\":[5,6]}": "tuple"
    }
}
"""

print(dumped)    
loaded = loads(dumped,authorized_classes = [dict_json_keys])
assert obj == loaded

Maybe dictionaries returned by default should not be checked for only string keys, to avoid double checking and consuming too much cpu. In addition, we should check before starting to serialize a dictionary, rather than calling default only when it encounters a TypeError, because if you write in a file stream, we will not be able to go back if we have started to serialize the beginning of the dictionary.

What do you think about it ? Do you think it's feasible ?

Thank you very much for your amazing python-rapidjson package .

Baptiste

opened by SmartAudioTools 21

Memory leaking in load

I see constantly growing memory with rapidjson.load comparing to standard json.load I unpacked about 10k gzipped json files like this

for file_name in os.listdir(path):
   with open(os.path.join(path, file_name), 'rb') as f:
        content = f.read()
        with BytesIO(contents) as file_object:
            with gzip.GzipFile(fileobj=file_object, filename='objects.json') as archive:
                return rapidjson.load(archive)

Memory consumption of rapidjson With json lib memory is almost constant.

opened by KKomarov 19

Generalize support for lists and tuples to extend to iterators

It would be great to be able to serialize, for example, a generator as that can be more efficient in terms of memory consumption; instead of having to construct a list of N items prior to serializing them, python-rapidjson could consume items from a generator and produce a JSON list.

opened by hexaclock 15
Benchmark: Might be worth including hyperjson and orjson

Hi!

The benchmark at https://python-rapidjson.readthedocs.io/en/latest/benchmarks.html does not include the potentially most interesting contender — orjson — now that ujson and yajl are dead. Are there plans benchmarking against orjson?

Best, Sebastian

opened by hartwork 14

Warnings during compile for v0.2.7

I was updating the conda-forge feedstock to 0.2.7 and noticed the following warnings in the builds for linux and os x. Full logs here: https://circleci.com/gh/conda-forge/python-rapidjson-feedstock/tree/pull%2F6 https://travis-ci.org/conda-forge/python-rapidjson-feedstock/builds/314361977

cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
 };
 ^
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
 };
 ^
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]

./rapidjson.cpp:1136:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"datetime_mode",

     ^

./rapidjson.cpp:1137:63: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(DecoderObject, datetimeMode), READONLY, "datetime_mode"},

                                                              ^

./rapidjson.cpp:1138:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"uuid_mode",

     ^

./rapidjson.cpp:1139:59: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(DecoderObject, uuidMode), READONLY, "uuid_mode"},

                                                          ^

./rapidjson.cpp:1140:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"number_mode",

     ^

./rapidjson.cpp:1141:61: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(DecoderObject, numberMode), READONLY, "number_mode"},

                                                            ^

./rapidjson.cpp:1142:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"parse_mode",

     ^

./rapidjson.cpp:1143:60: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(DecoderObject, parseMode), READONLY, "parse_mode"},

                                                           ^

./rapidjson.cpp:2233:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"skip_invalid_keys",

     ^

./rapidjson.cpp:2234:66: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_BOOL, offsetof(EncoderObject, skipInvalidKeys), READONLY, "skip_invalid_keys"},

                                                                 ^

./rapidjson.cpp:2235:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"ensure_ascii",

     ^

./rapidjson.cpp:2236:62: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_BOOL, offsetof(EncoderObject, ensureAscii), READONLY, "ensure_ascii"},

                                                             ^

./rapidjson.cpp:2237:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"indent",

     ^

./rapidjson.cpp:2238:57: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(EncoderObject, indent), READONLY, "indent"},

                                                        ^

./rapidjson.cpp:2239:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"sort_keys",

     ^

./rapidjson.cpp:2240:62: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_BOOL, offsetof(EncoderObject, ensureAscii), READONLY, "sort_keys"},

                                                             ^

./rapidjson.cpp:2241:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"max_recursion_depth",

     ^

./rapidjson.cpp:2242:68: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(EncoderObject, maxRecursionDepth), READONLY, "max_recursion_depth"},

                                                                   ^

./rapidjson.cpp:2243:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"datetime_mode",

     ^

./rapidjson.cpp:2244:63: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(EncoderObject, datetimeMode), READONLY, "datetime_mode"},

                                                              ^

./rapidjson.cpp:2245:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"uuid_mode",

     ^

./rapidjson.cpp:2246:59: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(EncoderObject, uuidMode), READONLY, "uuid_mode"},

                                                          ^

./rapidjson.cpp:2247:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"number_mode",

     ^

./rapidjson.cpp:2248:61: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(EncoderObject, numberMode), READONLY, "number_mode"},

                                                            ^

24 warnings generated.

opened by hajapy 14

object_hook memory leak

OS: MacOS High Sierra Python version: 3.7.1 Rapidjson version: 0.6.3

It seems I am facing a memory leak problem when using rapidjson.loads and utilizing an object hook. Here is an example code:

import rapidjson
from datetime import timedelta
import tracemalloc

tracemalloc.start()

def object_hook(td):
    if '__td__' in td:
        return timedelta(td['__td__'])
    else:
        return td

def parse_json():
    data = []
    for i in range(1, 100):
        data.append({"name": f"a{i}", "timestamp": timedelta(seconds=i)})

    a = rapidjson.dumps(
        data,
        datetime_mode=rapidjson.DM_ISO8601,
        default=lambda td: {"__td__": td.total_seconds()} if isinstance(td, timedelta) else td,
    )

    data = rapidjson.loads(
        a,
        datetime_mode=rapidjson.DM_ISO8601,
        object_hook=object_hook
    )
    return data

print("before")
if tracemalloc.is_tracing():
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('lineno')
    print([str(x) for x in top_stats[:10]])

for _ in range(1000):
    parse_json()

print("after")
if tracemalloc.is_tracing():
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('lineno')
    print([str(x) for x in top_stats[:10]])

After running this I get: 'memleaktest.py:9: size=3867 KiB, count=99000, average=40 B' as the first line of the tracemalloc output. I tried using objgraph as well, but it did not show any Python objects, so it seems like a C++ malloc issue here.

I am using rapidjson in production and my, fairly simple, API is slowly eating memory and this is the prime suspect (getting a tracemalloc pointer to the line basically the same as the one here).

Is this working as intended? Do you need any more information to pinpoint the problem?

opened by mklokocka 13

Benchmarks and ensure_ascii=False

Original JSON (Python 3.6) performance differs completely in performance for unicode strings that contain non-us-ascii characters. So, could you provide performance comparison for case ensure_ascii=False and national symbols in source strings? I mean dumps() / dump(). This is important since size of JSON document with ensure_ascii=True is mush bigger than with False.

for example, .dumps('привет, мир!' * 1000)

opened by socketpair 12
Handle serialization/deserialization of date, time and datetime instances

Serialize also date and time objects, and implement the complementary operation recognizing strings containing such values.

A new option, “DATETIME_MODE_ISO8601_UTC”, shifts the values to UTC.

This is basically an extended version of the datetime support implemented in https://github.com/lelit/nssjson.

opened by lelit 12

Tests fail: unknown hook 'pytest_benchmark_group_stats' in plugin ...

==================================================================================== test session starts =====================================================================================
platform freebsd13 -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /usr/local/bin/python3.8
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/disk-samsung/freebsd-ports/devel/py-python-rapidjson/work-py38/python-rapidjson-1.6/.hypothesis/examples')
rootdir: /disk-samsung/freebsd-ports/devel/py-python-rapidjson/work-py38/python-rapidjson-1.6, configfile: setup.cfg
plugins: forked-1.4.0, hypothesis-6.46.3, xdist-2.5.0, cov-2.9.0, rerunfailures-10.1, requests-mock-1.9.3, asyncio-0.18.3, flaky-3.7.0, trio-0.7.0
asyncio: mode=legacy
collected 940 items / 1 skipped                                                                                                                                                              
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 268, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 321, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 332, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 659, in perform_collect
INTERNALERROR>     self.config.pluginmanager.check_pending()
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/pluggy/_manager.py", line 262, in check_pending
INTERNALERROR>     raise PluginValidationError(
INTERNALERROR> pluggy._manager.PluginValidationError: unknown hook 'pytest_benchmark_group_stats' in plugin <module 'conftest' from '/disk-samsung/freebsd-ports/devel/py-python-rapidjson/work-py38/python-rapidjson-1.6/benchmarks/conftest.py'>

=============================================================================== 1 skipped, 1 warning in 0.71s ================================================================================

Version: 1.6 FreeBSD 13.1

opened by yurivict 10

RawJSON performance problem
Hello,

In serializejson , like you suggest in https://github.com/python-rapidjson/python-rapidjson/issues/100 , I heavily use RawJSON, in order to keep on the same line tuples or array of numerical values of the same type (single_line_array don't allow to treat differently array of same numerical types and others) But RawJSON seems to have big performance problems . Basicaly rapidjson.dumps should just copy string from RawJSON as it without any check for char needing escape or needing to be convert in ascii, but take 12x or 14x more time than a copy (20x if unicode chars) ! Do you have a idea why ? Moreover ensure_ascii True or False , should not have any performance differencies on RawJSON.

import rapidjson import copy @profile def test_RawJSON(): json =f'"[{"0,"*6000000}0]"' copie = json[:-1]+'"' # 3.6 msec rawjson = rapidjson.RawJSON(json) # 0.01 msec json_with_rawjson = rapidjson.dumps(rawjson, ensure_ascii=False) # 40 msec (12x time of a copy, for juste coping json !) json_with_rawjson_ascii = rapidjson.dumps(rawjson, ensure_ascii=True) # 50 msec (14x time of a copy, for juste coping json !) assert json == json_with_rawjson assert json == json_with_rawjson_ascii test_RawJSON()
opened by SmartAudioTools 9
set object is not JSON serializable with iterable_mode=IM_ANY_ITERABLE
>>> rapidjson.dumps({1}, iterable_mode=rapidjson.IM_ANY_ITERABLE) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: {1} is not JSON serializable

>>> rapidjson.__version__ '1.8'

Given that documentation states "any iterable will be dumped as a JSON array," I would expect a set object to serialize to an array.

Note, it does appear that creating an iterator object from the set will successfully serialize:

>>> rapidjson.dumps(iter({1}), iterable_mode=rapidjson.IM_ANY_ITERABLE) '[1]'
opened by gdub 4
ImportError: DLL load failed: The specified module could not be found.

I get the following error with versions greater than 1.0 when importing rapidjson:

ImportError: DLL load failed: The specified module could not be found.

opened by xyxz-web 21
Have a validator's __call__ return the parsed json if no validation error is raised?

It would be nice if I could validate and loads a json string/bytes in one swell foop. Assuming that the validator must parse the json when performing validation, it should be pretty easy to return the parsed results post validation, right?

opened by jessekrubin 3
Support for Apple's arm64 architecture (osx-arm64)

Hi, I'm trying to install python-rapidjson on a new Apple MacBook with Apple Silicon, but it fails to install via Conda due to no packages matching the architecture. Would it be possible to add support for this?

Thanks! Ryan

opened by kartzke 2