Rapidjson supports ISO8601 format for datetime object in dump. Possible to add support for unix timestamp? (can be handled with "default" parameter in dump but it is nice to have it built-in).

Under MacOS/clang, the compilation of the module with Python 3.4 runtime fails, see https://travis-ci.org/python-rapidjson/python-rapidjson/jobs/273856525

In order to allow serializejson to serialize dictionaries with no string keys, and enable serializejson to be a drop-in replacement for the Python pickle package, I wonder if it would be possible to add an option for automatically check if dictionaries contains only string keys, and call the "default" method if not, instead of raising a "TypeError: keys must be strings" .

In the following code, I'm able to serialize a dictionary containing non string keys, by simulating call to default method for this dictionary.

from serializejson import dumps,loads

def default(inst): 
    if isinstance(inst,dict):
        return {
              "__class__":"__main__.dict_json_keys",
              "__init__": {dumps(key,indent=None):value for key, value in inst.items()}
             } 
    return inst
        
def dict_json_keys(**args):
    return {loads(key):value for key,value in args.items()}
        
obj = {
        "string" : "str",
        b"bytes" : "bytes",
        True: "bool" , 
        2 : "int",
        3.4 : "float",
        (5,6) : "tuple",
        frozenset([7,8]) : "frozenset"
    }
print(obj)
"""{'string': 'str', b'bytes': 'bytes', True: 'bool', 2: 'int', 3.4: 'float', (5, 6): 'tuple', frozenset({8, 7}): 'frozenset'}"""

dumped = dumps(default(obj))

"""
{
    "__class__": "__main__.dict_json_keys",
    "__init__": {
            "\"string\"": "str",
            "2": "int",
            "3.4": "float",
            "true": "bool",
            "{\"__class__\":\"bytes\",\"__init__\":[\"bytes\",\"ascii\"]}": "bytes",
            "{\"__class__\":\"frozenset\",\"__init__\":[8,7]}": "frozenset",
            "{\"__class__\":\"tuple\",\"__init__\":[5,6]}": "tuple"
    }
}
"""

print(dumped)    
loaded = loads(dumped,authorized_classes = [dict_json_keys])
assert obj == loaded

Maybe dictionaries returned by default should not be checked for only string keys, to avoid double checking and consuming too much cpu. In addition, we should check before starting to serialize a dictionary, rather than calling default only when it encounters a TypeError, because if you write in a file stream, we will not be able to go back if we have started to serialize the beginning of the dictionary.

What do you think about it ? Do you think it's feasible ?

Thank you very much for your amazing python-rapidjson package .

Baptiste

I see constantly growing memory with rapidjson.load comparing to standard json.load I unpacked about 10k gzipped json files like this

for file_name in os.listdir(path):
   with open(os.path.join(path, file_name), 'rb') as f:
        content = f.read()
        with BytesIO(contents) as file_object:
            with gzip.GzipFile(fileobj=file_object, filename='objects.json') as archive:
                return rapidjson.load(archive)

Memory consumption of rapidjson With json lib memory is almost constant.

It would be great to be able to serialize, for example, a generator as that can be more efficient in terms of memory consumption; instead of having to construct a list of N items prior to serializing them, python-rapidjson could consume items from a generator and produce a JSON list.

Hi!

The benchmark at https://python-rapidjson.readthedocs.io/en/latest/benchmarks.html does not include the potentially most interesting contender — orjson — now that ujson and yajl are dead. Are there plans benchmarking against orjson?

Best, Sebastian

I was updating the conda-forge feedstock to 0.2.7 and noticed the following warnings in the builds for linux and os x. Full logs here: https://circleci.com/gh/conda-forge/python-rapidjson-feedstock/tree/pull%2F6 https://travis-ci.org/conda-forge/python-rapidjson-feedstock/builds/314361977

cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
 };
 ^
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:1145:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
 };
 ^
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
./rapidjson.cpp:2250:1: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]

./rapidjson.cpp:1136:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"datetime_mode",

     ^

./rapidjson.cpp:1137:63: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(DecoderObject, datetimeMode), READONLY, "datetime_mode"},

                                                              ^

./rapidjson.cpp:1138:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"uuid_mode",

     ^

./rapidjson.cpp:1139:59: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(DecoderObject, uuidMode), READONLY, "uuid_mode"},

                                                          ^

./rapidjson.cpp:1140:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"number_mode",

     ^

./rapidjson.cpp:1141:61: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(DecoderObject, numberMode), READONLY, "number_mode"},

                                                            ^

./rapidjson.cpp:1142:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"parse_mode",

     ^

./rapidjson.cpp:1143:60: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(DecoderObject, parseMode), READONLY, "parse_mode"},

                                                           ^

./rapidjson.cpp:2233:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"skip_invalid_keys",

     ^

./rapidjson.cpp:2234:66: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_BOOL, offsetof(EncoderObject, skipInvalidKeys), READONLY, "skip_invalid_keys"},

                                                                 ^

./rapidjson.cpp:2235:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"ensure_ascii",

     ^

./rapidjson.cpp:2236:62: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_BOOL, offsetof(EncoderObject, ensureAscii), READONLY, "ensure_ascii"},

                                                             ^

./rapidjson.cpp:2237:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"indent",

     ^

./rapidjson.cpp:2238:57: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(EncoderObject, indent), READONLY, "indent"},

                                                        ^

./rapidjson.cpp:2239:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"sort_keys",

     ^

./rapidjson.cpp:2240:62: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_BOOL, offsetof(EncoderObject, ensureAscii), READONLY, "sort_keys"},

                                                             ^

./rapidjson.cpp:2241:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"max_recursion_depth",

     ^

./rapidjson.cpp:2242:68: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(EncoderObject, maxRecursionDepth), READONLY, "max_recursion_depth"},

                                                                   ^

./rapidjson.cpp:2243:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"datetime_mode",

     ^

./rapidjson.cpp:2244:63: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(EncoderObject, datetimeMode), READONLY, "datetime_mode"},

                                                              ^

./rapidjson.cpp:2245:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"uuid_mode",

     ^

./rapidjson.cpp:2246:59: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(EncoderObject, uuidMode), READONLY, "uuid_mode"},

                                                          ^

./rapidjson.cpp:2247:6: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

    {"number_mode",

     ^

./rapidjson.cpp:2248:61: warning: conversion from string literal to 'char *' is deprecated [-Wc++11-compat-deprecated-writable-strings]

     T_UINT, offsetof(EncoderObject, numberMode), READONLY, "number_mode"},

                                                            ^

24 warnings generated.

OS: MacOS High Sierra Python version: 3.7.1 Rapidjson version: 0.6.3

It seems I am facing a memory leak problem when using rapidjson.loads and utilizing an object hook. Here is an example code:

import rapidjson
from datetime import timedelta
import tracemalloc

tracemalloc.start()

def object_hook(td):
    if '__td__' in td:
        return timedelta(td['__td__'])
    else:
        return td

def parse_json():
    data = []
    for i in range(1, 100):
        data.append({"name": f"a{i}", "timestamp": timedelta(seconds=i)})

    a = rapidjson.dumps(
        data,
        datetime_mode=rapidjson.DM_ISO8601,
        default=lambda td: {"__td__": td.total_seconds()} if isinstance(td, timedelta) else td,
    )

    data = rapidjson.loads(
        a,
        datetime_mode=rapidjson.DM_ISO8601,
        object_hook=object_hook
    )
    return data

print("before")
if tracemalloc.is_tracing():
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('lineno')
    print([str(x) for x in top_stats[:10]])

for _ in range(1000):
    parse_json()

print("after")
if tracemalloc.is_tracing():
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('lineno')
    print([str(x) for x in top_stats[:10]])

After running this I get: 'memleaktest.py:9: size=3867 KiB, count=99000, average=40 B' as the first line of the tracemalloc output. I tried using objgraph as well, but it did not show any Python objects, so it seems like a C++ malloc issue here.

I am using rapidjson in production and my, fairly simple, API is slowly eating memory and this is the prime suspect (getting a tracemalloc pointer to the line basically the same as the one here).

Is this working as intended? Do you need any more information to pinpoint the problem?

Original JSON (Python 3.6) performance differs completely in performance for unicode strings that contain non-us-ascii characters. So, could you provide performance comparison for case ensure_ascii=False and national symbols in source strings? I mean dumps() / dump(). This is important since size of JSON document with ensure_ascii=True is mush bigger than with False.

for example, .dumps('привет, мир!' * 1000)

Serialize also date and time objects, and implement the complementary operation recognizing strings containing such values.

A new option, “DATETIME_MODE_ISO8601_UTC”, shifts the values to UTC.

This is basically an extended version of the datetime support implemented in https://github.com/lelit/nssjson.

==================================================================================== test session starts =====================================================================================
platform freebsd13 -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /usr/local/bin/python3.8
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/disk-samsung/freebsd-ports/devel/py-python-rapidjson/work-py38/python-rapidjson-1.6/.hypothesis/examples')
rootdir: /disk-samsung/freebsd-ports/devel/py-python-rapidjson/work-py38/python-rapidjson-1.6, configfile: setup.cfg
plugins: forked-1.4.0, hypothesis-6.46.3, xdist-2.5.0, cov-2.9.0, rerunfailures-10.1, requests-mock-1.9.3, asyncio-0.18.3, flaky-3.7.0, trio-0.7.0
asyncio: mode=legacy
collected 940 items / 1 skipped                                                                                                                                                              
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 268, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 321, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 332, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 659, in perform_collect
INTERNALERROR>     self.config.pluginmanager.check_pending()
INTERNALERROR>   File "/usr/local/lib/python3.8/site-packages/pluggy/_manager.py", line 262, in check_pending
INTERNALERROR>     raise PluginValidationError(
INTERNALERROR> pluggy._manager.PluginValidationError: unknown hook 'pytest_benchmark_group_stats' in plugin <module 'conftest' from '/disk-samsung/freebsd-ports/devel/py-python-rapidjson/work-py38/python-rapidjson-1.6/benchmarks/conftest.py'>

=============================================================================== 1 skipped, 1 warning in 0.71s ================================================================================

Version: 1.6 FreeBSD 13.1

Hello,

In serializejson , like you suggest in https://github.com/python-rapidjson/python-rapidjson/issues/100 , I heavily use RawJSON, in order to keep on the same line tuples or array of numerical values of the same type (single_line_array don't allow to treat differently array of same numerical types and others) But RawJSON seems to have big performance problems . Basicaly rapidjson.dumps should just copy string from RawJSON as it without any check for char needing escape or needing to be convert in ascii, but take 12x or 14x more time than a copy (20x if unicode chars) ! Do you have a idea why ? Moreover ensure_ascii True or False , should not have any performance differencies on RawJSON.

import rapidjson
import copy
@profile
def test_RawJSON():
    json  =f'"[{"0,"*6000000}0]"'  
    copie = json[:-1]+'"' #  3.6 msec 
    rawjson  = rapidjson.RawJSON(json) # 0.01 msec
    json_with_rawjson = rapidjson.dumps(rawjson, ensure_ascii=False)   # 40 msec  (12x time of a copy, for juste coping json !)
    json_with_rawjson_ascii = rapidjson.dumps(rawjson, ensure_ascii=True)   # 50 msec  (14x time of a copy, for juste coping json !)
    assert json == json_with_rawjson
    assert json == json_with_rawjson_ascii
test_RawJSON()

>>> rapidjson.dumps({1}, iterable_mode=rapidjson.IM_ANY_ITERABLE)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: {1} is not JSON serializable

>>> rapidjson.__version__
'1.8'

Given that documentation states "any iterable will be dumped as a JSON array," I would expect a set object to serialize to an array.

Note, it does appear that creating an iterator object from the set will successfully serialize:

>>> rapidjson.dumps(iter({1}), iterable_mode=rapidjson.IM_ANY_ITERABLE)
'[1]'

I get the following error with versions greater than 1.0 when importing rapidjson:

ImportError: DLL load failed: The specified module could not be found.

It would be nice if I could validate and loads a json string/bytes in one swell foop. Assuming that the validator must parse the json when performing validation, it should be pretty easy to return the parsed results post validation, right?

Hi, I'm trying to install python-rapidjson on a new Apple MacBook with Apple Silicon, but it fails to install via Conda due to no packages matching the architecture. Would it be possible to add support for this?

Thanks! Ryan