A fast JSON parser/generator for C++ with both SAX/DOM style API

Tencent

Last update: Dec 30, 2022

Related tags

JSON rapidjson

Overview

A fast JSON parser/generator for C++ with both SAX/DOM style API

Tencent is pleased to support the open source community by making RapidJSON available.

RapidJSON GitHub
RapidJSON Documentation
- English
- 简体中文
- GitBook with downloadable PDF/EPUB/MOBI, without API reference.

Build status

Linux	Windows	Coveralls

Introduction

RapidJSON is a JSON parser and generator for C++. It was inspired by RapidXml.

RapidJSON is small but complete. It supports both SAX and DOM style API. The SAX parser is only a half thousand lines of code.
RapidJSON is fast. Its performance can be comparable to strlen(). It also optionally supports SSE2/SSE4.2 for acceleration.
RapidJSON is self-contained and header-only. It does not depend on external libraries such as BOOST. It even does not depend on STL.
RapidJSON is memory-friendly. Each JSON value occupies exactly 16 bytes for most 32/64-bit machines (excluding text string). By default it uses a fast memory allocator, and the parser allocates memory compactly during parsing.
RapidJSON is Unicode-friendly. It supports UTF-8, UTF-16, UTF-32 (LE & BE), and their detection, validation and transcoding internally. For example, you can read a UTF-8 file and let RapidJSON transcode the JSON strings into UTF-16 in the DOM. It also supports surrogates and "\u0000" (null character).

Highlights in v1.1 (2016-8-25)

Added JSON Pointer
Added JSON Schema
Added relaxed JSON syntax (comment, trailing comma, NaN/Infinity)
Iterating array/object with C++11 Range-based for loop
Reduce memory overhead of each Value from 24 bytes to 16 bytes in x86-64 architecture.

For other changes please refer to change log.

Compatibility

RapidJSON is cross-platform. Some platform/compiler combinations which have been tested are shown as follows.

Visual C++ 2008/2010/2013 on Windows (32/64-bit)
GNU C++ 3.8.x on Cygwin
Clang 3.4 on Mac OS X (32/64-bit) and iOS
Clang 3.4 on Android NDK

Users can build and run the unit tests on their platform/compiler.

Installation

RapidJSON is a header-only C++ library. Just copy the include/rapidjson folder to system or project's include path.

Alternatively, if you are using the vcpkg dependency manager you can download and install rapidjson with CMake integration in a single command:

vcpkg install rapidjson

RapidJSON uses following software as its dependencies:

CMake as a general build tool
(optional) Doxygen to build documentation
(optional) googletest for unit and performance testing

To generate user documentation and run tests please proceed with the steps below:

Execute git submodule update --init to get the files of thirdparty submodules (google test).
Create directory called build in rapidjson source directory.
Change to build directory and run cmake .. command to configure your build. Windows users can do the same with cmake-gui application.
On Windows, build the solution found in the build directory. On Linux, run make from the build directory.

On successful build you will find compiled test and example binaries in bin directory. The generated documentation will be available in doc/html directory of the build tree. To run tests after finished build please run make test or ctest from your build tree. You can get detailed output using ctest -V command.

It is possible to install library system-wide by running make install command from the build tree with administrative privileges. This will install all files according to system preferences. Once RapidJSON is installed, it is possible to use it from other CMake projects by adding find_package(RapidJSON) line to your CMakeLists.txt.

Usage at a glance

This simple example parses a JSON string into a document (DOM), make a simple modification of the DOM, and finally stringify the DOM to a JSON string.

using namespace rapidjson;

int main() {
    // 1. Parse a JSON string into DOM.
    const char* json = "{\"project\":\"rapidjson\",\"stars\":10}";
    Document d;
    d.Parse(json);

    // 2. Modify it by DOM.
    Value& s = d["stars"];
    s.SetInt(s.GetInt() + 1);

    // 3. Stringify the DOM
    StringBuffer buffer;
    Writer writer(buffer);
    d.Accept(writer);

    // Output {"project":"rapidjson","stars":11}
    std::cout << buffer.GetString() << std::endl;
    return 0;
}
">
 // rapidjson/example/simpledom/simpledom.cpp`
#include "rapidjson/document.h"
#include "rapidjson/writer.h"
#include "rapidjson/stringbuffer.h"
#include <iostream>

using namespace rapidjson;

int main() {
    // 1. Parse a JSON string into DOM.
    const char* json = "{\"project\":\"rapidjson\",\"stars\":10}";
    Document d;
    d.Parse(json);

    // 2. Modify it by DOM.
    Value& s = d["stars"];
    s.SetInt(s.GetInt() + 1);

    // 3. Stringify the DOM
    StringBuffer buffer;
    Writer writer(buffer);
    d.Accept(writer);

    // Output {"project":"rapidjson","stars":11}
    std::cout << buffer.GetString() << std::endl;
    return 0;
}

Note that this example did not handle potential errors.

The following diagram shows the process.

More examples are available:

DOM API
- tutorial: Basic usage of DOM API.
SAX API
- simplereader: Dumps all SAX events while parsing a JSON by Reader.
- condense: A command line tool to rewrite a JSON, with all whitespaces removed.
- pretty: A command line tool to rewrite a JSON with indents and newlines by PrettyWriter.
- capitalize: A command line tool to capitalize strings in JSON.
- messagereader: Parse a JSON message with SAX API.
- serialize: Serialize a C++ object into JSON with SAX API.
- jsonx: Implements a JsonxWriter which stringify SAX events into JSONx (a kind of XML) format. The example is a command line tool which converts input JSON into JSONx format.
Schema
- schemavalidator : A command line tool to validate a JSON with a JSON schema.
Advanced
- prettyauto: A modified version of pretty to automatically handle JSON with any UTF encodings.
- parsebyparts: Implements an AsyncDocumentParser which can parse JSON in parts, using C++11 thread.
- filterkey: A command line tool to remove all values with user-specified key.
- filterkeydom: Same tool as above, but it demonstrates how to use a generator to populate a Document.

Contributing

RapidJSON welcomes contributions. When contributing, please follow the code below.

Issues

Feel free to submit issues and enhancement requests.

Please help us by providing minimal reproducible examples, because source code is easier to let other people understand what happens. For crash problems on certain platforms, please bring stack dump content with the detail of the OS, compiler, etc.

Please try breakpoint debugging first, tell us what you found, see if we can start exploring based on more information been prepared.

Workflow

In general, we follow the "fork-and-pull" Git workflow.

Fork the repo on GitHub
Clone the project to your own machine
Checkout a new branch on your fork, start developing on the branch
Test the change before commit, Make sure the changes pass all the tests, including unittest and preftest, please add test case for each new feature or bug-fix if needed.
Commit changes to your own branch
Push your work back up to your fork
Submit a Pull request so that we can review your changes

NOTE: Be sure to merge the latest from "upstream" before making a pull request!

Copyright and Licensing

You can copy and paste the license summary from below.

Tencent is pleased to support the open source community by making RapidJSON available.

Copyright (C) 2015 THL A29 Limited, a Tencent company, and Milo Yip.

Licensed under the MIT License (the "License"); you may not use this file except
in compliance with the License. You may obtain a copy of the License at

http://opensource.org/licenses/MIT

Unless required by applicable law or agreed to in writing, software distributed 
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR 
CONDITIONS OF ANY KIND, either express or implied. See the License for the 
specific language governing permissions and limitations under the License.

Comments

Add schema error detection to GenericSchemaDocument

Fixes #1841 by adding detection of schema errors that would not be found by meta-schema validation. It works in a similar way to the validator errors, and creates a very similar error structure which you can access using GenericSchemaDocument::GetError(). The majority of errors relate to the processing of $ref keyword. Also moves Pointer errors into the error framework. Also improves the validation error for matching multiple oneOf items by including the indices of the first two matches.

opened by smhdfdl 26

How to store parsed JSON objects in a map

Hello I'd like to store parsed JSON objects in a static map to access it later and not to parse it again, is it possible? Private constructor doesn't allow to store Document objects, and their references get invalid after return.

typedef rapidjson::Document Json;

static std::unordered_map<size_t, JsonParser::Json&> mJsonMap;
static std::hash<std::string> hashFn;

    JsonParser::Json&
    JsonParser::parse(std::string jsonString)
    {
        size_t hash = hashFn(jsonString);
        
        if (mJsonMap.count(hash) == 0) {
            JsonParser::Json json;
            json.Parse(jsonString.c_str());
            mJsonMap.insert({hash, json});
        }
        
        return mJsonMap.at(hash);
    }

question

opened by schmidt9 23

Utilize SIMD to find expected end of token
picohttpparser uses SIMD to scan a string in order to find a character in a string. We can use it or something similar to be able to determine the end of the token. Say we have the following JSON:

{ "a": 123, "b": "fd" }

Instead of scanning each character we can find the boundaries of the integer value (that is ':' and ',') which will allow us to use the existing buffer sliced from start to end in order to cast the value to integer. There can be multiple characters to look ahead for like newline or " depending on the type of what is being parsed on the contents of the JSON file. If we'd be able to optimize for the common cases I think it can be useful.
performance
opened by thedrow 22

Question about the correct usage of AddMember and Value

Hi to all I have a question regarding the correct usage of AddMember and Value. I read doc many times, how to create Value and add member with StringRef but still random crashes. (1, 2) they crash, (3,4) I hope that they are ok.

So please correct me if I use the methods in the wrong way. Or link with more examples with different literal and constructs types are welcome. RAPIDJSON_HAS_STDSTRING is set to default (0).

Error that i get many times but random, differ on used addMember or Value function.

(rapidjson::GenericValuerapidjson::UTF8<char, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator >::AddMember(rapidjson::GenericStringRef, rapidjson::GenericValuerapidjson::UTF8<char, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator >&&, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator&)+66)

/// defined in app scope
static const char* LONG_LIVING_KEY = "LONG_LIVING_KEY";
static const char* LONG_LIVING_VALUE = "LONG_LIVING_VALUE";

// utility to convert json to string representation
template<>
std::string AHXLocalData::serialize< rapidjson::Document >(const rapidjson::Document &data){
    rapidjson::StringBuffer strbuf;
    rapidjson::Writer <rapidjson::StringBuffer> writer(strbuf);
    data.Accept(writer);
    return strbuf.GetString();
}

// crash
// using static const char* as key and value with StringRef
// inside class method
rapidjson::Document doc;
doc.SetObject();
rapidjson::Document::AllocatorType& allocator = doc.GetAllocator();
doc.AddMember(rapidjson::StringRef(LONG_LIVING_KEY), rapidjson::Value(rapidjson::StringRef(LONG_LIVING_VALUE)), allocator);
// will crash sometimes on rapidjson::Value( and addMember(
logEvent(serialize(doc).c_str());

// crash
// using static const char* as key and value, StringRef only on key
// inside class method
rapidjson::Document doc;
doc.SetObject();
rapidjson::Document::AllocatorType& allocator = doc.GetAllocator();
doc.AddMember(rapidjson::StringRef(LONG_LIVING_KEY), rapidjson::Value(LONG_LIVING_VALUE), allocator);
// will crash sometimes on rapidjson::Value( and addMember(
logEvent(serialize(doc).c_str());

// not crash
// using static const char* as key , value created from std::string
// inside class method
rapidjson::Document doc;
doc.SetObject();

std::string levelName = getTheLevelName(); // return std::string
rapidjson::Value rjv;
rjv.SetString(levelName.c_str(), allocator)
doc.AddMember(rapidjson::StringRef(LONG_LIVING_KEY), rjv, allocator);
logEvent(serialize(doc).c_str());

// not crash
// using compiled literal key and value
// inside class method
rapidjson::Document doc;
doc.SetObject();

rapidjson::Value rjv;
doc.AddMember("from", "dialog", allocator);
logEvent(serialize(doc).c_str());

Thank you for your time.

bug

opened by whitegfx 22

Deb packaging with CPack

Hello! First we would like to thank you for a nice library. Some of our Ubuntu-based solutions rely on Rapidjson, so we would like to make the use of it a bit more standardized. At the moment our projects acquire Rapidjson as a git submodule, but it's not very convenient due to the absence of of version control which is prefferable for our build system. That's why we would like to have packages.

This patch provides the opportunity for building binary deb packages with CPack tool (full-featured debianization seem to be overkill for this case). Everything is based on your actual CMakeLists.txt. So far we got two packages (according to the components specified in CMakeLists.txt):

rapidjson-dev
rapidjson-examples

root@9497495eef1e:~/rapidjson# cmake .
root@9497495eef1e:~/rapidjson# cpack
CPack: Create package using TGZ
CPack: Install projects
CPack: - Run preinstall target for: RapidJSON
CPack: - Install project: RapidJSON
CPack: Create package
CPack: - package: /root/rapidjson/rapidjson_0.12.0~git20151704-1_all.tar.gz generated.
CPack: Create package using DEB
CPack: Install projects
CPack: - Run preinstall target for: RapidJSON
CPack: - Install project: RapidJSON
CPack: -   Install component: dev
CPack: -   Install component: examples
CPack: Create package
CPack: - package: /root/rapidjson/rapidjson_0.12.0~git20151704-1_all-dev.deb generated.
CPack: - package: /root/rapidjson/rapidjson_0.12.0~git20151704-1_all-examples.deb generated.

You may check the content of packages:

root@9497495eef1e:~/rapidjson# dpkg -c rapidjson_0.12.0~git20151704-1_all-dev.deb
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/include/
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/include/rapidjson/
-rw-r--r-- root/root      2563 2015-04-17 11:55 ./usr/local/include/rapidjson/memorybuffer.h
-rw-r--r-- root/root      2882 2015-04-17 11:55 ./usr/local/include/rapidjson/filewritestream.h
-rw-r--r-- root/root     59281 2015-04-17 11:55 ./usr/local/include/rapidjson/reader.h
-rw-r--r-- root/root     14133 2015-04-17 11:55 ./usr/local/include/rapidjson/writer.h
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/include/rapidjson/error/
-rw-r--r-- root/root      3677 2015-04-17 11:55 ./usr/local/include/rapidjson/error/en.h
-rw-r--r-- root/root      5677 2015-04-17 11:55 ./usr/local/include/rapidjson/error/error.h
-rw-r--r-- root/root      3167 2015-04-17 11:55 ./usr/local/include/rapidjson/stringbuffer.h
-rw-r--r-- root/root      9903 2015-04-17 11:55 ./usr/local/include/rapidjson/encodedstream.h
-rw-r--r-- root/root      2471 2015-04-17 11:55 ./usr/local/include/rapidjson/memorystream.h
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/include/rapidjson/msinttypes/
-rw-r--r-- root/root      8372 2015-04-17 11:55 ./usr/local/include/rapidjson/msinttypes/inttypes.h
-rw-r--r-- root/root      9327 2015-04-17 11:55 ./usr/local/include/rapidjson/msinttypes/stdint.h
-rw-r--r-- root/root      9993 2015-04-17 11:55 ./usr/local/include/rapidjson/allocators.h
-rw-r--r-- root/root     23547 2015-04-17 11:55 ./usr/local/include/rapidjson/encodings.h
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/include/rapidjson/internal/
-rw-r--r-- root/root      3621 2015-04-17 11:55 ./usr/local/include/rapidjson/internal/pow10.h
-rw-r--r-- root/root      8134 2015-04-17 11:55 ./usr/local/include/rapidjson/internal/strtod.h
-rw-r--r-- root/root     10306 2015-04-17 11:55 ./usr/local/include/rapidjson/internal/itoa.h
-rw-r--r-- root/root      6781 2015-04-17 11:55 ./usr/local/include/rapidjson/internal/dtoa.h
-rw-r--r-- root/root     11227 2015-04-17 11:55 ./usr/local/include/rapidjson/internal/diyfp.h
-rw-r--r-- root/root      1419 2015-04-17 11:55 ./usr/local/include/rapidjson/internal/strfunc.h
-rw-r--r-- root/root      6841 2015-04-17 11:55 ./usr/local/include/rapidjson/internal/meta.h
-rw-r--r-- root/root      8833 2015-04-17 11:55 ./usr/local/include/rapidjson/internal/biginteger.h
-rw-r--r-- root/root      5766 2015-04-17 11:55 ./usr/local/include/rapidjson/internal/stack.h
-rw-r--r-- root/root      2886 2015-04-17 11:55 ./usr/local/include/rapidjson/internal/ieee754.h
-rw-r--r-- root/root     21431 2015-04-17 11:55 ./usr/local/include/rapidjson/rapidjson.h
-rw-r--r-- root/root      7926 2015-04-17 11:55 ./usr/local/include/rapidjson/prettywriter.h
-rw-r--r-- root/root      2889 2015-04-17 11:55 ./usr/local/include/rapidjson/filereadstream.h
-rw-r--r-- root/root     85502 2015-04-17 11:55 ./usr/local/include/rapidjson/document.h
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/lib/
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/lib/cmake/
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/lib/cmake/RapidJSON/
-rw-r--r-- root/root       455 2015-04-17 12:00 ./usr/local/lib/cmake/RapidJSON/RapidJSONConfigVersion.cmake
-rw-r--r-- root/root       197 2015-04-17 12:00 ./usr/local/lib/cmake/RapidJSON/RapidJSONConfig.cmake
root@9497495eef1e:~/rapidjson# dpkg -c rapidjson_0.12.0~git20151704-1_all-examples.deb
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/RapidJSON/
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/RapidJSON/examples/
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/RapidJSON/examples/capitalize/
-rw-r--r-- root/root      2436 2015-04-17 11:55 ./usr/local/share/doc/RapidJSON/examples/capitalize/capitalize.cpp
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/RapidJSON/examples/prettyauto/
-rw-r--r-- root/root      2288 2015-04-17 11:55 ./usr/local/share/doc/RapidJSON/examples/prettyauto/prettyauto.cpp
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/RapidJSON/examples/messagereader/
-rw-r--r-- root/root      2686 2015-04-17 11:55 ./usr/local/share/doc/RapidJSON/examples/messagereader/messagereader.cpp
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/RapidJSON/examples/pretty/
-rw-r--r-- root/root      1036 2015-04-17 11:55 ./usr/local/share/doc/RapidJSON/examples/pretty/pretty.cpp
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/RapidJSON/examples/simplereader/
-rw-r--r-- root/root      1678 2015-04-17 11:55 ./usr/local/share/doc/RapidJSON/examples/simplereader/simplereader.cpp
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/RapidJSON/examples/simpledom/
-rw-r--r-- root/root       685 2015-04-17 11:55 ./usr/local/share/doc/RapidJSON/examples/simpledom/simpledom.cpp
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/RapidJSON/examples/tutorial/
-rw-r--r-- root/root      6407 2015-04-17 11:55 ./usr/local/share/doc/RapidJSON/examples/tutorial/tutorial.cpp
-rw-r--r-- root/root       875 2015-04-17 11:55 ./usr/local/share/doc/RapidJSON/examples/CMakeLists.txt
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/RapidJSON/examples/condense/
-rw-r--r-- root/root      1034 2015-04-17 11:55 ./usr/local/share/doc/RapidJSON/examples/condense/condense.cpp
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/RapidJSON/examples/simplewriter/
-rw-r--r-- root/root       738 2015-04-17 11:55 ./usr/local/share/doc/RapidJSON/examples/simplewriter/simplewriter.cpp
drwxr-xr-x root/root         0 2015-04-17 12:15 ./usr/local/share/doc/RapidJSON/examples/serialize/
-rw-r--r-- root/root      4161 2015-04-17 11:55 ./usr/local/share/doc/RapidJSON/examples/serialize/serialize.cpp

build

opened by vitalyisaev2 22

CMake support
This is my try to add CMake support for RapidJSON and close #82. It was tested and is working for:

Debian Linux (w/ libgtest-dev installed and w/ gtest as submodule)

Windows XP ( VS2010, Cygwin); Building on MSYS/MinGW give many errors on gtest compilation it seems the last supported version of gtest for MSYS is 1.6.0, although it is not enough for RapidJSON to build using this version since there is no method EqFailure (see https://gist.github.com/jollyroger/45f29f4769cc41c7251f)

Some features available in this pull request:

[x] In-source and out-of-source builds are supported

[x] Separate options to build tests, examples and documentation

[x] Automatic documentation building (if Doxygen is available and respective option is set)

[x] Smart Google Test Toolkit lookup w/ build options check against Win32/Unix

[x] Pkgconfig and CMake package lookup and version support (if installed with make install)

[x] Running tests with ctest (using make test of ctest -V to get verbose output as before)

[ ] Compilation flags support provided in #126 and #185

As small addition: I fixed linking errors in the documentation.
opened by jollyroger 22
Use more stdint.h types and document disabling stdint.h
Summary

Replaces #145

Rename RAPIDJSON_NO_INT64DEFINE to RAPIDJSON_NO_STDINT and revise documentation to address @pah's concern about unconditionally using int32_t/uint32_t.

As I discussed in #145, I think this change improves several things:

Better type documentation

The name of the macro RAPIDJSON_NO_STDINT more accurately describes its purpose than RAPIDJSON_NO_INT64DEFINE did. It disables the #include <stdint.h> and doesn't only disable the definition of int64_t/uint64_t. Of course, this is more significant with the usage of int32_t/uint32_t.

@pah had the concern that the type definition int32_t implied the implementation type would be restricted to 32 bits on a 64-bit system. I'm certain all 64-bit systems will use the most efficient implementation type for int32_t. To check, I looked at stdint.h on Mac OS X and Linux as well as msinttypes/stdint.h: all of them use int.
opened by spl 20

Crash in Release mode on armv7 system

I have the following simple code snippet:

std::string optimizeJson(const std::string& json)
{
    rapidjson::Document document;
    if (document.Parse(json.c_str()).HasParseError())
    {
        std::cerr << "Error parsing internal JSON!" << std::endl;
        abort();
    }

    rapidjson::StringBuffer buffer;
    rapidjson::Writer<rapidjson::StringBuffer> writer(buffer);
    document.Accept(writer);

    return std::string(buffer.GetString(), buffer.GetSize());
}

This snippet converts from pretty printed json into single line, no spaces json.

In debug mode the program works, in release mode it crashes with the following call stack:

0   rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>::Malloc(unsigned int)           0x8066af2   
1   void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseString<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char>>, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>>(rapidjson::GenericStringStream<rapidjson::UTF8<char>>&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&, bool)           0x8066ec4   
2   void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseObject<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char>>, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>>(rapidjson::GenericStringStream<rapidjson::UTF8<char>>&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&)         0x8067084   
3   void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseArray<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char>>, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>>(rapidjson::GenericStringStream<rapidjson::UTF8<char>>&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&)          0x8067c06   
4   void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseObject<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char>>, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>>(rapidjson::GenericStringStream<rapidjson::UTF8<char>>&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&)         0x80670b2   
5   void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseArray<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char>>, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>>(rapidjson::GenericStringStream<rapidjson::UTF8<char>>&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&)          0x8067c06   
6   void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseObject<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char>>, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>>(rapidjson::GenericStringStream<rapidjson::UTF8<char>>&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&)         0x80670b2   
7   (anonymous namespace)::optimizeJson(std::string const&)         0x8067688

The following define is set for RapidJSON #define RAPIDJSON_HAS_STDSTRING 1.

The release and debug binary link to the same so files. I don't see why the code would crash as it does.

opened by cristianadam 17

Crash when passing nearly-empty document

RapidJSON (debian, 0.12~git20141031-1) seems to crash when I pass it as a string the simple document "{ }". Expected is either a graceful failure in hasParseError() or a return false on subsequent hasMember() requests. Instead, the following occurs:

Program received signal SIGSEGV, Segmentation fault.
1551            data_.o.members = (Member*)allocator.Malloc(count * sizeof(Member));
(gdb) bt
#0  SetObjectRaw (allocator=..., count=0, members=0x1edc20, this=0x1edc08) at /usr/include/rapidjson/document.h:1551
#1  EndObject (memberCount=0, this=<optimized out>) at /usr/include/rapidjson/document.h:1852
#2  rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::ParseObject<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char> >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> > (this=<optimized out>, is=..., 
    handler=...) at /usr/include/rapidjson/reader.h:467
#3  0x00034662 in Parse<0u, rapidjson::GenericStringStream<rapidjson::UTF8<> >, rapidjson::GenericDocument<rapidjson::UTF8<> > > (handler=..., is=..., this=0xbefff4c8)
    at /usr/include/rapidjson/reader.h:397
#4  ParseStream<0u, rapidjson::UTF8<>, rapidjson::GenericStringStream<rapidjson::UTF8<> > > (is=..., this=0xbefff500) at /usr/include/rapidjson/document.h:1698
#5  Parse<0u, rapidjson::UTF8<> > (str=<optimized out>, this=0xbefff500) at /usr/include/rapidjson/document.h:1774
#6  Parse<0u> (str=<optimized out>, this=0xbefff500) at /usr/include/rapidjson/document.h:1783
#7  Parse (str=<optimized out>, this=0xbefff500) at /usr/include/rapidjson/document.h:1790
#8  Settings::fillByJSON (this=this@entry=0x1edbc0, json="{ }") at /Embedded/src/Util/Settings.cpp:75
#9  0x0003dd96 in ArcusView::doSettings (this=this@entry=0x97b80, msg=msg@entry=0xb3b00518) at /Embedded/src/View/ArcusView.cpp:56
#10 0x0002c17a in ArcusView::main (this=this@entry=0x97b80) at /Embedded/src/View/ArcusView.cpp:364
#11 0x0002c8a6 in main (argc=<optimized out>, argv=<optimized out>) at /Embedded/src/main.cpp:35

Although I hardly know the code, I suspect this might have something to do with malloc() being called with size 0. But I'll leave that bit to the experts! Could you look into this please?

bug

opened by RSpliet 17

Parse non-null-terminated strings / Parse with std::string

I would like to parse strings that are not null-terminated but do have a string length.

What's the best way to do that now? Implement the Stream concept? I guess it would involve something like copying the GenericStringStream, adding a length-remaining member, and modifying Peek() to return '\0' when the end is reached.

I think this would be useful for RapidJSON in general. It could be exposed as overloaded Parse(const Ch *, size_t) and ParseInsitu (Ch *, size_t) methods.
enhancement question

opened by spl 17
Stringify/parse C++ objects like in JavaScript's JSON.stringify()/JSON.parse()

As I don't want to code the whole serialization / deserialation of an object with many members by handling every value for that member, I want to have achieve this by calling one method. I would not like to do implement this for every class member because a) It takes too much time to implement b) Takes much effort to maintain if a class gets new members etc.

See this example: https://stackoverflow.com/questions/8220130/converting-c-class-to-json There is an answer (https://stackoverflow.com/a/19974486/4397899) that says ThorsSerializer would be able to do something like that, at least something more beautiful like handling every member value manually in code.

Is something similar to either JavaScript's JSON.stringify()/JSON.parse() or the way ThorsSerializer does it possible with rapidjson?

Best regards

opened by CodingSpiderFox 16

If an array is too long, the json string will be incomplete.

environment

Ubuntu 18.04.6 LTS CMake version 3.10.2 rapidjson v1.1.0 gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)

codes and questions

I am uploading the data to server by mqtt and I need to upload a lot of data in josn.

string laserScanMsger(const sensor_msgs::LaserScan::ConstPtr &ptr)
{
    auto pt = ptr.get();
    Document d;
    d.Parse(json_laserScan);
    Document::AllocatorType& allocator = d.GetAllocator();

    d["id"].SetInt64(9876);

    d["data"]["angle_min"].SetDouble(pt->angle_min);
    d["data"]["angle_max"].SetDouble(pt->angle_max);
    d["data"]["angle_increment"].SetDouble(pt->angle_increment);
    d["data"]["time_increment"].SetDouble(pt->time_increment);
    d["data"]["scan_time"].SetDouble(pt->scan_time);
    d["data"]["range_min"].SetDouble(pt->range_min);
    d["data"]["range_max"].SetDouble(pt->range_max);

    {
        int k = 0;
        auto data = pt->ranges;
        ROS_INFO("size is : %d", data.size());
        for(auto i = data.begin(); i != data.end(); i++) {
            d["data"]["ranges"].GetArray()
                .PushBack(Value().SetDouble(*i), allocator);
            ++k;
        }
        ROS_INFO("k is %d.", k);
    }
    {
        auto data = pt->intensities;
        for(auto i = data.begin(); i != data.end(); i++)
        {
            d["data"]["intensities"].GetArray()
                .PushBack(Value().SetDouble(*i), allocator);
        }
    }
    StringBuffer buffer;
    Writer<StringBuffer> writer(buffer);
    d.Accept(writer);
    ROS_INFO("String len: %d", buffer.GetSize());
    buffer.Flush();
    return buffer.GetString();
}

ROS_INFO is same as printf. the string json_laserScan is

const char json_laserScan[] = R"json(
{
	"id":123,
	"data":{
		"angle_min":1.0,
		"angle_max":1.0,
		"angle_increment":1.0,
		"time_increment":1.0,
		"scan_time":1.0,
		"range_min":1.0,
		"range_max":1.0,
		"ranges":[],
		"intensities":[]
	}
}
)json";

the length of the array named ranges is 360. this length is fixed.

I print the context of the return of laserScanMsger.

{"id":9876,"data":{"angle_min":0.0,"angle_max":6.2657318115234375,"angle_increment":0.01745329238474369,"time_increment":2.0483409457483504e-7,"scan_time":0.00016366243653465062,"range_min":0.15000000596046448,"range_max":8.0,"ranges":[1.9229999780654907,1.9190000295639038,1.9157500267028809,1.906000018119812,1.902999997138977,1.8990000486373901,1.8902499675750732,1.8892500400543213,1.8892500400543213,1.8862500190734863,1.8852499723434448,1.8852499723434448,1.8845000267028809,1.8852499723434448,1.8872499465942383,1.8872499465942383,1.8902499675750732,1.8892500400543213,1.8990000486373901,1.9019999504089355,1.8902499675750732,1.312999963760376,1.3380000591278076,1.9210000038146973,1.9320000410079956,1.934999942779541,1.941249966621399,1.9672499895095825,

the end of json string is uncomplete. I tried to print information about the size of range array.

the length of ranges is correct and the intensities array also is not outputed. the length of json string is same as the outputed size, ROS_INFO can output all context of josn string.

wants

Is there a limit in the length of the json string? How to set the limit for the length of the json string?

Others

I am Chinese. I am poor at English. If you can speak Chinese, please response to me by Chinese.

opened by Ircon 3

Security: is RapidJson a safe deserializer?

Security question:

Can we consider RapidJSON a safe deserializer?

As defined here: https://cheatsheetseries.owasp.org/cheatsheets/Deserialization_Cheat_Sheet.html

opened by BrianVaille 0
Fix the address sanitizer problems
This PR is intended to address the SIMD unit test sanitizer problem identified in issue #2019, and any other sanitizer problems that are being thrown up when running other RapidJSON unit tests under valgrind . #2019 contains the list but in summary:

SIMD.SkipWhitespace

Invalid read of size 16 (the original subject of issue #2019)

Conditional jump or move depends on uninitialised value(s)

Use of uninitialised value of size 8

Uri.Parse_UTF16 with std::string

Invalid read of size 32

I am seeing all these on master branch (Fedora 36).
opened by smhdfdl 7
Add support of parsing INFINITY and NAN according to the standard.

Variables of floating point types that have non-normal values, such as INFINITY and NAN, cannot be parsed by GenericDocument::Parse(const Ch* str) even if the flag kParseNanAndInfFlag is set, and that's due to the fact that the parser tests these exceptional values against NaN, Infinity, -Infinity, Inf and -Inf. However, according to the standard, when converting floating point types to string, e.g. using std::to_string(double), these exceptional values get converted as nan, inf or -inf! This pull request fixes this issue (#2099).

opened by ChevronOne 4
The parser cannot parse non-normal values!

Hi, The parser GenericDocument::Parse(const Ch* str) fails to parse non-normal values such as INFINITY, -INFINITY and NAN for floating point data types. These exceptional values are supported officially by the standard, however the parser still can't parse them correctly!

This is actually very argent from our end, thus any soon feedback would be appreciated;-) thanks.

opened by ChevronOne 3

Releases(v1.1.0)

v1.1.0(Aug 25, 2016)
Highlights

Added JSON Pointer

Added JSON Schema

Added relaxed JSON syntax (comment, trailing comma, NaN/Infinity)

Iterating array/object with C++11 Range-based for loop

Reduce memory overhead of each Value from 24 bytes to 16 bytes in x86-64 architecture.

Added

Add GenericDocument ctor overload to specify JSON type (#369)

Add FAQ (#372, #373, #374, #376)

Add forward declaration header fwd.h

Add @PlatformIO Library Registry manifest file (#400)

Implement assignment operator for BigInteger (#404)

Add comments support (#443)

Adding coapp definition (#460)

documenttest.cpp: EXPECT_THROW when checking empty allocator (470)

GenericDocument: add implicit conversion to ParseResult (#480)

Use <wchar.h> with C++ linkage on Windows ARM (#485)

Detect little endian for Microsoft ARM targets

Check Nan/Inf when writing a double (#510)

Add JSON Schema Implementation (#522)

Add iostream wrapper (#530)

Add Jsonx example for converting JSON into JSONx (a XML format) (#531)

Add optional unresolvedTokenIndex parameter to Pointer::Get() (#532)

Add encoding validation option for Writer/PrettyWriter (#534)

Add Writer::SetMaxDecimalPlaces() (#536)

Support {0, } and {0, m} in Regex (#539)

Add Value::Get/SetFloat(), Value::IsLossLessFloat/Double() (#540)

Add stream position check to reader unit tests (#541)

Add Templated accessors and range-based for (#542)

Add (Pretty)Writer::RawValue() (#543)

Add Document::Parse(std::string), Document::Parse(const char*, size_t length) and related APIs. (#553)

Add move constructor for GenericSchemaDocument (#554)

Add VS2010 and VS2015 to AppVeyor CI (#555)

Add parse-by-parts example (#556, #562)

Support parse number as string (#564, #589)

Add kFormatSingleLineArray for PrettyWriter (#577)

Added optional support for trailing commas (#584)

Added filterkey and filterkeydom examples (#615)

Added npm docs (#639)

Allow options for writing and parsing NaN/Infinity (#641)

Add std::string overload to PrettyWriter::Key() when RAPIDJSON_HAS_STDSTRING is defined (#698)

Fixed

Fix gcc/clang/vc warnings (#350, #394, #397, #444, #447, #473, #515, #582, #589, #595, #667)

Fix documentation (#482, #511, #550, #557, #614, #635, #660)

Fix emscripten alignment issue (#535)

Fix missing allocator to uses of AddMember in document (#365)

CMake will no longer complain that the minimum CMake version is not specified (#501)

Make it usable with old VC8 (VS2005) (#383)

Prohibit C++11 move from Document to Value (#391)

Try to fix incorrect 64-bit alignment (#419)

Check return of fwrite to avoid warn_unused_result build failures (#421)

Fix UB in GenericDocument::ParseStream (#426)

Keep Document value unchanged on parse error (#439)

Add missing return statement (#450)

Fix Document::Parse(const Ch*) for transcoding (#478)

encodings.h: fix typo in preprocessor condition (#495)

Custom Microsoft headers are necessary only for Visual Studio 2012 and lower (#559)

Fix memory leak for invalid regex (26e69ffde95ba4773ab06db6457b78f308716f4b)

Fix a bug in schema minimum/maximum keywords for 64-bit integer (e7149d665941068ccf8c565e77495521331cf390)

Fix a crash bug in regex (#605)

Fix schema "required" keyword cannot handle duplicated keys (#609)

Fix cmake CMP0054 warning (#612)

Added missing include guards in istreamwrapper.h and ostreamwrapper.h (#634)

Fix undefined behaviour (#646)

Fix buffer overrun using PutN (#673)

Fix rapidjson::value::Getstd::string() may returns wrong data (#681)

Add Flush() for all value types (#689)

Handle malloc() fail in PoolAllocator (#691)

Fix builds on x32 platform. #703

Changed

Clarify problematic JSON license (#392)

Move Travis to container based infrastructure (#504, #558)

Make whitespace array more compact (#513)

Optimize Writer::WriteString() with SIMD (#544)

x86-64 48-bit pointer optimization for GenericValue (#546)

Define RAPIDJSON_HAS_CXX11_RVALUE_REFS directly in clang (#617)

Make GenericSchemaDocument constructor explicit (#674)

Optimize FindMember when use std::string (#690)

Source code(tar.gz)
Source code(zip)
v1.0.2(May 14, 2015)
Added

Add Value::XXXMember(...) overloads for std::string (#335)

Fixed

Include rapidjson.h for all internal/error headers.

Parsing some numbers incorrectly in full-precision mode (kFullPrecisionParseFlag) (#342)

Fix alignment of 64bit platforms (#328)

Fix MemoryPoolAllocator::Clear() to clear user-buffer (0691502573f1afd3341073dd24b12c3db20fbde4)

Changed

CMakeLists for include as a thirdparty in projects (#334, #337)

Change Document::ParseStream() to use stack allocator for Reader (ffbe38614732af8e0b3abdc8b50071f386a4a685)

Source code(tar.gz)
Source code(zip)
v1.0.1(Apr 25, 2015)
Added

Changelog following Keep a CHANGELOG suggestions.

Fixed

Parsing of some numbers (e.g. "1e-00011111111111") causing assertion (#314).

Visual C++ 32-bit compilation error in diyfp.h (#317).

Source code(tar.gz)
Source code(zip)
v1.0.0(Apr 22, 2015)
This is the final v1.0.0 release of RapidJSON.

After the v1.0-beta, a lot of efforts have been put to make RapidJSON 100% line-of-code covered by the unit tests.

Major changes since v1.0-beta

Fixed a bug in trimming long number sequence (4824f12efbf01af72b8cb6fc96fae7b097b73015)

Fix double quote in unicode escape (#288)

Fix negative zero roundtrip (double only) (#289)

Remove an invalid Document::ParseInsitu() API (e7f1c6dd08b522cfcf9aed58a333bd9a0c0ccbeb)

Remove dead branches and add more unit tests for coverage

Standardize behavior of memcpy() and malloc() (0c5c1538dcfc7f160e5a4aa208ddf092c787be5a, #305, 0e8bbe5e3ef375e7f052f556878be0bd79e9062d)

Add version macros (#311)

Source code(tar.gz)
Source code(zip)
v1.0-beta(Apr 8, 2015)
This is the first pre-release of RapidJSON at GitHub.

There have been a lot of improvements and bug fixes since v0.11. Some notable ones are listed as follows.

Features

RFC 7159 (#101)

Optional Iterative Parser (#76)

Deep-copy values (#20)

Error code and message (#27)

ASCII Encoding (#70)

kParseStopWhenDoneFlag (#83)

kParseFullPrecisionFlag (881c91d696f06b7f302af6d04ec14dd08db66ceb)

Add Key() to handler concept (#134)

C++11 compatibility and support (#128)

Optimizations

Optimized number-to-string and vice versa conversions (#137, #80)

Short-String Optimization (#131)

Local stream optimization by traits (#32)

Build

Migrating from Premake to CMAKE (#192)

Remove other JSON libraries for performance comparison (#180)

Travis & Appveyor Continuous Integration, with Valgrind verification (#24, #242)

Resolve all warning reports

Documentation

Redo all documentation

English, Simplified Chinese

Doxygen with Markdown

Gitbook

Notes

Note that the copyright owner is changed but the project is still distributed under MIT license.

Please help testing this release in your development platform. The API shall not be changed in v1.0. Only bug fixes are permitted.

It is planned to release the official v1.0 at 15 April 2015.
Source code(tar.gz)
Source code(zip)