This only happens after simplejson is installed so I'm logging the issue here.

The backtrace is a bit skimpy, but unfortunately this doesn't reproduce with python-dbg.

Neither it does reproduce if I compile a custom 2.7.9 Python. Everyone should be using that, but alas, Ubuntu's LTS release comes with 2.7.3, unlikely many people will have 2.7.9 :cry:

ionel@minibox:~$ pip --version
pip 6.0.6 from /usr/local/lib/python2.7/dist-packages (python 2.7)

ionel@minibox:~$ easy_install --version
setuptools 11.3.1

ionel@minibox:~$ python --version
Python 2.7.3

ionel@minibox:~$ lsb
lsblk        lsb_release

ionel@minibox:~$ lsb_release
No LSB modules are available.

ionel@minibox:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.5 LTS
Release:        12.04
Codename:       precise

ionel@minibox:~$ pip wheel --wheel-dir=. simplejson
Collecting simplejson
  Using cached simplejson-3.6.5.tar.gz
Building wheels for collected packages: simplejson
  Running setup.py bdist_wheel for simplejson
  Destination directory: /home/ionel
Successfully built simplejson

ionel@minibox:~$ rm -rf .ve
virtualenv .ve --system-site-packages
New python executable in .ve/bin/python
Installing setuptools, pip...done.

ionel@minibox:~$ .ve/bin/pip install -I simplejson-3.6.5-cp27-none-linux_x86_64.whl
Processing ./simplejson-3.6.5-cp27-none-linux_x86_64.whl
Installing collected packages: simplejson
Successfully installed simplejson-3.6.5

ionel@minibox:~$ .ve/bin/pip install -I simplejson-3.6.5-cp27-none-linux_x86_64.whl
Processing ./simplejson-3.6.5-cp27-none-linux_x86_64.whl
Installing collected packages: simplejson
Successfully installed simplejson-3.6.5
Segmentation fault (core dumped)

ionel@minibox:~$ LD_PRELOAD=/lib/x86_64-linux-gnu/libSegFault.so .ve/bin/pip install -I simplejson-3.6.5-cp27-none-linux_x86_64.whl
Processing ./simplejson-3.6.5-cp27-none-linux_x86_64.whl
Installing collected packages: simplejson

Successfully installed simplejson-3.6.5
*** Segmentation fault
Register dump:

 RAX: 0000000000000000   RBX: 00007f57897ee4e0   RCX: 0000000001ed4f00
 RDX: 0000000001ed4f18   RSI: 0000000000000000   RDI: 00007f57897ee4e0
 RBP: 0000000000555de0   R8 : 00000000024c2190   R9 : 0000000002486840
 R10: 0000000000000008   R11: 0000000000000206   R12: 000000000000000f
 R13: 00007f57897ee4e0   R14: 0000000000000000   R15: 0000000000000002
 RSP: 00007fffaf78abd0

 RIP: 0000000000555de8   EFLAGS: 00010206

 CS: 0033   FS: 0000   GS: 0000

 Trap: 0000000e   Error: 00000004   OldMask: 00000000   CR2: 000000a9

 FPUCW: 0000037f   FPUSW: 00000000   TAG: 00000000
 RIP: 00000000   RDP: 00000000

 ST(0) 0000 0000000000000000   ST(1) 0000 0000000000000000
 ST(2) 0000 0000000000000000   ST(3) 0000 0000000000000000
 ST(4) 0000 0000000000000000   ST(5) 0000 0000000000000000
 ST(6) 0000 0000000000000000   ST(7) 0000 0000000000000000
 mxcsr: 1fa0
 XMM0:  00000000000000000000000000000000 XMM1:  00000000000000000000000000000000
 XMM2:  00000000000000000000000000000000 XMM3:  00000000000000000000000000000000
 XMM4:  00000000000000000000000000000000 XMM5:  00000000000000000000000000000000
 XMM6:  00000000000000000000000000000000 XMM7:  00000000000000000000000000000000
 XMM8:  00000000000000000000000000000000 XMM9:  00000000000000000000000000000000
 XMM10: 00000000000000000000000000000000 XMM11: 00000000000000000000000000000000
 XMM12: 00000000000000000000000000000000 XMM13: 00000000000000000000000000000000
 XMM14: 00000000000000000000000000000000 XMM15: 00000000000000000000000000000000

Backtrace:
/home/ionel/.ve/bin/python[0x555de8]
/home/ionel/.ve/bin/python[0x554666]
/home/ionel/.ve/bin/python[0x539045]
/home/ionel/.ve/bin/python(Py_Finalize+0x1ec)[0x4f62fc]
/home/ionel/.ve/bin/python(Py_Exit+0x8)[0x4f6768]
/home/ionel/.ve/bin/python[0x4f689c]
/home/ionel/.ve/bin/python(PyRun_SimpleFileExFlags+0x3fe)[0x4c14be]
/home/ionel/.ve/bin/python(Py_Main+0x87f)[0x4c754f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f578afcc76d]
/home/ionel/.ve/bin/python[0x41ba41]

Memory map:

00400000-00672000 r-xp 00000000 08:01 6829994                            /home/ionel/.ve/bin/python
00871000-00872000 r--p 00271000 08:01 6829994                            /home/ionel/.ve/bin/python
00872000-008db000 rw-p 00272000 08:01 6829994                            /home/ionel/.ve/bin/python
008db000-008ed000 rw-p 00000000 00:00 0
01703000-02643000 rw-p 00000000 00:00 0                                  [heap]
7f578897b000-7f5788988000 r-xp 00000000 08:01 6043731                    /usr/lib/python2.7/lib-dynload/_elementtree.so
7f5788988000-7f5788b87000 ---p 0000d000 08:01 6043731                    /usr/lib/python2.7/lib-dynload/_elementtree.so
7f5788b87000-7f5788b88000 r--p 0000c000 08:01 6043731                    /usr/lib/python2.7/lib-dynload/_elementtree.so
7f5788b88000-7f5788b89000 rw-p 0000d000 08:01 6043731                    /usr/lib/python2.7/lib-dynload/_elementtree.so
7f5788b89000-7f5788b98000 r-xp 00000000 08:01 524293                     /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f5788b98000-7f5788d97000 ---p 0000f000 08:01 524293                     /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f5788d97000-7f5788d98000 r--p 0000e000 08:01 524293                     /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f5788d98000-7f5788d99000 rw-p 0000f000 08:01 524293                     /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f5788d99000-7f5788da0000 r-xp 00000000 08:01 6043736                    /usr/lib/python2.7/lib-dynload/bz2.so
7f5788da0000-7f5788f9f000 ---p 00007000 08:01 6043736                    /usr/lib/python2.7/lib-dynload/bz2.so
7f5788f9f000-7f5788fa0000 r--p 00006000 08:01 6043736                    /usr/lib/python2.7/lib-dynload/bz2.so
7f5788fa0000-7f5788fa2000 rw-p 00007000 08:01 6043736                    /usr/lib/python2.7/lib-dynload/bz2.so
7f5788fa2000-7f5788fa8000 r-xp 00000000 08:01 6043759                    /usr/lib/python2.7/lib-dynload/_csv.so
7f5788fa8000-7f57891a7000 ---p 00006000 08:01 6043759                    /usr/lib/python2.7/lib-dynload/_csv.so
7f57891a7000-7f57891a8000 r--p 00005000 08:01 6043759                    /usr/lib/python2.7/lib-dynload/_csv.so
7f57891a8000-7f57891aa000 rw-p 00006000 08:01 6043759                    /usr/lib/python2.7/lib-dynload/_csv.so
7f57891aa000-7f57891d1000 r-xp 00000000 08:01 524520                     /lib/x86_64-linux-gnu/libexpat.so.1.5.2
7f57891d1000-7f57893d1000 ---p 00027000 08:01 524520                     /lib/x86_64-linux-gnu/libexpat.so.1.5.2
7f57893d1000-7f57893d3000 r--p 00027000 08:01 524520                     /lib/x86_64-linux-gnu/libexpat.so.1.5.2
7f57893d3000-7f57893d4000 rw-p 00029000 08:01 524520                     /lib/x86_64-linux-gnu/libexpat.so.1.5.2
7f57893d4000-7f57893e2000 r-xp 00000000 08:01 6043761                    /usr/lib/python2.7/lib-dynload/pyexpat.so
7f57893e2000-7f57895e1000 ---p 0000e000 08:01 6043761                    /usr/lib/python2.7/lib-dynload/pyexpat.so
7f57895e1000-7f57895e2000 r--p 0000d000 08:01 6043761                    /usr/lib/python2.7/lib-dynload/pyexpat.so
7f57895e2000-7f57895e4000 rw-p 0000e000 08:01 6043761                    /usr/lib/python2.7/lib-dynload/pyexpat.so
7f57895e4000-7f57895ee000 r-xp 00000000 08:01 6830780                    /home/ionel/.ve/lib/python2.7/site-packages/simplejson/_speedups.so
7f57895ee000-7f57897ed000 ---p 0000a000 08:01 6830780                    /home/ionel/.ve/lib/python2.7/site-packages/simplejson/_speedups.so
7f57897ed000-7f57897ee000 r--p 00009000 08:01 6830780                    /home/ionel/.ve/lib/python2.7/site-packages/simplejson/_speedups.so
7f57897ee000-7f57897ef000 rw-p 0000a000 08:01 6830780                    /home/ionel/.ve/lib/python2.7/site-packages/simplejson/_speedups.so
7f57897ef000-7f57897f3000 r-xp 00000000 08:01 559770                     /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f57897f3000-7f57899f2000 ---p 00004000 08:01 559770                     /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f57899f2000-7f57899f3000 r--p 00003000 08:01 559770                     /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f57899f3000-7f57899f4000 rw-p 00004000 08:01 559770                     /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f57899f4000-7f57899ff000 r-xp 00000000 08:01 6043756                    /usr/lib/python2.7/lib-dynload/_json.so
7f57899ff000-7f5789bfe000 ---p 0000b000 08:01 6043756                    /usr/lib/python2.7/lib-dynload/_json.so
7f5789bfe000-7f5789bff000 r--p 0000a000 08:01 6043756                    /usr/lib/python2.7/lib-dynload/_json.so
7f5789bff000-7f5789c00000 rw-p 0000b000 08:01 6043756                    /usr/lib/python2.7/lib-dynload/_json.so
7f5789c00000-7f5789c04000 r-xp 00000000 08:01 6043742                    /usr/lib/python2.7/lib-dynload/termios.so
7f5789c04000-7f5789e03000 ---p 00004000 08:01 6043742                    /usr/lib/python2.7/lib-dynload/termios.so
7f5789e03000-7f5789e04000 r--p 00003000 08:01 6043742                    /usr/lib/python2.7/lib-dynload/termios.so
7f5789e04000-7f5789e06000 rw-p 00004000 08:01 6043742                    /usr/lib/python2.7/lib-dynload/termios.so
7f5789e06000-7f578a0cf000 r--p 00000000 08:01 6035410                    /usr/lib/locale/locale-archive
7f578a0cf000-7f578a0e2000 r-xp 00000000 08:01 6043735                    /usr/lib/python2.7/lib-dynload/parser.so
7f578a0e2000-7f578a2e1000 ---p 00013000 08:01 6043735                    /usr/lib/python2.7/lib-dynload/parser.so
7f578a2e1000-7f578a2e2000 r--p 00012000 08:01 6043735                    /usr/lib/python2.7/lib-dynload/parser.so
7f578a2e2000-7f578a2e3000 rw-p 00013000 08:01 6043735                    /usr/lib/python2.7/lib-dynload/parser.so
7f578a2e3000-7f578a2f7000 r-xp 00000000 08:01 6043764                    /usr/lib/python2.7/lib-dynload/datetime.so
7f578a2f7000-7f578a4f6000 ---p 00014000 08:01 6043764                    /usr/lib/python2.7/lib-dynload/datetime.so
7f578a4f6000-7f578a4f7000 r--p 00013000 08:01 6043764                    /usr/lib/python2.7/lib-dynload/datetime.so
7f578a4f7000-7f578a4fb000 rw-p 00014000 08:01 6043764                    /usr/lib/python2.7/lib-dynload/datetime.so
7f578a4fb000-7f578a51a000 r-xp 00000000 08:01 6043753                    /usr/lib/python2.7/lib-dynload/_io.so
7f578a51a000-7f578a719000 ---p 0001f000 08:01 6043753                    /usr/lib/python2.7/lib-dynload/_io.so
7f578a719000-7f578a71a000 r--p 0001e000 08:01 6043753                    /usr/lib/python2.7/lib-dynload/_io.so
7f578a71a000-7f578a723000 rw-p 0001f000 08:01 6043753                    /usr/lib/python2.7/lib-dynload/_io.so
7f578a723000-7f578a764000 rw-p 00000000 00:00 0
7f578a764000-7f578a785000 r-xp 00000000 08:01 6043762                    /usr/lib/python2.7/lib-dynload/_ctypes.so
7f578a785000-7f578a984000 ---p 00021000 08:01 6043762                    /usr/lib/python2.7/lib-dynload/_ctypes.so
7f578a984000-7f578a985000 r--p 00020000 08:01 6043762                    /usr/lib/python2.7/lib-dynload/_ctypes.so
7f578a985000-7f578a989000 rw-p 00021000 08:01 6043762                    /usr/lib/python2.7/lib-dynload/_ctypes.so
7f578a989000-7f578ab90000 rw-p 00000000 00:00 0
7f578ab90000-7f578ab93000 r-xp 00000000 08:01 6043740                    /usr/lib/python2.7/lib-dynload/_heapq.so
7f578ab93000-7f578ad92000 ---p 00003000 08:01 6043740                    /usr/lib/python2.7/lib-dynload/_heapq.so
7f578ad92000-7f578ad93000 r--p 00002000 08:01 6043740                    /usr/lib/python2.7/lib-dynload/_heapq.so
7f578ad93000-7f578ad95000 rw-p 00003000 08:01 6043740                    /usr/lib/python2.7/lib-dynload/_heapq.so
7f578ad95000-7f578adaa000 r-xp 00000000 08:01 524332                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f578adaa000-7f578afa9000 ---p 00015000 08:01 524332                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f578afa9000-7f578afaa000 r--p 00014000 08:01 524332                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f578afaa000-7f578afab000 rw-p 00015000 08:01 524332                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f578afab000-7f578b160000 r-xp 00000000 08:01 572001                     /lib/x86_64-linux-gnu/libc-2.15.so
7f578b160000-7f578b35f000 ---p 001b5000 08:01 572001                     /lib/x86_64-linux-gnu/libc-2.15.so
7f578b35f000-7f578b363000 r--p 001b4000 08:01 572001                     /lib/x86_64-linux-gnu/libc-2.15.so
7f578b363000-7f578b365000 rw-p 001b8000 08:01 572001                     /lib/x86_64-linux-gnu/libc-2.15.so
7f578b365000-7f578b36a000 rw-p 00000000 00:00 0
7f578b36a000-7f578b465000 r-xp 00000000 08:01 572018                     /lib/x86_64-linux-gnu/libm-2.15.so
7f578b465000-7f578b664000 ---p 000fb000 08:01 572018                     /lib/x86_64-linux-gnu/libm-2.15.so
7f578b664000-7f578b665000 r--p 000fa000 08:01 572018                     /lib/x86_64-linux-gnu/libm-2.15.so
7f578b665000-7f578b666000 rw-p 000fb000 08:01 572018                     /lib/x86_64-linux-gnu/libm-2.15.so
7f578b666000-7f578b67c000 r-xp 00000000 08:01 524517                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f578b67c000-7f578b87b000 ---p 00016000 08:01 524517                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f578b87b000-7f578b87c000 r--p 00015000 08:01 524517                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f578b87c000-7f578b87d000 rw-p 00016000 08:01 524517                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f578b87d000-7f578ba2f000 r-xp 00000000 08:01 527949                     /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f578ba2f000-7f578bc2e000 ---p 001b2000 08:01 527949                     /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f578bc2e000-7f578bc49000 r--p 001b1000 08:01 527949                     /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f578bc49000-7f578bc54000 rw-p 001cc000 08:01 527949                     /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f578bc54000-7f578bc58000 rw-p 00000000 00:00 0
7f578bc58000-7f578bcac000 r-xp 00000000 08:01 527944                     /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f578bcac000-7f578beac000 ---p 00054000 08:01 527944                     /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f578beac000-7f578beaf000 r--p 00054000 08:01 527944                     /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f578beaf000-7f578beb6000 rw-p 00057000 08:01 527944                     /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f578beb6000-7f578beb8000 r-xp 00000000 08:01 572024                     /lib/x86_64-linux-gnu/libutil-2.15.so
7f578beb8000-7f578c0b7000 ---p 00002000 08:01 572024                     /lib/x86_64-linux-gnu/libutil-2.15.so
7f578c0b7000-7f578c0b8000 r--p 00001000 08:01 572024                     /lib/x86_64-linux-gnu/libutil-2.15.so
7f578c0b8000-7f578c0b9000 rw-p 00002000 08:01 572024                     /lib/x86_64-linux-gnu/libutil-2.15.so
7f578c0b9000-7f578c0bb000 r-xp 00000000 08:01 572029                     /lib/x86_64-linux-gnu/libdl-2.15.so
7f578c0bb000-7f578c2bb000 ---p 00002000 08:01 572029                     /lib/x86_64-linux-gnu/libdl-2.15.so
7f578c2bb000-7f578c2bc000 r--p 00002000 08:01 572029                     /lib/x86_64-linux-gnu/libdl-2.15.so
7f578c2bc000-7f578c2bd000 rw-p 00003000 08:01 572029                     /lib/x86_64-linux-gnu/libdl-2.15.so
7f578c2bd000-7f578c2d5000 r-xp 00000000 08:01 571994                     /lib/x86_64-linux-gnu/libpthread-2.15.so
7f578c2d5000-7f578c4d4000 ---p 00018000 08:01 571994                     /lib/x86_64-linux-gnu/libpthread-2.15.so
7f578c4d4000-7f578c4d5000 r--p 00017000 08:01 571994                     /lib/x86_64-linux-gnu/libpthread-2.15.so
7f578c4d5000-7f578c4d6000 rw-p 00018000 08:01 571994                     /lib/x86_64-linux-gnu/libpthread-2.15.so
7f578c4d6000-7f578c4da000 rw-p 00000000 00:00 0
7f578c4da000-7f578c4de000 r-xp 00000000 08:01 572025                     /lib/x86_64-linux-gnu/libSegFault.so
7f578c4de000-7f578c6dd000 ---p 00004000 08:01 572025                     /lib/x86_64-linux-gnu/libSegFault.so
7f578c6dd000-7f578c6de000 r--p 00003000 08:01 572025                     /lib/x86_64-linux-gnu/libSegFault.so
7f578c6de000-7f578c6df000 rw-p 00004000 08:01 572025                     /lib/x86_64-linux-gnu/libSegFault.so
7f578c6df000-7f578c701000 r-xp 00000000 08:01 571993                     /lib/x86_64-linux-gnu/ld-2.15.so
7f578c734000-7f578c838000 rw-p 00000000 00:00 0
7f578c869000-7f578c8f1000 rw-p 00000000 00:00 0
7f578c8fd000-7f578c8fe000 rw-p 00000000 00:00 0
7f578c8fe000-7f578c8ff000 rwxp 00000000 00:00 0
7f578c8ff000-7f578c901000 rw-p 00000000 00:00 0
7f578c901000-7f578c902000 r--p 00022000 08:01 571993                     /lib/x86_64-linux-gnu/ld-2.15.so
7f578c902000-7f578c904000 rw-p 00023000 08:01 571993                     /lib/x86_64-linux-gnu/ld-2.15.so
7fffaf757000-7fffaf78c000 rw-p 00000000 00:00 0                          [stack]
7fffaf7fe000-7fffaf800000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Segmentation fault (core dumped)

Hi,

Sorry for taking a while to report this. It took me some time to find the JSON (in about 2.5TB of data) and then I realised I should probably trim it down to the relevant section.

I am unable to parse the following JSON with simplejson 3.1.0 under Linux. This JSON is taken from a result set from ElasticSearch and obviously trimmed down to be as small as possible.

{"a": "\ud8e9"}

I done some testing on a fresh vagrant Ubuntu Precise 64bit virtual machine but previously we have been seeing the error with lucid on EC2.

The error get is;

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/__init__.py", line 398, in load
    use_decimal=use_decimal, **kw)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/__init__.py", line 454, in loads
    return _default_decoder.decode(s)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/decoder.py", line 393, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Unpaired high surrogate: line 1 column 8 (char 7)

If I disable the speedups with simplejson._toggle_speedups(False) I get a (possibly) slightly more helpful error.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/__init__.py", line 398, in load
    use_decimal=use_decimal, **kw)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/__init__.py", line 454, in loads
    return _default_decoder.decode(s)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/decoder.py", line 393, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/scanner.py", line 119, in scan_once
    return _scan_once(string, idx)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/scanner.py", line 90, in _scan_once
    _scan_once, object_hook, object_pairs_hook, memo)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/decoder.py", line 198, in JSONObject
    value, end = scan_once(s, end)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/scanner.py", line 87, in _scan_once
    return parse_string(string, idx + 1, encoding, strict)
  File "/home/vagrant/.virtualenvs/a6e8a75c0b63d8f2/local/lib/python2.7/site-packages/simplejson/decoder.py", line 118, in py_scanstring
    raise JSONDecodeError(msg, s, end)
simplejson.scanner.JSONDecodeError: Unpaired high surrogate: line 1 column 8 (char 7)

However, I'm not totally sure where the blame lies here. I've done a bit of research and this is what I've found.

• I mentioned before I only have this problem under Linux. Locally on my mac it works fine. I've not been able to test Windows. I was pointed to this issue by @bigkevmcd: http://bugs.python.org/issue11489 which seems relevant but not conclusive.
• I have tested this file with a few other languages for comparison - works fine in Java and Ruby. JavaScript seems to mostly work, although I noticed some display issues in the Chrome console.
• I have discovered some other JSON documents cause the same error however they do work when I turn off the speedups. I've not been logging these as I'm turning off the speedups and trying again and then only logging those that still fail. I'll do another run (probably not until Monday now) and add logging of these so I can provide them.
• In my tweet I was confused why it worked, failed and then worked after a pip re-install. This was with a different document and because simplejson was originally installed on the server with a package. Then when I pip installed it failed to install speedups - thus I "disabled" them without realising.

So, yeah, that's what I've found. I'm not sure how helpful it is. It does seem that the JSON that work without c speedups (but not with) only are causing a bug but it also seems to me that unpaired surrogate issues should either always work or always not work...

Got a strange bug when upgrading 3.10.0 to 3.11.0 via pip.

A test started failing that used a custom sort function via item_sort_key. It seems that setting this kwarg has the effect of enabling sorting using the default itemgetter in 3.11.0.

This occurs when using the C encoder implementation.

Python is 2.7.11 on Ubuntu 16.04.

Using this code:

import simplejson

def sortit(x):
  return x[1]

a = {
  "c": "3",
  "b": "1",
  "a": "2"
}

print simplejson.dumps(a, item_sort_key=sortit)

On 3.10.0:

$ python test.py
{"b": "1", "c": "2", "a": "3"}

And on 3.11.0:

$ python test.py
{"a": "2", "b": "1", "c": "3"}

I was wondering if you'd be open to including a helper hook for encoding arbitrary JSON objects.

My use case is that I have classes which subclass dict, but need to have mutated keys before serialization to JSON. Because they subclass dict, they don't get picked up by the default() method.

I'd like to see something where an optional hook can be called to return a more serializable representation (which may need recursion to serialize nested objects). For example:

class Example(object):
    def for_json(self):
        return {'example': 1}

This would be very similar to how the _asdict hook is called currently, but would use encode_listencode_obj (and _iterencode in Python), instead of the dict varieties of those functions.

Any thoughts on this? Would you be open to including this feature into the simplejson library?

I'm kind-of jumping the gun on this, because I haven't forked and implemented this, but I'm about to, so I wanted to ask early enough that I might not need to create a forked version on PyPI. :)

I'm currently working on an issue where I'm writing a library that will interface with other moving parts that use simplejson internally. However, I need to be able to pass them a custom object and have that be serialized. Unfortunately, since the json encoding is done within these projects and isn't directly called by my code, I can't pass it a custom encoder.

Pyramid (one of the moving parts I'm using) has a useful function that will use the json attribute to encode a custom object, but the other one I'm using does not have that. While searching around, I found this patch from the code.google.com repo.

It would be awesome if this behavior could be included in simplejson by default. The current behavior of using the output from repr(o) makes it very difficult to have my object serializable by simplejson in an agnostic manner, since I'm not working with the library directly.

Using json.dumps() gets slightly complicated when you try to serialize more complex objects, like datetime.datetime or uuid.UUID objects, as simplejson and the built-in json module do not know how to serialize these objects:

import simplejson as json
import uuid
from datetime import datetime

json.dumps({'now': datetime.now()})

# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File ".../simplejson/__init__.py", line 382, in dumps
#     return _default_encoder.encode(obj)
#   File ".../simplejson/encoder.py", line 385, in encode
#     chunks = list(chunks)
#   File ".../simplejson/encoder.py", line 770, in _iterencode
#     for chunk in _iterencode_dict(o, _current_indent_level):
#   File ".../simplejson/encoder.py", line 727, in _iterencode_dict
#     for chunk in chunks:
#   File ".../simplejson/encoder.py", line 790, in _iterencode
#     o = _default(o)
#   File ".../simplejson/encoder.py", line 88, in _method
#     return method.__get__(instance, owner)(*args, **kwargs)
#   File ".../simplejson/encoder.py", line 360, in default
#     o.__class__.__name__)
# TypeError: Object of type datetime is not JSON serializable

json.dumps({'uuid': uuid.uuid4()})
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File ".../simplejson/__init__.py", line 382, in dumps
#     return _default_encoder.encode(obj)
#   File ".../simplejson/encoder.py", line 385, in encode
#     chunks = list(chunks)
#   File ".../simplejson/encoder.py", line 770, in _iterencode
#     for chunk in _iterencode_dict(o, _current_indent_level):
#   File ".../simplejson/encoder.py", line 727, in _iterencode_dict
#     for chunk in chunks:
#   File ".../simplejson/encoder.py", line 790, in _iterencode
#     o = _default(o)
#   File ".../simplejson/encoder.py", line 88, in _method
#     return method.__get__(instance, owner)(*args, **kwargs)
#   File ".../simplejson/encoder.py", line 360, in default
#     o.__class__.__name__)
# TypeError: Object of type UUID is not JSON serializable

There are two workarounds for this:

1. Subclass simplejson.encoder.JSONEncoder and override its default method, then explicitly call JSONEncoder.encode() everywhere (this is exactly what Django does).

2. Define a function that accepts a single argument and serializes it appropriately, then call json.dumps() with the default argument everywhere:

   def my_encoder(obj):
       if isinstance(obj, datetime):
           return obj.isoformat()
       elif isinstance(obj, uuid.UUID):
           return str(obj)
       else:
           raise TypeError("Cannot serialize object of type '%s'" % (type(obj)))
   
   json.dumps({'now': datetime.now(), 'uuid': uuid.uuid4()}, default=my_encoder)
   
   # '{"now": "2018-01-21T12:34:42.993445", "uuid": "5f9a8bbe-d3b2-4a32-8bc0-83ac87cb195c"}'
   

Both of these workarounds require users to modify/override the custom serializer function rather than extending it, and then to use the modified code everywhere instead of directly using the built-in json module or simplejson. This generally means frameworks like Django force all users to use their own serializers with their own serializer functions. It's also rather confusing for new Django users, who ask questions like this on Stack Overflow "How do I encode a UUID to make it JSON serializable?".

Furthermore, for a non-trivial project, the custom default function can become unwieldy and begins to look exactly like the usecase for singledispatch.

This PR simply uses a modified singledispatch (called singledispatchmethod) decorator to extend the default function for specific object types. This makes the simplejson.encoder.JSONEncoder.default method better adhere to the Open/Closed Principle - open to extension but closed to modification (or override). This decorator should also be completely backwards compatible - subclasses of JSONEncoder can still override it, and calls to simplejson.dumps() with a default= argument will still use the given custom encoder function.

This change allows frameworks to define and register those functions anywhere that is automatically imported (like, for instance, django/__init__.py):

import simplejson as json
import uuid
from datetime import datetime

@json.JSONEncoder.default.register(datetime)
def jsonify_datetime(encoder, dt):
    return dt.isoformat()

@json.JSONEncoder.default.register(uuid.UUID)
def jsonify_uuid(encoder, uuid_):
    return str(uuid_)

That's it - now every user of Django can simply call simplejson.dumps() directly and it will be able to properly serialize datetimes and UUIDs:

import simplejson as json
import uuid
from datetime import datetime


json.dumps({'now': datetime.now(), 'uuid': uuid.uuid4()})
# '{"now": "2018-01-22T03:00:49.847345", "uuid": "629a7f1e-3eb6-4bf3-a6cf-6717910e4a92"}'

Now, I realize that Django is a huge framework, and it has DRF, which implements its own framework for registering serializers. But I think this PR has the potential to greatly simplify framework code.

The singledispatch decorator does not work on instance methods, but there is a PR to fix this. I have included the two implementations (one naïve, one that is more complex but handles corner cases well) in this PR FOR REFERENCE ONLY. I will be removing them and replacing them the proper imports from functools once PR python/cpython#4987 or something like it is merged.

This PR should also fix or obviate the needs to fix the following issues and PRs:

• #149
• #158
• #159
• #164

I'm happy to respond to any criticisms or answer any questions. I would eventually like to push this API into Python's built-in json module, but it's written in C and I wanted to get feedback on this Python implementation before I do that.

Thanks!

In Python 3.2 and 3.1 you get a syntax error with something like this:

x = u"Hello world"

Only you have 3 lines incompatible with Python 3.1 and 3.2:

https://github.com/simplejson/simplejson/blob/master/simplejson/encoder.py#L21 https://github.com/simplejson/simplejson/blob/master/simplejson/encoder.py#L256 https://github.com/simplejson/simplejson/blob/master/simplejson/encoder.py#L347

Here is a patch for running on Python 2.4. I know the docs state that it runs on Python 2.5 and later, but this is a very small change that is required for running Sphinx on RHEL5. Hopefully it's useful and will be merged. I'm new to git, so thought I would keep it simple by attaching a diff.

Python 2.4 was raising a SyntaxError and the following change was made to fix this. I'm running simplejson 2.5 from PyPI along with Sphinx 1.1.3. I ran the unit tests with the change and they all pass.

Cheers, Chris LeBlanc

diff --git a/simplejson/encoder.py b/simplejson/encoder.py index 05dd05c..5710e67 100644 --- a/simplejson/encoder.py +++ b/simplejson/encoder.py @@ -394,10 +394,10 @@ def _make_iterencode(markers, _default, _encoder, _indent, _floatstr, elif value is False: yield buf + 'false' elif isinstance(value, (int, long)):

•            yield ((buf + str(value))
  

•                   if (not _bigint_as_string or
  

•                       (-1 << 53) < value < (1 << 53))
  

•                       else (buf + '"' + str(value) + '"'))
  

•            if not _bigint_as_string or (-1 << 53) < value < (1 << 53):
  

•                yield buf + str(value)
  

•            else:
  

•                yield buf + '"' + str(value) + '"'
       elif isinstance(value, float):
           yield buf + _floatstr(value)
       elif _use_decimal and isinstance(value, Decimal):
  

  @@ -487,10 +487,10 @@ def _make_iterencode(markers, _default, _encoder, _indent, _floatstr, elif value is False: yield 'false' elif isinstance(value, (int, long)):

•            yield (str(value)
  

•                   if (not _bigint_as_string or
  

•                       (-1 << 53) < value < (1 << 53))
  

•                       else ('"' + str(value) + '"'))
  

•            if not _bigint_as_string or (-1 << 53) < value < (1 << 53):
  

•                yield str(value)
  

•            else:
  

•                yield '"' + str(value) + '"'
       elif isinstance(value, float):
           yield _floatstr(value)
       elif _use_decimal and isinstance(value, Decimal):
  

  @@ -528,10 +528,10 @@ def _make_iterencode(markers, _default, _encoder, _indent, _floatstr, elif o is False: yield 'false' elif isinstance(o, (int, long)):

•        yield (str(o)
  

•               if (not _bigint_as_string or
  

•                   (-1 << 53) < o < (1 << 53))
  

•                   else ('"' + str(o) + '"'))
  

•        if not _bigint_as_string or (-1 << 53) < o < (1 << 53):
  

•            yield str(o)
  

•        else:
  

•            yield '"' + str(o) + '"'
   elif isinstance(o, float):
       yield _floatstr(o)
   elif isinstance(o, list):
  

This was causing failures in software which passes objects which are neither of type str, int, or long, but rather a custom numerical class, as the value of the indent parameter. Since this custom numerical class understands multiplication by strings, it should be accepted by simplejson's encoder and treated as a numerical argument.

... is what it says in the commit message. Of course, I make no such bold claim about what "should" happen, but am just submitting this pull request to see if you agree :) This is in some sense a regression because everything worked fine before string arguments for the indent parameter became supported, and the docstring suggests that backwards compatibility was aimed for.

pip install simplejson==3.16.1 fails on CentOS 6.8.

bash-4.1# uname -or
2.6.32-696.6.3.el6.x86_64 GNU/Linux
bash-4.1# rpm --query centos-release
centos-release-6-8.el6.centos.12.3.x86_64

simplejson-3.16.1-cp27-cp27m-manylinux1_x86_64.whl is not a supported wheel on this platform.
Exception information:
Traceback (most recent call last):
  File "/opt/venv/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/opt/venv/lib/python2.7/site-packages/pip/commands/install.py", line 312, in run
    wheel_cache
  File "/opt/venv/lib/python2.7/site-packages/pip/basecommand.py", line 297, in populate_requirement_set
    requirement_set.add_requirement(req)
  File "/opt/venv/lib/python2.7/site-packages/pip/req/req_set.py", line 245, in add_requirement
    wheel.filename
InstallationError: simplejson-3.16.1-cp27-cp27m-manylinux1_x86_64.whl is not a supported wheel on this platform.
/

Seems probably related to https://github.com/simplejson/simplejson/pull/231

There are a lot of "hooks" built into the encoding/decoding process for the values of the dictionary you are converting to/from JSON.

There are none for the keys of said dictionary. There are numerous ways to handle custom objects' encode/decode behavior: custom Encoders, default=func, for_json method.

However, these are not applied to the keys of the dictionary.

Reviewing the source, I see a rather similar process being used on the keys that is used to process the values:

https://github.com/simplejson/simplejson/blob/master/simplejson/encoder.py#L490

    def _stringify_key(key):
        if isinstance(key, string_types): # pragma: no cover
            pass
        elif isinstance(key, binary_type):
            key = key.decode(_encoding)
        elif isinstance(key, float):
            key = _floatstr(key)
        elif key is True:
            key = 'true'
        elif key is False:
            key = 'false'
        elif key is None:
            key = 'null'
        elif isinstance(key, integer_types):
            key = str(key)
        elif _use_decimal and isinstance(key, Decimal):
            key = str(key)
        elif _skipkeys:
            key = None
        else:
            raise TypeError("key " + repr(key) + " is not a string")
        return key

A custom "hook" is built in here for the Decimal class, which is great, but why do we treat this class as special? Is there a reason we could not expose this conversion process to extension, as we do in default=?

Unfortunately, this function is buried inside another, much larger function, and there's no trivial way to swap it out, so I'm faced with some sloppy options if I want to encode a dictionary like:

{MyObject('Something'): 10}

Upon a very light review, it seems this could be implemented in a similar manner as the default= kwarg argument. I understand there may be performance concerns, but I suppose I don't see them as any different than those opened up by the default= kwarg.

Currently int_as_string_bitcount supports only up to 63bits. Please add support for any bitlength or at least 127(128) bits. Snowflake supports 128bit integers, currently any bigints that I load have to be converted to a string if they are less than 64bits.

ideally all of the following should be stored as a number given the following code:

test = {
    "max_int": 99999999999999999999999999999999999999,
    "63 bits": 999999999999999999,
    "53 bits": 999999999999999,
    "overflow": 999999999999999999999999999999999999991,
}
result = simplejson.dumps(test, int_as_string_bitcount=128)

print(result)

If one needs to change the representation of floats then they have to extend the encoder and override iterencode by copying over most of the code and just:

1. replacing the _repr=FLOAT_REPR to something that they need.
2. removing the C encoder option

OS: centos 7.9
python: 3.8.2

Package    Version
---------- -------
pip        21.0
setuptools 52.0.0
wheel      0.36.2

When i run command python setup.py bdist_wheel,i get the result as follows:

After modify setup.py one line

#packages=['simplejson', 'simplejson.tests']
packages=['simplejson']

i get the result as follows：

Size compare: delete the simplejson/tests directory could reduce 28kb.

So when running simplejson,whether it is necessary to package the simplejson/tests directory in the wheel package?

Hi,

Given the popularity of simplejson I was thinking that it would be nice to set up continuous fuzzing of simplejson, by way of OSS-Fuzz. In this PR: https://github.com/google/oss-fuzz/pull/4905 I have done exactly that, namely created the necessary logic from an OSS-Fuzz perspective to integrate simplejson. This includes developing initial fuzzers as well as integrating into OSS-Fuzz, however, it is preferable to have the fuzzers upstream so I included it in this PR - if you are happy with having the fuzzers here then I will remove them from the OSS-Fuzz repository.

Essentially, OSS-Fuzz is a free service run by Google that performs continuous fuzzing of important open source projects. Python support was recently provided and it can also fuzz native extensions, i.e. for memory corruption errors. The only expectation of integrating into OSS-Fuzz is that bugs will be fixed. This is not a "hard" requirement in that no one enforces this and the main point is if bugs are not fixed then it is a waste of resources to run the fuzzers, which we would like to avoid.

If you would like to integrate, the only thing I need is as list of email(s) that will get access to the data produced by OSS-Fuzz, such as bug reports, coverage reports and more stats. Notice the emails affiliated with the project will be public in the OSS-Fuzz repo, as they will be part of a configuration file.

When passing the attached input to the following script, I get an unexpected RecursionError exception, while using the latest git version of simplejson.

import sys
import simplejson

with open(sys.argv[1], 'rb') as f:
    data = f.read()
try:
    simplejson.dumps(simplejson.loads(data))
except simplejson.errors.JSONDecodeError:
    pass
except UnicodeDecodeError:
    pass

$ python simplejson_repro.py test.txt
Traceback (most recent call last):
  File "simplejson_repro.py", line 7, in <module>
    simplejson.dumps(simplejson.loads(data))
  File "/home/user/ven/lib/python3.8/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/home/user/ven/lib/python3.8/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/home/user/ven/lib/python3.8/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
RecursionError: maximum recursion depth exceeded while decoding a JSON object from a unicode string

test.txt