numberparsingcheck crashes about simdjson HOT 9 CLOSED

Thanks for the report. It seems to crash on "free".

from simdjson.

Any chance you could build with sanitizers...?

make clean
make SANITIZE=1

from simdjson.

On Fedora 29 (make SANITIZE=1 -j8 test)

...
File                      fail39_EXCLUDE.json  is valid      --- integers:          2 floats:          0 invalid:          0 total numbers:          2 
include/simdjson/numberparsing.h:393:9: runtime error: signed integer overflow: 9223372036854775800 + 8 cannot be represented in type 'long int'
Warning: foundInvalidNumber 9223372036854775808
 whereas strtod parses it to 9223372036854775808.000000,  while parsing jsonchecker/fail40_s64boverflow.json 
File                 fail40_s64boverflow.json  is not valid  --- integers:          0 floats:          0 invalid:          1 total numbers:          1 
include/simdjson/numberparsing.h:393:14: runtime error: signed integer overflow: 1844674407370955161 * 10 cannot be represented in type 'long int'
Warning: foundInvalidNumber 18446744073709551616
 whereas strtod parses it to 18446744073709551616.000000,  while parsing jsonchecker/fail41_toolarge.json 
...
File                              pass01.json  is valid      --- integers:         22 floats:         10 invalid:          0 total numbers:         32 
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: 3141592653589793238 * 10 cannot be represented in type 'long int'
File                              pass04.json  is valid      --- integers:          0 floats:          2 invalid:          0 total numbers:          2 
include/simdjson/numberparsing.h:406:13: runtime error: signed integer overflow: -4611686018427387904 * 10 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: -9223372036854775808 * 10 cannot be represented in type 'long int'
File                              pass05.json  is valid      --- integers:          0 floats:          1 invalid:          0 total numbers:          1 
...
=================================================================
==12572==ERROR: AddressSanitizer: unknown-crash on address 0x6060000019cd at pc 0x00000041e304 bp 0x7ffc7c726270 sp 0x7ffc7c726260
WRITE of size 32 at 0x6060000019cd thread T0
    #0 0x41e303 in _mm256_storeu_si256(long long __vector(4)*, long long __vector(4)) /usr/lib/gcc/x86_64-redhat-linux/8/include/avxintrin.h:927
    #1 0x41e303 in parse_string(unsigned char const*, unsigned long, ParsedJson&, unsigned int, unsigned int) include/simdjson/stringparsing.h:95
    #2 0x41e303 in unified_machine(unsigned char const*, unsigned long, ParsedJson&) src/stage2_build_tape.cpp:352
    #3 0x430344 in json_parse(unsigned char const*, unsigned long, ParsedJson&, bool) src/jsonparser.cpp:44
    #4 0x42d9fa in json_parse(char const*, unsigned long, ParsedJson&, bool) (/home/dgunchev/sources/simdjson/numberparsingcheck+0x42d9fa)
    #5 0x42dae9 in json_parse(std::basic_string_view<char, std::char_traits<char> > const&, ParsedJson&, bool) (/home/dgunchev/sources/simdjson/numberparsingcheck+0x42dae9)
    #6 0x42a00d in validate(char const*) tests/numberparsingcheck.cpp:149
    #7 0x42a866 in main tests/numberparsingcheck.cpp:186
    #8 0x7fb4e55fe412 in __libc_start_main (/lib64/libc.so.6+0x24412)
    #9 0x40360d in _start (/home/dgunchev/sources/simdjson/numberparsingcheck+0x40360d)

0x6060000019e0 is located 0 bytes to the right of 64-byte region [0x6060000019a0,0x6060000019e0)
allocated by thread T0 here:
    #0 0x7fb4e66eb650 in operator new[](unsigned long) (/lib64/libasan.so.5+0xf1650)
    #1 0x42c0a7 in ParsedJson::allocateCapacity(unsigned long, unsigned long) include/simdjson/parsedjson.h:53
    #2 0x429e87 in validate(char const*) tests/numberparsingcheck.cpp:140
    #3 0x42a866 in main tests/numberparsingcheck.cpp:186
    #4 0x7fb4e55fe412 in __libc_start_main (/lib64/libc.so.6+0x24412)

SUMMARY: AddressSanitizer: unknown-crash /usr/lib/gcc/x86_64-redhat-linux/8/include/avxintrin.h:927 in _mm256_storeu_si256(long long __vector(4)*, long long __vector(4))
Shadow bytes around the buggy address:
  0x0c0c7fff82e0: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
  0x0c0c7fff82f0: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c7fff8300: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x0c0c7fff8310: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
  0x0c0c7fff8320: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
=>0x0c0c7fff8330: fa fa fa fa 00 00 00 00 00[00]00 00 fa fa fa fa
  0x0c0c7fff8340: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff8350: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff8360: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff8370: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff8380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==12572==ABORTING
make: *** [Makefile:60: test] Error 1

Similar problem with clang 7.0.1

==12782==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x606000001a4c at pc 0x000000549b37 bp 0x7ffc69110e50 sp 0x7ffc69110e48
WRITE of size 32 at 0x606000001a4c thread T0
    #0 0x549b36  (/home/dgunchev/sources/simdjson/numberparsingcheck+0x549b36)
    #1 0x56ea2d  (/home/dgunchev/sources/simdjson/numberparsingcheck+0x56ea2d)
    #2 0x56bb37  (/home/dgunchev/sources/simdjson/numberparsingcheck+0x56bb37)
    #3 0x5694ec  (/home/dgunchev/sources/simdjson/numberparsingcheck+0x5694ec)
    #4 0x55d093  (/home/dgunchev/sources/simdjson/numberparsingcheck+0x55d093)
    #5 0x55dada  (/home/dgunchev/sources/simdjson/numberparsingcheck+0x55dada)
    #6 0x7fe044cff412  (/lib64/libc.so.6+0x24412)
    #7 0x41c6dd  (/home/dgunchev/sources/simdjson/numberparsingcheck+0x41c6dd)

0x606000001a4c is located 12 bytes to the right of 64-byte region [0x606000001a00,0x606000001a40)
allocated by thread T0 here:
    #0 0x529b80  (/home/dgunchev/sources/simdjson/numberparsingcheck+0x529b80)
    #1 0x568a17  (/home/dgunchev/sources/simdjson/numberparsingcheck+0x568a17)
    #2 0x55ce83  (/home/dgunchev/sources/simdjson/numberparsingcheck+0x55ce83)
    #3 0x55dada  (/home/dgunchev/sources/simdjson/numberparsingcheck+0x55dada)
    #4 0x7fe044cff412  (/lib64/libc.so.6+0x24412)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/home/dgunchev/sources/simdjson/numberparsingcheck+0x549b36) 
Shadow bytes around the buggy address:
  0x0c0c7fff82f0: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c7fff8300: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x0c0c7fff8310: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
  0x0c0c7fff8320: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c7fff8330: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
=>0x0c0c7fff8340: 00 00 00 00 00 00 00 00 fa[fa]fa fa fa fa fa fa
  0x0c0c7fff8350: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff8360: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff8370: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff8380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff8390: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==12782==ABORTING
make: *** [Makefile:60: test] Error 1

from simdjson.

[szydell@laPtak simdjson]$ make SANITIZE=1
g++ -std=c++17 -march=native -Wall -Wextra -Wshadow -Iinclude -Ibenchmark/linux -g3 -O0 -fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined -o parse src/jsonioutil.cpp src/jsonparser.cpp src/stage1_find_marks.cpp src/stage2_build_tape.cpp benchmark/parse.cpp
/usr/bin/ld: cannot find /usr/lib64/libasan.so.5.0.0 inside /
collect2: error: ld returned 1 exit status
make: *** [Makefile:93: parse] Error 1

from simdjson.

@szydell

Your particular error message is not related to simdjson: the -fsanitize=address should work on recent compilers... you appear to have a system configuration error.

from simdjson.

Sorry. 2 libs missing.

[szydell@laPtak simdjson]$ make SANITIZE=1 -j12
g++ -std=c++17 -march=native -Wall -Wextra -Wshadow -Iinclude -Ibenchmark/linux -g3 -O0 -fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined -o parse src/jsonioutil.cpp src/jsonparser.cpp src/stage1_find_marks.cpp src/stage2_build_tape.cpp benchmark/parse.cpp
g++ -std=c++17 -march=native -Wall -Wextra -Wshadow -Iinclude -Ibenchmark/linux -g3 -O0 -fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined -o minify src/jsonminifier.cpp src/jsonioutil.cpp src/jsonparser.cpp src/stage1_find_marks.cpp src/stage2_build_tape.cpp tools/minify.cpp -I.
g++ -std=c++17 -march=native -Wall -Wextra -Wshadow -Iinclude -Ibenchmark/linux -g3 -O0 -fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined -o json2json tools/json2json.cpp src/jsonioutil.cpp src/jsonparser.cpp src/stage1_find_marks.cpp src/stage2_build_tape.cpp -I.
g++ -std=c++17 -march=native -Wall -Wextra -Wshadow -Iinclude -Ibenchmark/linux -g3 -O0 -fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined -o jsonstats tools/jsonstats.cpp src/jsonioutil.cpp src/jsonparser.cpp src/stage1_find_marks.cpp src/stage2_build_tape.cpp -I.
g++ -std=c++17 -march=native -Wall -Wextra -Wshadow -Iinclude -Ibenchmark/linux -g3 -O0 -fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined -o statisticalmodel src/jsonioutil.cpp src/jsonparser.cpp src/stage1_find_marks.cpp src/stage2_build_tape.cpp benchmark/statisticalmodel.cpp
[szydell@laPtak simdjson]$ make SANITIZE=1 -j12 test
g++ -std=c++17 -march=native -Wall -Wextra -Wshadow -Iinclude -Ibenchmark/linux -g3 -O0 -fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined -o jsoncheck src/jsonioutil.cpp src/jsonparser.cpp src/stage1_find_marks.cpp src/stage2_build_tape.cpp tests/jsoncheck.cpp -I.
g++ -std=c++17 -march=native -Wall -Wextra -Wshadow -Iinclude -Ibenchmark/linux -g3 -O0 -fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined -o numberparsingcheck tests/numberparsingcheck.cpp src/jsonioutil.cpp src/jsonparser.cpp src/stage1_find_marks.cpp -I. -DJSON_TEST_NUMBERS
g++ -std=c++17 -march=native -Wall -Wextra -Wshadow -Iinclude -Ibenchmark/linux -g3 -O0 -fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined -o stringparsingcheck tests/stringparsingcheck.cpp src/jsonioutil.cpp src/jsonparser.cpp src/stage1_find_marks.cpp -I. -DJSON_TEST_STRINGS
./numberparsingcheck
Usage: ./numberparsingcheck
We are going to assume you mean to use the 'jsonchecker' and 'jsonexamples' directories.
File fail11.json is not valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File fail13.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail14.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail29.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail30.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail31.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail37.json is not valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File fail38.json is not valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File fail39_EXCLUDE.json is valid --- integers: 2 floats: 0 invalid: 0 total numbers: 2
include/simdjson/numberparsing.h:393:9: runtime error: signed integer overflow: 9223372036854775800 + 8 cannot be represented in type 'long int'
Warning: foundInvalidNumber 9223372036854775808
whereas strtod parses it to 9223372036854775808.000000, while parsing jsonchecker/fail40_s64boverflow.json
File fail40_s64boverflow.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
include/simdjson/numberparsing.h:393:14: runtime error: signed integer overflow: 1844674407370955161 * 10 cannot be represented in type 'long int'
Warning: foundInvalidNumber 18446744073709551616
whereas strtod parses it to 18446744073709551616.000000, while parsing jsonchecker/fail41_toolarge.json
File fail41_toolarge.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail43.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber -2.]
��������������������������� whereas strtod parses it to -2.000000, while parsing jsonchecker/fail44.json
File fail44.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber 0.e1]
�������������������������� whereas strtod parses it to 0.000000, while parsing jsonchecker/fail45.json
File fail45.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber 2.e+3]
������������������������� whereas strtod parses it to 2000.000000, while parsing jsonchecker/fail46.json
File fail46.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber 2.e-3]
������������������������� whereas strtod parses it to 0.002000, while parsing jsonchecker/fail47.json
File fail47.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber 2.e3]
�������������������������� whereas strtod parses it to 2000.000000, while parsing jsonchecker/fail48.json
File fail48.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber -.123]
������������������������� whereas strtod parses it to -0.123000, while parsing jsonchecker/fail49.json
File fail49.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber 1.]
���������������������������� whereas strtod parses it to 1.000000, while parsing jsonchecker/fail50.json
File fail50.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail55.json is not valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File pass01.json is valid --- integers: 22 floats: 10 invalid: 0 total numbers: 32
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: 3141592653589793238 * 10 cannot be represented in type 'long int'
File pass04.json is valid --- integers: 0 floats: 2 invalid: 0 total numbers: 2
include/simdjson/numberparsing.h:406:13: runtime error: signed integer overflow: -4611686018427387904 * 10 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: -9223372036854775808 * 10 cannot be represented in type 'long int'
File pass05.json is valid --- integers: 0 floats: 1 invalid: 0 total numbers: 1
File pass08.json is valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File pass11.json is valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File pass12.json is valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File pass13.json is valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1

12397==ERROR: AddressSanitizer: unknown-crash on address 0x6060000019cd at pc 0x00000041e304 bp 0x7ffeb4aed7f0 sp 0x7ffeb4aed7e0
WRITE of size 32 at 0x6060000019cd thread T0
#0 0x41e303 in _mm256_storeu_si256(long long __vector(4), long long __vector(4)) /usr/lib/gcc/x86_64-redhat-linux/8/include/avxintrin.h:927
#1 0x41e303 in parse_string(unsigned char const, unsigned long, ParsedJson&, unsigned int, unsigned int) include/simdjson/stringparsing.h:95
#2 0x41e303 in unified_machine(unsigned char const*, unsigned long, ParsedJson&) src/stage2_build_tape.cpp:352
#3 0x430344 in json_parse(unsigned char const*, unsigned long, ParsedJson&, bool) src/jsonparser.cpp:44
#4 0x42d9fa in json_parse(char const*, unsigned long, ParsedJson&, bool) (/home/szydell/development/simdjson/numberparsingcheck+0x42d9fa)
#5 0x42dae9 in json_parse(std::basic_string_view<char, std::char_traits > const&, ParsedJson&, bool) (/home/szydell/development/simdjson/numberparsingcheck+0x42dae9)
#6 0x42a00d in validate(char const*) tests/numberparsingcheck.cpp:149
#7 0x42a866 in main tests/numberparsingcheck.cpp:186
#8 0x7fdcd080a412 in __libc_start_main (/lib64/libc.so.6+0x24412)
#9 0x40360d in _start (/home/szydell/development/simdjson/numberparsingcheck+0x40360d)

0x6060000019e0 is located 0 bytes to the right of 64-byte region [0x6060000019a0,0x6060000019e0)
allocated by thread T0 here:
#0 0x7fdcd18f7650 in operator new[](unsigned long) (/lib64/libasan.so.5+0xf1650)
#1 0x42c0a7 in ParsedJson::allocateCapacity(unsigned long, unsigned long) include/simdjson/parsedjson.h:53
#2 0x429e87 in validate(char const*) tests/numberparsingcheck.cpp:140
#3 0x42a866 in main tests/numberparsingcheck.cpp:186
#4 0x7fdcd080a412 in __libc_start_main (/lib64/libc.so.6+0x24412)

SUMMARY: AddressSanitizer: unknown-crash /usr/lib/gcc/x86_64-redhat-linux/8/include/avxintrin.h:927 in _mm256_storeu_si256(long long __vector(4)*, long long __vector(4))
Shadow bytes around the buggy address:
0x0c0c7fff82e0: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
0x0c0c7fff82f0: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
0x0c0c7fff8300: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
0x0c0c7fff8310: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
0x0c0c7fff8320: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
=>0x0c0c7fff8330: fa fa fa fa 00 00 00 00 00[00]00 00 fa fa fa fa
0x0c0c7fff8340: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0c7fff8350: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0c7fff8360: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0c7fff8370: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0c7fff8380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==12397==ABORTING
make: *** [Makefile:60: test] Error 1

from simdjson.

Please try to do a git pull to grab my latest commit and rerun.

from simdjson.

"It looks like the code is good!"

Full output here: https://gist.github.com/szydell/4982253537f404317efb0952c7afc32e

Thanks!

from simdjson.

I am going to close this issue. We now have drone.io CI running and so these problems should not reoccur in this manner. (Until now we did not have CI.)

from simdjson.