ohler55 / oj Goto Github PK

View Code? Open in Web Editor NEW

3.1K 40.0 250.0 4.03 MB

Optimized JSON

Home Page: http://www.ohler.com/oj

License: MIT License

Ruby 33.96% C 65.01% Shell 0.26% RPC 0.77%

json-parser json ruby c oj-gem ruby-json-parser marshaller rails

oj's Introduction

gem

A fast JSON parser and Object marshaller as a Ruby gem.

Version 3.13 is out with a much faster parser (Oj::Parser) and option isolation.

Using

require 'oj'

h = { 'one' => 1, 'array' => [ true, false ] }
json = Oj.dump(h)

# json =
# {
#   "one":1,
#   "array":[
#     true,
#     false
#   ]
# }

h2 = Oj.load(json)
puts "Same? #{h == h2}"
# true

Installation

gem install oj

or in Bundler:

gem 'oj'

Rails and json quickstart

See the Quickstart sections of the Rails and json docs.

multi_json

Code which uses multi_json will automatically prefer Oj if it is installed.

Support

Get supported Oj with a Tidelift Subscription. Security updates are supported.

Releases

See {file:CHANGELOG.md} and {file:RELEASE_NOTES.md}

oj's People

Contributors

Stargazers

Watchers

Forkers

msnexploder saulius gnagel mat nevans frsyuki rabidpraxis ecielam khua mutru jarmo getaroom rolftimmermans glebtv davedeveloper khanderson wangjianbo88 sadiqj rgraff dvdplm skaes lautis webmat msykiino kookster wolfhesse dchelimsky mikepence krasnoukhov joshvar88 dataeverywhere nyantani rakibulislam igor-drozdov jondeandres kylefritz eunmin parix deepj aidansteele vincentpaca cdwijayarathna agend igigi alanho habermeier benlovell fxfilmxf shopify marcgg asurin chancedowns enebo estum krainboltgreene nobu comboy mfittko rhenium manlon juanpfcruzx thedrow wanderfalke dtan4 mikfreedman sforce100 hsbt ebenoist intercom jmmalaca stuarthadfield daniilfranks faucct gencer nebolsin wintonpc ianks stereobooster julienitard krallin theorygeek snoopie thomasdziedzic ilyafinkelshteyn mwhobrey misterbyrne bdewater ashmaroli yui-knk knappe hhff smcfarlane yuya-takeyama gerrywastaken mrzacarias irobayna maxlaverse prdn olleolleolle bf4

oj's Issues

times equal to or earlier than 1970-01-01 00:00:00 UTC encode incorrectly

Times are encoded incorrectly (invalid JSON) when their integer representation is zero or negative:

require 'oj'
require 'yajl'
require 'json'
require 'bigdecimal'

stuff = [
  BigDecimal.new('10'),
  Date.new(1970,1,1),
  Time.utc(1970,1,1),
  DateTime.new(1970,1,1)
]

puts Yajl.dump stuff
puts JSON.dump stuff
puts Oj.dump stuff, :mode => :compat

#=> ["0.1E2","1970-01-01","1970-01-01 00:00:00 UTC","1970-01-01T00:00:00+00:00"]
#=> ["0.1E2","1970-01-01","1970-01-01 00:00:00 UTC","1970-01-01T00:00:00+00:00"]
#=> [0.1E2,"1970-01-01",.000000000,"1970-01-01T00:00:00+00:00"]

Oj.dump ignores :symbol_keys option

When dumping a hash, the :symbol_keys option does not seem to have any effect when passed as false, example:

require 'oj'

# notice the ':' prefix of they key
puts Oj.dump({:abc => 123}, :symbol_keys => false)
#=> {":abc":123}

# same result
puts Oj.dump({:abc => 123}, :symbol_keys => true)
#=> {":abc":123}

Is this expected behavior?

Use JSON.create_id

It would be nice if you could implement the possibility do use an other key for creation of custom objects. I think the proper way would be to use the value of JSON.create_id.

https://github.com/ohler55/oj/blob/master/ext/oj/load.c#L479

Fixnum keys are not converted to Strings in compat mode

This was brought to my attention by @rwz in a pull request submitted to MultiJSON: intridea/multi_json#65

Judging from the error message, it appears this behavior is intentional:

TypeError: In :compat mode all Hash keys must be Strings or Symbols, not Fixnum.

I thought the idea behind :compat mode was to be fully compatible with the standard library JSON parser (which converts Fixnum keys to strings)?

What do you think?

Can the circular mode work with client-side JSON libraries?

Hi, I'm in desperate need of JSON libraries in Ruby AND Javascript to transfer complex objects that have a lot of internal references. In this regards oj seems to solve the problem at Ruby's end. However, how do I de-serialize (in the browser) the JSON generated by OJ so that object references are sanely maintained? Are there any Javascript equivalents of OJ?

Also, I noticed that you have come up with your own convention wrt internal object references, i.e. ^i1 and ^r1. Wouldn't it be better to follow either one of the conventions that some people are already recommending:

JSONPath used by the JSON-js library
Oracle's circular reference convention

Install Fails on JRuby

Attempting install through bundler. Debug info here: https://gist.github.com/4126905

Are you able to reproduce? Please let me know if/how I can help fix this.

Thanks!

Oj causing MultiJSON build to fail

For some reason, the MultiJSON's continusous integration broke 7 months ago and didn't get fixed until today. At some point over the past 7 month, a new error related to Oj emerged. It may very well be a bug in MultiJSON, not in Oj. I'm having trouble reproducing it locally. I thought you might have an idea about what the issue is. It does seem to be related to Oj, since the JRuby specs, which skip Oj, are passing. Arguably, we should not skip the Oj specs on JRuby, but that's a different issue.

Anyway, let me know if you have any ideas.

JSYNC

I notice Oj has a serialization format, Object JSON. That's cool. But what I'd really like to see is support for JSYNC (http://jsync.org/).

Oj in :object mode doesn't dump and restore exceptions correctly in ruby 1.8.7

Oj.default_options[:mode]
# => :object
Oj.load(Oj.dump(Exception.new("oops!"))).message
# => "Exception"

This occurs in ree-1.8.7-2012.02, ruby-1.8.7-p358, and rbx-1.2.4, and and rbx-head. I couldn't get oj to build on jruby so I couldn't check. It doesn't affect ruby-1.9.2-p320, ruby-1.9.3-p194.

Syck had/has a similar bug: ruby/ruby@eb1eee0

[windows] oj 1.4.2 tests fail on tcs-ruby

Jarmo and I've discussed on this thread at the tcs-ruby group and I toyed with a extconf.rb quickfix for tcs-ruby.

The main issue is that my trickery with RUBY_DESCRIPTION is messing up your defines/CPPFLAGS from your 'ruby' == type checks. I also DRY'd up your windows platform checks by using RbConfig::CONFIG['host_os'].

I don't like the type.delete('tcs-') nonsense, but I don't have time to look into alternatives like RbConfig::CONFIG['RUBY_BASE_NAME'] which is 1.9-only, not available in JRuby 1.6.8, and I don't know about rubinius.

Test Failures


C:\Users\Jon\Documents\CDev\oj-git>git log -1 --oneline
82e33d5 BigDecimals dumped and loaded in all modes

C:\Users\Jon\Documents\CDev\oj-git>gem build oj.gemspec
  Successfully built RubyGem
  Name: oj
  Version: 1.4.2
  File: oj-1.4.2.gem

C:\Users\Jon\Documents\CDev\oj-git>gem i oj-1.4.2.gem
Building native extensions.  This could take a while...
Successfully installed oj-1.4.2
1 gem installed

C:\Users\Jon\Documents\CDev\oj-git>gem li oj

*** LOCAL GEMS ***

oj (1.4.2, 1.4.1.a1)

C:\Users\Jon\Documents\CDev\oj-git>ruby -v test\tests.rb
tcs-ruby 1.9.3p231 (2012-05-25, TCS patched 2012-05-27) [i386-mingw32]
Run options:

# Running tests:

....................................F........................E......................

Finished tests in 0.031200s, 2692.3077 tests/s, 3974.3590 assertions/s.

  1) Failure:
test_encode(Juice) [test/tests.rb:881]:
<"\u3074\u30FC\u305F\u30FC"> expected but was
<"\xE3\x81\xB4\xE3\x83\xBC\xE3\x81\x9F\xE3\x83\xBC">.

  2) Error:
test_range_object(Juice):
ArgumentError: bad value for range
    test/tests.rb:880:in `initialize'
    test/tests.rb:880:in `load'
    test/tests.rb:880:in `dump_and_load'
    test/tests.rb:622:in `test_range_object'

84 tests, 124 assertions, 1 failures, 1 errors, 0 skips

C:\Users\Jon\Documents\CDev\oj-git>ruby -v test\test_mimic.rb
tcs-ruby 1.9.3p231 (2012-05-25, TCS patched 2012-05-27) [i386-mingw32]
Run options:

# Running tests:

.........E.......

Finished tests in 0.015600s, 1089.7436 tests/s, 1346.1538 assertions/s.

  1) Error:
test_load_proc(Mimic):
NotImplementedError: Calling a Proc with a block not supported in this version. Use func() {|x| } syntax instead.
    test/test_mimic.rb:79:in `load'
    test/test_mimic.rb:79:in `test_load_proc'

17 tests, 21 assertions, 0 failures, 1 errors, 0 skips

C:\Users\Jon\Documents\CDev\oj-git>ruby -v test\test_fast.rb
tcs-ruby 1.9.3p231 (2012-05-25, TCS patched 2012-05-27) [i386-mingw32]
Run options:

# Running tests:

.................................

Finished tests in 0.015600s, 2115.3846 tests/s, 6730.7692 assertions/s.

33 tests, 105 assertions, 0 failures, 0 errors, 0 skips

Oj.load's :symbol_keys options produces segfault when symbol keys given

Hi!

This code produces segfault:

Oj.load('{":s": 1}', :symbol_keys => true)

My Ruby is ruby-1.9.3-p362

BigDecimal values require :compat mode but should not

OJ fails to compile with old version of math.h

The following was witnessed on an old Fedora 10 box:

$ cat gem_make.out 
/opt/ruby-enterprise-1.8.7-2011.03/bin/ruby extconf.rb 
>>>>> Creating Makefile for ree version 1.8.7 <<<<<
creating Makefile

make
gcc  -I. -I/opt/local/include -I/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/i686-linux -I/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/i686-linux -I. -D_FILE_OFFSET_BITS=64  -DHAS_TOP_LEVEL_ST_H=1 -DRUBY_TYPE=ree -DHAS_ENCODING_SUPPORT=0 -DRUBY_VERSION=1.8.7 -DHAS_IVAR_HELPERS=0 -DRUBY_VERSION_MICRO=7 -DRUBY_VERSION_MAJOR=1 -DHAS_PROC_WITH_BLOCK=0 -DHAS_RSTRUCT=1 -DHAS_RB_TIME_TIMESPEC=0 -DREE_RUBY -DRUBY_VERSION_MINOR=8 -DHAS_NANO_TIME=0 -Wall -fPIC -g -O2    -c dump.c
dump.c: In function ‘dump_float’:
dump.c:410: warning: floating constant exceeds range of ‘double’
dump.c:413: warning: floating constant exceeds range of ‘double’
gcc  -I. -I/opt/local/include -I/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/i686-linux -I/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/i686-linux -I. -D_FILE_OFFSET_BITS=64  -DHAS_TOP_LEVEL_ST_H=1 -DRUBY_TYPE=ree -DHAS_ENCODING_SUPPORT=0 -DRUBY_VERSION=1.8.7 -DHAS_IVAR_HELPERS=0 -DRUBY_VERSION_MICRO=7 -DRUBY_VERSION_MAJOR=1 -DHAS_PROC_WITH_BLOCK=0 -DHAS_RSTRUCT=1 -DHAS_RB_TIME_TIMESPEC=0 -DREE_RUBY -DRUBY_VERSION_MINOR=8 -DHAS_NANO_TIME=0 -Wall -fPIC -g -O2    -c load.c
load.c: In function ‘read_num’:
load.c:669: error: ‘INFINITY’ undeclared (first use in this function)
load.c:669: error: (Each undeclared identifier is reported only once
load.c:669: error: for each function it appears in.)
make: *** [load.o] Error 1

I added the following to the top of load.c, and was able to get things make and almost all of the tests to pass:

//Workaround:
#ifndef INFINITY
#define INFINITY (1.0/0.0)
#endif

The only failing test:

$ ruby -wW1 tests.rb
Loaded suite tests
Started
.......................................................F.........
Finished in 0.019552 seconds.

  1) Failure:
test_time_compat(Juice) [tests.rb:249]:
<"1325775487.000000000"> expected but was
<"1325825887.000000000">.

65 tests, 100 assertions, 1 failures, 0 errors

I'm not sure if that is safe to use everywhere, but it's better than not compiling, right? I'll send a pull request shortly.

Oj should not raise SyntaxError when parsing invalid JSON

According to RubyDoc, SyntaxError is "raised when encountering Ruby code with an invalid syntax."

In my opinion, it should not be raised when attempting to parse JSON. SyntaxError (and other descendants of ScriptError) are typically only raised by the Ruby interpreter. Specifically, SyntaxError doesn't descend from StandardError, making it difficult to catch JSON parsing error while still raising Ruby syntax errors that may occur within the same code block.

I'd encourage you to define a custom Oj::ParseError class that inherits from StandardError. I believe this is what the other JSON libraries do.

oj 1.2.9 can't dump BigDecimals, Dates, DateTimes

example program:

require 'oj'
require 'yajl'
require 'json'
require 'bigdecimal'

stuff = [
  BigDecimal.new('10'),
  Date.today,
  Time.now,
  DateTime.now
]

puts Yajl.dump stuff
puts JSON.dump stuff
puts Oj.dump stuff, :mode => :compat

output:

["0.1E2","2012-06-05","2012-06-05 14:56:29 -0700","2012-06-05T14:56:29-07:00"]
["0.1E2","2012-06-05","2012-06-05 14:56:29 -0700","2012-06-05T14:56:29-07:00"]
[null,null,1338933389.857938566,null]

why not?

Add :symbolize_names => true as options to Oj#load under :compat mode

The standard 'json' gem supports the :symbolize_names => true, option to symbolize all hash keys to ruby symbols upon parsing. When there is large number of deserialized JSON objects with similar keys, this could could potentially save considerable amount of memory and is faster for key comparisons.

As such I would suggest Oj to add the support for this option under the :compat mode, both for the reasons above and making :compat mode closer to the 'json' gem.

Tag releases

I recently wanted to see what code changed in versions 1.3.6 and 1.3.7 using GitHub's Compare View but couldn't easily diff those versions because the releases were not tagged.

It appears that you tagged v1.0.6 but haven't tagged any releases before or since.

When I release gems, I use a rake task that automatically:

builds a gem from the gemspec,
pushes a tag to GitHub, and
pushes the gem to RubyGems.org.

I believe oj could benefit from a similar automation of the release process that includes tagging.

For example, here's the Compare View for the 3.4.1 release of the twitter gem. If you want to see a wider range of changes, you can simply change the tags referenced in the URL, for example, here's the Compare View for all the changes between 3.4.0 and 3.5.0.

Not compatible with 1.8.7

The readme states compatibility with 1.8.7 but when I tried to use it locally it appears that there are 1.9-only method signatures and defines being used.

So far from what I can see, DBL2NUM and rb_hash_lookup2 are the two offenders. There could be more but I haven't had a chance to dig down further. DBL2NUM appears to just use rb_float_new (which exists in 1.8.7 as well) but the change from rb_hash_lookup2 will require some small refactoring I think?

OJ Fails to compile on OS X

Hey There,

Was trying to use the T gem which pulls in oj as a dependency. Trying to install OJ with sudo gem install oj fails in this manner (running OS X 10.7.3 with system ruby 1.8.7):

Error

└(~/src/puppetlabs-training)▷ sudo gem install oj
Building native extensions.  This could take a while...
ERROR:  Error installing oj:
    ERROR: Failed to build gem native extension.

/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb
>>>>> Creating Makefile for ruby version 1.8.7 <<<<<
creating Makefile

make
gcc -I. -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin11.0 -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin11.0 -I. -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE   -DHAS_TOP_LEVEL_ST_H=1 -DRUBY_TYPE=ruby -DHAS_ENCODING_SUPPORT=0 -DRUBY_VERSION=1.8.7 -DHAS_IVAR_HELPERS=0 -DRUBY_VERSION_MICRO=7 -DRUBY_RUBY -DRUBY_VERSION_MAJOR=1 -DHAS_PROC_WITH_BLOCK=0 -DHAS_RSTRUCT=1 -DHAS_RB_TIME_TIMESPEC=0 -DRUBY_VERSION_MINOR=8 -DHAS_NANO_TIME=0 -Wall -fno-common -arch i386 -arch x86_64 -g -Os -pipe -fno-common -DENABLE_DTRACE  -fno-common  -pipe -fno-common   -c cache.c
gcc -I. -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin11.0 -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin11.0 -I. -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE   -DHAS_TOP_LEVEL_ST_H=1 -DRUBY_TYPE=ruby -DHAS_ENCODING_SUPPORT=0 -DRUBY_VERSION=1.8.7 -DHAS_IVAR_HELPERS=0 -DRUBY_VERSION_MICRO=7 -DRUBY_RUBY -DRUBY_VERSION_MAJOR=1 -DHAS_PROC_WITH_BLOCK=0 -DHAS_RSTRUCT=1 -DHAS_RB_TIME_TIMESPEC=0 -DRUBY_VERSION_MINOR=8 -DHAS_NANO_TIME=0 -Wall -fno-common -arch i386 -arch x86_64 -g -Os -pipe -fno-common -DENABLE_DTRACE  -fno-common  -pipe -fno-common   -c cache8.c
gcc -I. -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin11.0 -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin11.0 -I. -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE   -DHAS_TOP_LEVEL_ST_H=1 -DRUBY_TYPE=ruby -DHAS_ENCODING_SUPPORT=0 -DRUBY_VERSION=1.8.7 -DHAS_IVAR_HELPERS=0 -DRUBY_VERSION_MICRO=7 -DRUBY_RUBY -DRUBY_VERSION_MAJOR=1 -DHAS_PROC_WITH_BLOCK=0 -DHAS_RSTRUCT=1 -DHAS_RB_TIME_TIMESPEC=0 -DRUBY_VERSION_MINOR=8 -DHAS_NANO_TIME=0 -Wall -fno-common -arch i386 -arch x86_64 -g -Os -pipe -fno-common -DENABLE_DTRACE  -fno-common  -pipe -fno-common   -c dump.c
In file included from dump.c:38:
oj.h:84: error: expected specifier-qualifier-list before ‘uint8_t’
oj.h:119: error: expected specifier-qualifier-list before ‘uint8_t’
dump.c:58: error: expected specifier-qualifier-list before ‘uint32_t’
dump.c:166: error: expected ‘)’ before ‘num’
dump.c: In function ‘dump_unicode’:
dump.c:228: error: ‘uint32_t’ undeclared (first use in this function)
dump.c:228: error: (Each undeclared identifier is reported only once
dump.c:228: error: for each function it appears in.)
dump.c:228: error: expected ‘;’ before ‘code’
dump.c:229: error: ‘uint8_t’ undeclared (first use in this function)
dump.c:229: error: expected ‘;’ before ‘b’
dump.c:232: error: ‘b’ undeclared (first use in this function)
dump.c:234: error: ‘code’ undeclared (first use in this function)
dump.c:252: error: expected expression before ‘)’ token
dump.c:259: error: expected ‘;’ before ‘c1’
dump.c:262: error: ‘c1’ undeclared (first use in this function)
dump.c: In function ‘dump_array’:
dump.c:598: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:598: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:607: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:609: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:611: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:615: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:632: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:634: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:636: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:640: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c: In function ‘hash_cb_strict’:
dump.c:666: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:666: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:670: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:672: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:674: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:678: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:682: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:682: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:686: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:688: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:691: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:693: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c: In function ‘hash_cb_compat’:
dump.c:715: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:715: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:719: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:721: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:723: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:727: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:745: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:745: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:749: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:751: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:754: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:756: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c: In function ‘hash_cb_object’:
dump.c:796: error: ‘struct _Out’ has no member named ‘hash_cnt’
dump.c:798: error: ‘struct _Out’ has no member named ‘hash_cnt’
dump.c: In function ‘dump_hash’:
dump.c:877: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:877: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:881: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:883: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:885: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:889: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c: In function ‘dump_obj_to_json’:
dump.c:1301: error: ‘struct _Out’ has no member named ‘hash_cnt’
dump.c: In function ‘dump_leaf_str’:
dump.c:1355: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c:1364: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c: In function ‘dump_leaf_fixnum’:
dump.c:1371: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c:1384: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c: In function ‘dump_leaf_float’:
dump.c:1391: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c:1400: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c: In function ‘dump_leaf’:
dump.c:1485: error: ‘struct _Leaf’ has no member named ‘type’
dump.c:1511: error: ‘struct _Leaf’ has no member named ‘type’
dump.c: In function ‘dump_leaf_to_json’:
dump.c:1523: error: ‘struct _Out’ has no member named ‘hash_cnt’
In file included from dump.c:38:
oj.h:84: error: expected specifier-qualifier-list before ‘uint8_t’
oj.h:119: error: expected specifier-qualifier-list before ‘uint8_t’
dump.c:58: error: expected specifier-qualifier-list before ‘uint32_t’
dump.c:166: error: expected ‘)’ before ‘num’
dump.c: In function ‘dump_unicode’:
dump.c:228: error: ‘uint32_t’ undeclared (first use in this function)
dump.c:228: error: (Each undeclared identifier is reported only once
dump.c:228: error: for each function it appears in.)
dump.c:228: error: expected ‘;’ before ‘code’
dump.c:229: error: ‘uint8_t’ undeclared (first use in this function)
dump.c:229: error: expected ‘;’ before ‘b’
dump.c:232: error: ‘b’ undeclared (first use in this function)
dump.c:234: error: ‘code’ undeclared (first use in this function)
dump.c:252: error: expected expression before ‘)’ token
dump.c:259: error: expected ‘;’ before ‘c1’
dump.c:262: error: ‘c1’ undeclared (first use in this function)
dump.c: In function ‘dump_array’:
dump.c:598: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:598: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:607: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:609: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:611: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:615: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:632: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:634: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:636: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:640: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c: In function ‘hash_cb_strict’:
dump.c:666: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:666: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:670: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:672: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:674: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:678: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:682: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:682: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:686: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:688: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:691: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:693: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c: In function ‘hash_cb_compat’:
dump.c:715: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:715: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:719: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:721: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:723: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:727: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:745: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:745: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:749: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:751: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:754: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:756: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c: In function ‘hash_cb_object’:
dump.c:796: error: ‘struct _Out’ has no member named ‘hash_cnt’
dump.c:798: error: ‘struct _Out’ has no member named ‘hash_cnt’
dump.c: In function ‘dump_hash’:
dump.c:877: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:877: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:881: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:883: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:885: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:889: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c: In function ‘dump_obj_to_json’:
dump.c:1301: error: ‘struct _Out’ has no member named ‘hash_cnt’
dump.c: In function ‘dump_leaf_str’:
dump.c:1355: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c:1364: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c: In function ‘dump_leaf_fixnum’:
dump.c:1371: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c:1384: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c: In function ‘dump_leaf_float’:
dump.c:1391: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c:1400: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c: In function ‘dump_leaf’:
dump.c:1485: error: ‘struct _Leaf’ has no member named ‘type’
dump.c:1511: error: ‘struct _Leaf’ has no member named ‘type’
dump.c: In function ‘dump_leaf_to_json’:
dump.c:1523: error: ‘struct _Out’ has no member named ‘hash_cnt’
lipo: can't open input file: /var/tmp//cckBhLRR.out (No such file or directory)
make: *** [dump.o] Error 1


Gem files will remain installed in /Library/Ruby/Gems/1.8/gems/oj-1.2.5 for inspection.
Results logged to /Library/Ruby/Gems/1.8/gems/oj-1.2.5/ext/oj/gem_make.out

I presume I'll probably need a couple of extra flags to compile on OS X, but I thought I'd file a ticket on it.

Oj.mimic_JSON throws an error if json is already required.

Hi,

nothing like an example to describe this issue:

╰─○ irb
>> require 'json'
=> true
>> require 'oj'
=> true
>> Oj.mimic_JSON
TypeError: Generator is not a class
        from (irb):3:in `mimic_JSON'
        from (irb):3
        from /home/fbernier/.rvm/rubies/ruby-1.9.3-p194/bin/irb:16:in `<main>'
>> Oj.mimic_JSON
=> JSON

╰─○ irb
>> require 'oj'
=> true
>> Oj.mimic_JSON
=> JSON

As you can see, if json is already included, it Oj,mimic_JSON thorws an error, but if I call it again, it works. Any clue on this?

Installation fails under Windows

Trying to install under Windows (ruby 1.9.3p231), but failing:

C:\Users\jarmo>gem install oj
Temporarily enhancing PATH to include DevKit...
Building native extensions.  This could take a while...
ERROR:  Error installing oj:
        ERROR: Failed to build gem native extension.

        C:/tcs-ruby193_require_fenix_gc_hash_20120527/bin/ruby.exe extconf.rb
>>>>> Creating Makefile for tcs-ruby version 1.9.3 <<<<<
creating Makefile

make
generating oj-i386-mingw32.def
compiling cache.c
<command-line>:0:4: warning: missing whitespace after the macro name
cache.c: In function 'oj_cache_new':
cache.c:54:5: warning: implicit declaration of function 'bzero'
cache.c:54:5: warning: incompatible implicit declaration of built-in function 'bzero'
compiling cache8.c
<command-line>:0:4: warning: missing whitespace after the macro name
compiling dump.c
<command-line>:0:4: warning: missing whitespace after the macro name
In file included from c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby/defines.h:223:0,
                 from c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby/ruby.h:67,
                 from c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby.h:32,
                 from oj.h:43,
                 from dump.c:39:
c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby/win32.h:282:12: error: conflicting types for 'gettimeofday'
c:\bin\devkit\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/sys/time.h:39:29: note: previous declaration of 'gettimeofday' was here
In file included from c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby/ruby.h:1381:0,
                 from c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby.h:32,
                 from oj.h:43,
                 from dump.c:39:
c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby/missing.h:48:8: error: redefinition of 'struct timezone'
c:\bin\devkit\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/sys/time.h:26:8: note: originally defined here
In file included from dump.c:39:0:
oj.h:50:21: fatal error: pthread.h: No such file or directory
compilation terminated.
make: *** [dump.o] Error 1


Gem files will remain installed in C:/tcs-ruby193_require_fenix_gc_hash_20120527/lib/ruby/gems/1.9.1/gems/oj-1.3.4 for inspection.
Results logged to C:/tcs-ruby193_require_fenix_gc_hash_20120527/lib/ruby/gems/1.9.1/gems/oj-1.3.4/ext/oj/gem_make.out

:compat differs from JSON & YAJL for Time instances

In Rails: puts JSON.dump([Time.now]) # -> ["2012-12-05T16:52:54+01:00"]
In Plain Ruby: puts JSON.dump([Time.now]) # -> ["2012-12-05 16:54:23 +0100"]
Oj: puts Oj.dump([Time.now], mode: :compat) # -> [1354722798.022583000]

The rails version makes more sense than the plain ruby one, since it can be used by JS' Date constructor (new Date("2012-12-05T16:52:54+01:00") is valid and correct).

I'm not sure about the proper solution, though.

Proposal: support JSON+Comments

Most other JSON decoders strip out //...\n and /*...*/ comments before processing. oj is one of the few that does not. The JSON spec does not require decoders to do so, but it does offer the option. It would be nice to have consistency across all decoders.

cf intridea/multi_json#38

JSON is too deeply nested (64 bits)

I have an issue loading JSON object with Oj in a set of machines. Some examples:

1.9.3p0 :016 > res=Oj.dump({:a=>'b'})
 => "{\":a\":\"b\"}"
1.9.3p0 :015 > Oj.load(res)
SystemStackError: JSON is too deeply nested
1.9.3p0 :015 > Oj.load('{}')
SystemStackError: JSON is too deeply nested

I'm have the same Ruby version 1.9.3p0 in both machines. The working machines are 32 bits (Ubuntu 10.04) and the failing machines are 64 bits (12.10 and 12.04).

Some suggestion?

generate json same as default json library?

require 'json'
require 'oj' # json

h = {:url => "http://google.com" }
puts "to_json: " << h.to_json
puts "Oj.dump: " << Oj.dump(h)

outputs:

to_json: {"url":"http://google.com"}
Oj.dump: {":url":"http://google.com"}

How can I get it to work the same as to_json (remove ":" from symbol, not escape \ )

Mimic_JSON fails

Based on this #5 when I try the mimic option I get the following error -

1.9.3-p194 :001 > Oj.mimic_JSON
TypeError: JSON module already exists. Can not mimic. Do not require 'json' before calling mimic_JSON.

This is a Rails 3.2.8 and is using oj 1.3.2

Thanks for your help.

Benchmarks

In my production application, I ran some benchmarks to find the fastest JSON rendering engine. I'm using RABL to render.

I ran the same test, for 6 different templates (our most used ones), with each record from our DB, 10 times each.
I ran once for yajl and once for oj. I did not perform any configuration for either.
I rant the tests on a staging machine (same config as production), from a machine in the same EC2 zone.
Results were consistent when run multiple times.
Rails was running 3.2.9, Ruby 1.9.3, Apache.

I found that yajl was faster. I wasn't expecting it to be. Is this possible/reasonable in some circumstances (every app is different)? Is there anything I else I might have missed or might want to consider before moving ahead?

Thanks.

Long Floats throw parse errors

>> Oj.load("149235374128953450.0")
SyntaxError: invalid format, extra characters at line 1, column 19 [load.c:935]

    from (irb):8:in `load'
    from (irb):8
    from /Users/alx/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `<main>'
>> Oj.load("[1492353741289534500000000.0]")
SyntaxError: invalid format, expected , or ] while in an array at line 1, column 27 [load.c:570]

    from (irb):9:in `load'
    from (irb):9
    from /Users/alx/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `<main>'

Is that valide behavior? Do you need more input?

:mode => :compat is strict?

I'm using oj, multi_json, and rabl to spit out json from my rails-api. The problem I'm having is that in production mode the options are set to this:

{:indent=>0, :circular=>false, :auto_define=>true, :ascii_only=>false, :symbol_keys=>false, :mode=>:compat, :create_id=>"json_class"}

And yet I get this error:

TypeError (In :strict mode all Hash keys must be Strings.)

Which sucks, because it's not actually in strict mode, but compat mode.

I think the problem is here:

https://github.com/ohler55/oj/blob/master/ext/oj/dump.c#L739

Empty String causing Oj::ParseError

Should an empty string really cause a ParseError?

I'm currently migrating from Yajl which didn't throw an exception if parsing an empty string and wondered if there is a reason behind not catching empty strings and simply returning?

I'm sure the reason is that "" is infact invalid json, however from a library perspective it would be nice to see that case caught instead of throwing an exception imo.

Thanks 👍

Install fails on jruby-1.6.7.2-d19

The full details are on Travis:
http://travis-ci.org/#!/simeonwillbanks/nerd_quiz/jobs/1525709

Thanks for Oj!

Simeon

Less precise xmlschema format

For better compatibility with other JSON libraries, it'd be nice if Oj could be configured to use the a date format option that acted like xmlschema but always treated nsec as zero.

Seg fault with 1.2.9 / ruby 1.9.3-p194

I can make Oj.load segfault with the json from https://gist.github.com/2937493. I've also added the terminal output and the osx crash reporter output to that gist.

Anything I can do to help fix this? Would really like some v. fast json serialisation for my project.

:mode => :compat silently prefers as_json over to_json

Docs state that the :compat mode:

... will serialize any Object but will check to see if the Object implements a to_hash() or to_json() method. If either exists that method is used for serializing the Object

Then in dump.c:977 there is a test for as_json:

    } else if (rb_respond_to(obj, oj_as_json_id)) {
    dump_val(rb_funcall(obj, oj_as_json_id, 0), depth, out);
    } else if (rb_respond_to(obj, oj_to_json_id)) {

which is undocumented (so not sure if this is cruft or an omission in the docs).

If you then combine this with ActiveSupport which mixes as_json into Object, then to_json will never get called.

JSON Gem compatibility in Rails

Do I need to configure anything to get JSON Gem compatibility in Rails?

JSON::Ext::Class is not a module on ubuntu 12.04

i am upgrading my box from 10.04 to 12.04 and in 12.04 i get the following error. 10.04 works as expected. I am bundling these versions:

Installing oj (1.4.5) with native extensions
Installing oj_mimic_json (1.0.0)

rake aborted!
JSON::Ext::Class is not a module

(See full trace by running task with --trace)

bundle exec rake --trace parallel:drop parallel:create parallel:migrate parallel:prepare
rake aborted!
JSON::Ext::Class is not a module
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/json_pure-1.7.3/lib/json/ext.rb:14:in require' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/json_pure-1.7.3/lib/json/ext.rb:14:inmodule:Ext'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/json_pure-1.7.3/lib/json/ext.rb:12:in <module:JSON>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/json_pure-1.7.3/lib/json/ext.rb:9:in<top (required)>'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/json_pure-1.7.3/lib/json.rb:58:in require' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/json_pure-1.7.3/lib/json.rb:58:inmodule:JSON'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/json_pure-1.7.3/lib/json.rb:54:in <top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/activesupport-3.0.14/lib/active_support/core_ext/object/to_json.rb:3:inrequire'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/activesupport-3.0.14/lib/active_support/core_ext/object/to_json.rb:3:in <top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/activesupport-3.0.14/lib/active_support/json/encoding.rb:1:inrequire'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/activesupport-3.0.14/lib/active_support/json/encoding.rb:1:in <top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/activesupport-3.0.14/lib/active_support/json.rb:2:inrequire'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/activesupport-3.0.14/lib/active_support/json.rb:2:in <top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/prototype_helper.rb:2:inrequire'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/prototype_helper.rb:2:in <top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/javascript_helper.rb:38:inmodule:JavaScriptHelper'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/javascript_helper.rb:37:in <module:Helpers>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/javascript_helper.rb:5:inmodule:ActionView'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/javascript_helper.rb:3:in <top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/url_helper.rb:1:inrequire'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/url_helper.rb:1:in <top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/form_tag_helper.rb:16:inmodule:FormTagHelper'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/form_tag_helper.rb:13:in <module:Helpers>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/form_tag_helper.rb:7:inmodule:ActionView'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/form_tag_helper.rb:5:in <top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/form_helper.rb:4:inrequire'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/form_helper.rb:4:in <top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/active_model_helper.rb:1:inrequire'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/active_model_helper.rb:1:in <top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers.rb:37:inmodule:Helpers'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers.rb:4:in <module:ActionView>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers.rb:3:in<top (required)>'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/base.rb:164:in <class:Base>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/base.rb:160:inmodule:ActionView'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/base.rb:8:in <top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/bundler/gems/s3-swf-upload-plugin-f669a2322b7b/lib/s3_swf_upload/view_helpers.rb:171:in<top (required)>'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/bundler/gems/s3-swf-upload-plugin-f669a2322b7b/lib/s3_swf_upload.rb:4:in require' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/bundler/gems/s3-swf-upload-plugin-f669a2322b7b/lib/s3_swf_upload.rb:4:in<top (required)>'
/usr/lib/ruby/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/runtime.rb:68:in require' /usr/lib/ruby/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/runtime.rb:68:inblock (2 levels) in require'
/usr/lib/ruby/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/runtime.rb:66:in each' /usr/lib/ruby/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/runtime.rb:66:inblock in require'
/usr/lib/ruby/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/runtime.rb:55:in each' /usr/lib/ruby/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/runtime.rb:55:inrequire'
/usr/lib/ruby/gems/1.9.1/gems/bundler-1.2.1/lib/bundler.rb:128:in require' /tmp/jenkins/workspace/monorail-1204/config/application.rb:4:in<top (required)>'
/tmp/jenkins/workspace/monorail-1204/Rakefile:6:in require' /tmp/jenkins/workspace/monorail-1204/Rakefile:6:in<top (required)>'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_module.rb:25:in load' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_module.rb:25:inload_rakefile'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:501:in raw_load_rakefile' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:82:inblock in load_rakefile'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:133:in standard_exception_handling' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:81:inload_rakefile'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:65:in block in run' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:133:instandard_exception_handling'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:63:in run' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/bin/rake:33:in<top (required)>'
/tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/bin/rake:23:in load' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/bin/rake:23:in'

Add an :ascii_only option to the encoder

I haven't had a chance to really dig through to see if UTF-8 characters are always \uXXXX escaped or not, but it would be nice to have the option to allow them to go right into the output stream unescaped (except for characters which must be escaped per the JSON spec).

The :ascii_only option could maybe default to whatever the behavior is today.

What do you think?

SyntaxError when string starts with caret & i followed by any other character

>> Oj.load("\"i \"")
=> "i "
>> Oj.load("\"^i\"")
=> "^i"
>> Oj.load("\"^i \"")
SyntaxError: Not a valid ID number at line 1, column 6 [load.c:212]

    from (irb):9:in `load'
    from (irb):9
    from /Users/alx/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `<main>'
>> Oj.load("\"^ia\"")
SyntaxError: Not a valid ID number at line 1, column 6 [load.c:212]

    from (irb):5:in `load'
    from (irb):5
    from /Users/alx/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `<main>'

ruby: 1.9.2p290
oj: 1.2.9

Please let me know, if you need more info.

TypeError at /sidekiq/retries/1345106995.7055612

When I switched from using yajl to oj I ran into this issue and when I switch back to yajl it works fine again. It's not sidekiqs fault. For some reason oj interprets dates as float or vice versa.

See this related Sidekiq issue for more details sidekiq/sidekiq#354

Avoiding backslash for JSON strings

Hello, Oj looks quite impressive. First time using it. But I have a question:

The regular JSON string outputs something like this:

{"json":[{"id":1,"title":"Casablanca"}]}

While the Oj.dump puts backslashes before the double quotes:

{"json":"[{\"id\":1,\"title\":\"Casablanca\"}]"}

I was wondering if there is a way to avoid them, short of replacing them of course which would make things much slower.

Thanks.

test_exception fail on MRI trunk

I'm getting failures on both Win7 32bit and Arch 32bit on different versions of MRI trunk. Last example is from a passing run on 1.9.3p297 on Win7.

~bt is null in both of the trunk failing cases.

Arch 3.6.2 failure (trunk@37049)


[jon@archee ~]$ uname -a
Linux archee 3.6.2-1-ARCH #1 SMP PREEMPT Sat Oct 13 07:19:40 CEST 2012 i686 GNU/Linux

[jon@archee oj-git]$ gem i oj-1.4.2.a1.gem  --user-install
Building native extensions.  This could take a while...
Successfully installed oj-1.4.2.a1
1 gem installed
[jon@archee oj-git]$ ruby -v test/tests.rb
ruby 2.0.0dev (2012-09-28 trunk 37049) [i686-linux]
Run options:

# Running tests:

[38/84] Juice#test_exception*** {
  "^o":"StandardError",
  "~mesg":"A Message",
  "~bt":null}
 = 0.00 s
  1) Failure:
test_exception(Juice) [test/tests.rb:580]:
<["test/tests.rb:570:in `test_exception'",
 "/usr/local/lib/ruby/2.0.0/test/unit.rb:844:in `run_test'",
 "/usr/local/lib/ruby/2.0.0/minitest/unit.rb:1211:in `run'",
 "/usr/local/lib/ruby/2.0.0/test/unit/testcase.rb:17:in `run'",
 "/usr/local/lib/ruby/2.0.0/minitest/unit.rb:837:in `block in _run_suite'",
 "/usr/local/lib/ruby/2.0.0/minitest/unit.rb:829:in `map'",
 "/usr/local/lib/ruby/2.0.0/minitest/unit.rb:829:in `_run_suite'",
 "/usr/local/lib/ruby/2.0.0/test/unit.rb:650:in `block in _run_suites'",
 "/usr/local/lib/ruby/2.0.0/test/unit.rb:648:in `each'",
 "/usr/local/lib/ruby/2.0.0/test/unit.rb:648:in `_run_suites'",
 "/usr/local/lib/ruby/2.0.0/minitest/unit.rb:789:in `_run_anything'",
 "/usr/local/lib/ruby/2.0.0/minitest/unit.rb:978:in `run_tests'",
 "/usr/local/lib/ruby/2.0.0/minitest/unit.rb:965:in `block in _run'",
 "/usr/local/lib/ruby/2.0.0/minitest/unit.rb:964:in `each'",
 "/usr/local/lib/ruby/2.0.0/minitest/unit.rb:964:in `_run'",
 "/usr/local/lib/ruby/2.0.0/minitest/unit.rb:953:in `run'",
 "/usr/local/lib/ruby/2.0.0/test/unit.rb:21:in `run'",
 "/usr/local/lib/ruby/2.0.0/test/unit.rb:767:in `run'",
 "/usr/local/lib/ruby/2.0.0/test/unit.rb:364:in `block (2 levels) in autorun'",
 "/usr/local/lib/ruby/2.0.0/test/unit.rb:27:in `run_once'",
 "/usr/local/lib/ruby/2.0.0/test/unit.rb:363:in `block in autorun'"]> expected but was
<nil>.

Finished tests in 0.063030s, 1332.7092 tests/s, 2030.7949 assertions/s.
84 tests, 128 assertions, 1 failures, 0 errors, 0 skips

Win7 failure (trunk@37268)


C:\Users\Jon\Documents\CDev\oj-git>gem i oj-1.4.2.a1.gem
Temporarily enhancing PATH to include DevKit...
Building native extensions.  This could take a while...
Successfully installed oj-1.4.2.a1
1 gem installed

C:\Users\Jon\Documents\CDev\oj-git>ruby -v test\tests.rb
ruby 2.0.0dev (2012-10-19 trunk 37268) [i386-mingw32]
Run options:

# Running tests:

[38/84] Juice#test_exception*** {
  "^o":"StandardError",
  "~mesg":"A Message",
  "~bt":null}
 = 0.00 s
  1) Failure:
test_exception(Juice) [test/tests.rb:580]:
<["test/tests.rb:570:in `test_exception'",
 "C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:844:in `run_test'",
 "C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:1211:in `run'",
 "C:/rubytrunk/lib/ruby/2.0.0/test/unit/testcase.rb:17:in `run'",
 "C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:837:in `block in _run_suite'",
 "C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:829:in `map'",
 "C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:829:in `_run_suite'",
 "C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:650:in `block in _run_suites'",
 "C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:648:in `each'",
 "C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:648:in `_run_suites'",
 "C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:789:in `_run_anything'",
 "C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:978:in `run_tests'",
 "C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:965:in `block in _run'",
 "C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:964:in `each'",
 "C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:964:in `_run'",
 "C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:953:in `run'",
 "C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:21:in `run'",
 "C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:767:in `run'",
 "C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:364:in `block (2 levels) in autorun'",
 "C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:27:in `run_once'",
 "C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:363:in `block in autorun'"]> expected but was
<nil>.

Finished tests in 0.123515s, 680.0793 tests/s, 1036.3114 assertions/s.
84 tests, 128 assertions, 1 failures, 0 errors, 0 skips

Win7 success (1.9.3p297)


C:\Users\Jon\Documents\CDev\oj-git>ruby -v test\tests.rb
ruby 1.9.3p297 (2012-10-19 revision 37266) [i386-mingw32]
Run options:

# Running tests:

.....................................*** {
  "^o":"StandardError",
  "~mesg":"A Message",
  "~bt":[
    "test/tests.rb:570:in `test_exception'",
    "C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'",
    "C:/ruby193/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'",
    "C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'",
    "C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'",
    "C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'",
    "C:/ruby193/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'",
    "C:/ruby193/lib/ruby/1.9.1/test/unit.rb:563:in `each'",
    "C:/ruby193/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'",
    "C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'",
    "C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'",
    "C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'",
    "C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'",
    "C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'",
    "C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'",
    "C:/ruby193/lib/ruby/1.9.1/test/unit.rb:21:in `run'",
    "C:/ruby193/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'",
    "C:/ruby193/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'",
    "C:/ruby193/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'"
  ]}
...............................................

Finished tests in 0.038004s, 2210.2937 tests/s, 3394.3795 assertions/s.

84 tests, 129 assertions, 0 failures, 0 errors, 0 skips

Default encoding for Ruby 1.9

Hi,

I think Oj should respect the default encoding on Ruby 1.9. For example, I'd expect an UTF-8 string when dumping a hash in this context:

[1] pry(main)> puts Encoding.default_external
UTF-8
=> nil
[2] pry(main)> require "oj"
=> true
[3] pry(main)> puts Oj.dump(foo: "éà").encoding
ASCII-8BIT

Bus Error when parsing deeply-nested objects

If an untrusted source returns a JSON document that is deeply nested, it can cause oj to crash hard.

Reproduction steps:

> ruby -r oj -e 'Oj.load(-> n { ?[ * n  + ?] * n }.(9999999))'
-e:1: [BUG] Bus Error

(you may need to add nines, depending on how much free memory you have)

See intridea/multi_json#59 for an in-depth discussion of the issue.

Can't override timestamp format for Time objects

I'm trying to replace my use of the json gem with oj, but I need to ensure backwards compatibility in timestamp formats. I'm using the equivalent of Time#xmlschema (e.g. "2012-07-06T17:44:33-04:00"). Oj insists on a Unix timestamp format for Time objects.

I've tried monkey patching a Time#to_json method, with no success. Do I have any other options?

Dump Time in rails native format

Hey Petr,
can you please add support for rails native time format (ISO 8061)? It can be called rails :time_format => :rails

Here is a small sample, so you can see a difference:

t = [Time.now]

# rails native
puts JSON.dump t
# => ["2013-01-25T14:55:10+01:00"]

# oj modes
puts Oj.dump t, {:mode => :compat, :time_format => :unix}
# => [1359122110.187338000]
puts Oj.dump t, {:mode => :compat, :time_format => :ruby}
# => ["2013-01-25 14:55:10 +0100"]
puts Oj.dump t, {:mode => :compat, :time_format => :xmlschema}
# => ["2013-01-25T14:55:10.187338000+01:00"]

Basically its almost exactly the same like :xmlschema format just without milliseconds.

double free or corruption

I'm currently experiencing strange aborts with the latest Oj (1.2.11). i've tracked it down with gdb to fast.c:1121 where xfree(json) is called. the abort trace is as follows:

#0  0x00007ffff6b52b45 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007ffff6b53fc6 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007ffff6b8dfe7 in __libc_message () from /lib64/libc.so.6
No symbol table info available.
#3  0x00007ffff6b93516 in malloc_printerr () from /lib64/libc.so.6
No symbol table info available.
#4  0x00007ffff6b9839c in free () from /lib64/libc.so.6
No symbol table info available.
#5  0x00007fffe07a43f2 in doc_open (clas=578516760, str=703305680) at fast.c:1121
        json = 0x2a327b10 "\240w2*"
        len = 201005
        obj = 707934560
        given = 1
        allocate = 1
#6  0x00007ffff7b162c1 in vm_call_cfunc (me=0x227e3820, blockptr=0x2a1f85f0, recv=<optimized out>, num=1, reg_cfp=0x2a1f85c8, th=<optimized out>) at vm_insnhelper.c:404
        val = 0
        def = 0x227e3850
        cfp = 0x6ae4
#7  vm_call_method (th=<optimized out>, cfp=0x2a1f85c8, num=<optimized out>, blockptr=0x2a1f85f0, flag=<optimized out>, id=<optimized out>, me=0x227e3820, recv=578516760)
    at vm_insnhelper.c:534
        val = <optimized out>

i'm not sure where the double free is coming from ... i've also tried to disable GC and got the same results.

if you need more information please let me know.

Oj parsing issue

Oj shouldn't produce symbols when parsing a JSON message:

res = Oj.dump {:a=>'b'} #=> "{\":a\":\"b\"}"
Oj.load(res) #=> {:a => 'b'}

It's wrong because the format never specifies the symbol type which opens unexpected behaviors but more importantly, symbols are never garbage collected which means that an attacker could easily flood a VM with unused objects.

undefined method `unparse' for JSON:Module

After calling Oj.mimic_JSON, I noticed that the unparse method is not supported.

Should that module function also be redefined by Oj?

Fiber stack limits

So this is a tough one as it is/was incredibly difficult to debug. I used oj with Sidekiq, which is using the Celluloid actor framework. I ran into the issue that workers were dying silently. After almost a week of debugging, it turned out that Celluloid Fibers exceeded the static 4k limit and died (without leaving much of a backtrace).

I traced to cause of the SystemStackError: stack level too deep down to oj. Once I switched to yajl the errors were gone - no more dead actors. I was wondering, is there an incompatibility between oj and Fibers? Is oj using the stack too aggressively? Anything I can do to help debug this further?

Here is the full story: sidekiq/sidekiq#233

Define RSTRING_NOT_MODIFIED for a Rubinius speedup

Defining RSTRING_NOT_MODIFIED before you include ruby.h will tell Rubinius to shut off it's pointer caching compatibility API. By doing so, you're making a promise to rbx that you won't modify any pointers you get back from the Ruby VM.

I haven't had a chance to look over the codebase here to see if you're doing that anywhere, but this could offer a pretty significant speedup on Rubinius (I saw something like a 25x speedup with yajl-ruby).

The Rubinius guys can offer a much more detailed explanation of how it works if you're interested. They hang out in the #rubinius channel on chat.freenode.net.

Here's the example from yajl-ruby: https://github.com/brianmario/yajl-ruby/blob/master/ext/yajl/yajl_ext.h#L27-30