take-cheeze / mruby-marshal Goto Github PK

View Code? Open in Web Editor NEW

12.0 12.0 9.0 119 KB

mruby implementation of cruby marshaling.

Ruby 20.46% C 3.02% C++ 76.52%

mruby-marshal's People

Contributors

Stargazers

Watchers

Forkers

ysei pbosetti asfluido waveformdelta krissi holywyvern pulsejet

mruby-marshal's Issues

Uncaught exception when using this gem on iOS

I am using this gem with the ios-embedded-ruby project and including it in the MRubyiOSExample app. However, including this (or any C++ gem) into the project causes a problem with exception handling when I run it on the device. I can cause an unrecoverable crash with code as simple as:

begin
  raise StandardError, "This is an example error"
rescue => ex
  p "Caught error: #{ex.class}, #{ex.message}"
end

On a device, the raise statement will crash with a SIGABRT on the call to MRB_THROW in the mrb_exc_raise() method in the error.c source file with the message:

libc++abi.dylib: terminating with uncaught exception of type int

Additional information

This only happens on a device, not a simulator (note that ios-embedded-ruby builds for both targets)
This is not specific to this gem, but seems to be a problem with building C++ code for iOS; I verified this by making a 'dummy' gem that uses C++ and including only that, and got the same behavior; if I build it as C, the crash doesn't happen
To get this mruby-marshal gem to build for iOS, I had to replace the mruby-onig-regexp gem dependency with the mruby-pure-regexp gem
I added the 'conf.cxx' flags to the ios-ruby-embedded Rakefile (copy of the 'conf.cc' flags)
I tried the enable_cxx_exception and enable_cxx_abi options in the Rakefile, but did not find a combination that solved the problem
I had to add the -lc++ flag to "Other Linker Flags" in the Xcode project
I have posted this issue to the ios-ruby-embedded project and StackOverflow as well

Bug. Dumped data collapsed

When I dumped a aobject which contains several float objects, the data collapsed.

The test code is as follows.

class MarshalTest
  def initialize
    @a = 0.1
    @b = 0.2
    @c = 0.3
  end
end

o = MarshalTest.new
p o

File.open('test.bin', 'wb') do |f|
  Marshal.dump(o, f)
end

File.open('test.bin', 'rb') do |f|
  p Marshal.load(f)
end

The output is as follows.

#<MarshalTest:0x8802a0 @a=0.1, @b=0.2, @c=0.3>
#<MarshalTest:0x8869a0 @a=0.1, @b=0.1, @c=0.3>

MarshalTest.b is collapsed.

A break is missing!

Dear reader:

I found a showstopper in the library. A break is missing around line 242 of marshal.cpp. I had needed to change another couple of things to get the gem to compile. Here below you find the patch. If a so-called pull-request is needed you need to tell me exactly how to proceed...

diff --git a/mrbgem.rake b/mrbgem.rake
index 473bb49..acac331 100644
--- a/mrbgem.rake
+++ b/mrbgem.rake
@@ -3,6 +3,8 @@ MRuby::Gem::Specification.new('mruby-marshal') do |spec|
   spec.author = 'take-cheeze'
   spec.summary = 'Marhshal module for mruby'
 
+  spec.cxx.flags.push('-fpermissive')
+  
   add_dependency 'mruby-onig-regexp', :github => 'mattn/mruby-onig-regexp'
   add_dependency 'mruby-string-ext', :core => 'mruby-string-ext'
 end
diff --git a/src/marshal.cpp b/src/marshal.cpp
index 3fc47ab..168968e 100644
--- a/src/marshal.cpp
+++ b/src/marshal.cpp
@@ -6,6 +6,7 @@
 #include 
 #include 
 
+#include 
 #include 
 #include 
 #include 
@@ -137,7 +138,7 @@ struct write_context : public utility {
 
   bool is_struct(mrb_value const& v) const {
     return mrb_array_p(v) and
-        mrb_const_defined_at(M, mrb_class(M, v), mrb_intern_lit(M, "__members__"));
+      mrb_const_defined_at(M, mrb_obj_value(mrb_class(M, v)), mrb_intern_lit(M, "__members__"));
   }
 
   write_context& link(int const l) {
@@ -242,7 +243,7 @@ write_context& write_context::marshal(mrb_value const& v) {
         char buf[256];
         sprintf(buf, "%.16g", mrb_float(v));
         tag<'f'>().string(buf);
-      }
+      } break; 
 
       case MRB_TT_ARRAY: {
         uclass(v, M->array_class).tag<'['>().fixnum(RARRAY_LEN(v));

Dumped data still collapsed

I ran new test code on Win7 with newest mruby and CRuby(2.5.3p105 (2018-10-18 revision 65156) [i386-mingw32]), and the bug still occurs.

rake test

the result is

Fail: #29 (mrbgems: mruby-marshal)

Assertion[1]
Expected: "o:\x10MarshalTest\b:\a@af\b0.1:\a@bf\b0.2:\a@cf\b0.3"
Actual: "o:\x10MarshalTest\b:\a@af\b0.1:\a@b@\x06:\a@cf\b0.3"

On the other hand, on Ubuntu18.04, the test code passed with no error.

I don't know why this bug occurs on Windows and how to fix it.

Maybe it is related to this issue mruby#4547

Do someone have any ideas to fix this bug?

Missing support for loading from IO handle

The standard Marshal module allows you to load not just from a string but from an IO handle, so that the following should work:

File.open(filename, "rb") { |f|
  @data = Marshal.load(f)
}

The above usage isn't supported by this gem; it would be useful to have.

linker issues

tried to include it into h2o, but its not linking (anymore), probably due to mruby upgrade

Undefined symbols for architecture x86_64:
  "typeinfo for long long", referenced from:
      _mrb_exc_raise in libmruby_core.a(error.cxx.o)
      _mrb_funcall_with_block in libmruby_core.a(vm.cxx.o)
      _mrb_vm_exec in libmruby_core.a(vm.cxx.o)
      GCC_except_table3 in libmruby_core.a(vm.cxx.o)
      GCC_except_table17 in libmruby_core.a(vm.cxx.o)
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
      codegen_error(scope*, char const*) in libmruby_core.a(codegen.cxx.o)
      ...
  "___cxa_allocate_exception", referenced from:
      _mrb_exc_raise in libmruby_core.a(error.cxx.o)
      _mrb_funcall_with_block in libmruby_core.a(vm.cxx.o)
      _mrb_vm_exec in libmruby_core.a(vm.cxx.o)
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
      codegen_error(scope*, char const*) in libmruby_core.a(codegen.cxx.o)
  "___cxa_begin_catch", referenced from:
      _mrb_funcall_with_block in libmruby_core.a(vm.cxx.o)
      _mrb_vm_exec in libmruby_core.a(vm.cxx.o)
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
  "___cxa_end_catch", referenced from:
      _mrb_funcall_with_block in libmruby_core.a(vm.cxx.o)
      _mrb_vm_exec in libmruby_core.a(vm.cxx.o)
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
  "___cxa_guard_acquire", referenced from:
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
      scope_new(mrb_state*, scope*, mrb_ast_node*) in libmruby_core.a(codegen.cxx.o)
  "___cxa_guard_release", referenced from:
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
      scope_new(mrb_state*, scope*, mrb_ast_node*) in libmruby_core.a(codegen.cxx.o)
  "___cxa_throw", referenced from:
      _mrb_exc_raise in libmruby_core.a(error.cxx.o)
      _mrb_funcall_with_block in libmruby_core.a(vm.cxx.o)
      _mrb_vm_exec in libmruby_core.a(vm.cxx.o)
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
      codegen_error(scope*, char const*) in libmruby_core.a(codegen.cxx.o)
  "___gxx_personality_v0", referenced from:
      _mrb_funcall_with_block in libmruby_core.a(vm.cxx.o)
      _mrb_vm_exec in libmruby_core.a(vm.cxx.o)
      Dwarf Exception Unwind Info (__eh_frame) in libmruby_core.a(vm.cxx.o)
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
      Dwarf Exception Unwind Info (__eh_frame) in libmruby_core.a(codegen.cxx.o)
ld: symbol(s) not found for architecture x86_64
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
Undefined symbols for architecture x86_64:
  "typeinfo for long long", referenced from:
      _mrb_exc_raise in libmruby_core.a(error.cxx.o)
      _mrb_funcall_with_block in libmruby_core.a(vm.cxx.o)
      _mrb_vm_exec in libmruby_core.a(vm.cxx.o)
      GCC_except_table3 in libmruby_core.a(vm.cxx.o)
      GCC_except_table17 in libmruby_core.a(vm.cxx.o)
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
      codegen_error(scope*, char const*) in libmruby_core.a(codegen.cxx.o)
      ...
  "___cxa_allocate_exception", referenced from:
      _mrb_exc_raise in libmruby_core.a(error.cxx.o)
      _mrb_funcall_with_block in libmruby_core.a(vm.cxx.o)
      _mrb_vm_exec in libmruby_core.a(vm.cxx.o)
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
      codegen_error(scope*, char const*) in libmruby_core.a(codegen.cxx.o)
  "___cxa_begin_catch", referenced from:
      _mrb_funcall_with_block in libmruby_core.a(vm.cxx.o)
      _mrb_vm_exec in libmruby_core.a(vm.cxx.o)
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
  "___cxa_end_catch", referenced from:
      _mrb_funcall_with_block in libmruby_core.a(vm.cxx.o)
      _mrb_vm_exec in libmruby_core.a(vm.cxx.o)
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
  "___cxa_guard_acquire", referenced from:
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
      scope_new(mrb_state*, scope*, mrb_ast_node*) in libmruby_core.a(codegen.cxx.o)
  "___cxa_guard_release", referenced from:
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
      scope_new(mrb_state*, scope*, mrb_ast_node*) in libmruby_core.a(codegen.cxx.o)
  "___cxa_throw", referenced from:
      _mrb_exc_raise in libmruby_core.a(error.cxx.o)
      _mrb_funcall_with_block in libmruby_core.a(vm.cxx.o)
      _mrb_vm_exec in libmruby_core.a(vm.cxx.o)
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
      codegen_error(scope*, char const*) in libmruby_core.a(codegen.cxx.o)
  "___gxx_personality_v0", referenced from:
      _mrb_funcall_with_block in libmruby_core.a(vm.cxx.o)
      _mrb_vm_exec in libmruby_core.a(vm.cxx.o)
      Dwarf Exception Unwind Info (__eh_frame) in libmruby_core.a(vm.cxx.o)
      _mrb_generate_code in libmruby_core.a(codegen.cxx.o)
      Dwarf Exception Unwind Info (__eh_frame) in libmruby_core.a(codegen.cxx.o)
ld: symbol(s) not found for architecture x86_64
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
rake aborted!
Command Failed: [clang  -o "/Volumes/data/yannick/code/c/h2o-yannick/build-dist/mruby/host/bin/mrbc" "/Volumes/data/yannick/code/c/h2o-yannick/build-dist/mruby/host/mrbgems/mruby-bin-mrbc/tools/mrbc/mrbc.o" "/Volumes/data/yannick/code/c/h2o-yannick/build-dist/mruby/host/lib/libmruby_core.a"  -lm ]

make[2]: *** [CMakeFiles/mruby] Error 1
make[1]: *** [CMakeFiles/mruby.dir/all] Error 2
make: *** [all] Error 2

I throw the towel

Take this simple test script:

t={a:[1.0,2,3,2],b:[4,5,1.0,2,2]}

p t
s=Marshal::dump(t)
p Marshal::load(s)

If you run it without the added break at around line 246, it segfaults (spills from float processing into array processing)

If you run it with the break, but leaving the lines around line 544 intact, you get

mruby: /data/code/motri/mruby/mruby-marshal/src/marshal.cpp:413: mrb_value {anonymous}::read_context::marshal(): Assertion `id < (((struct RArray*)((objects).value.p))->len)' failed.

and if you check, the link for the second floating number should be nr. 2, but there are no values in the "objects" array.

If you use the code that's currently in my fork, the resulting object is wrong. Here is the output:

{:a=>[1.0, 2, 3, 2], :b=>[4, 5, 1.0, 2, 2]}
{:a=>[1.0, 2, 3, 2], :b=>[4, 5, [1.0, 2, 3, 2], 2, 2]}

As you see, the reference to the floating number is misinterpreted.

I have tried several things, but I am out of my wits. I hope the author can step in and fix this.

Replace IO#seek with IO#ungetc

It would be great if you could replace the seek in restore_byte by ungetc. This would make it possible to read directly from STDIN, Sockets, etc (STDIN.seek(-1) #=> RuntimeError). It is also possible to reuse the IO for multiple objects.

For example, with MRI:

ruby -e 'Marshal.dump("foo", STDOUT); Marshal.dump("bar", STDOUT)' | ruby -e 'loop { p Marshal.load(STDIN) }'
"foo"
"bar"
Traceback (most recent call last):
	3: from -e:1:in `<main>'
	2: from -e:1:in `loop'
	1: from -e:1:in `block in <main>'
-e:1:in `load': end of file reached (EOFError)

With this changes below, this would be possible with mruby as well:

mruby/bin/mruby -e 'Marshal.dump("foo", STDOUT); Marshal.dump("bar", STDOUT)' | mruby/bin/mruby -e 'loop { p Marshal.load(STDIN) }'      
"foo"
"bar"
trace (most recent call last):
	[1] -e:1
-e:1: invalid marshal version: 0.0 (expected: 4.8) (TypeError)

I am not sure why MRI raises an EOFError which is what I expected, but mruby continues parsing marshal data.

My changes (sorry, I dont really know any CPP):

diff --git a/src/marshal.cpp b/src/marshal.cpp
index 6024f97..3edd146 100644
--- a/src/marshal.cpp
+++ b/src/marshal.cpp
@@ -480,7 +480,7 @@ mrb_value read_context<In>::marshal() {
 
     case ':': // symbol
     case ';': // symbol link
-      in_.restore_byte(); // restore tag
+      in_.restore_byte(tag); // restore tag
       return mrb_symbol_value(symbol());
 
     case 'I': { // instance variable
@@ -662,7 +662,7 @@ struct string_in {
     return *(current++);
   }
 
-  void restore_byte() { --current; }
+  void restore_byte(uint8_t byte) { --current; }
 
   mrb_value byte_array(size_t len) {
     if((current + len) > end) {
@@ -687,10 +687,12 @@ struct io_in {
     return RSTRING_PTR(buf)[0];
   }
 
-  void restore_byte() {
-    mrb_funcall(M, io, "seek", 2, mrb_fixnum_value(-1),
-                mrb_const_get(M, mrb_obj_value(mrb_class_get(M, "IO")),
-                              mrb_intern_lit(M, "SEEK_CUR")));
+  void restore_byte(uint8_t byte) {
+    if(byte == ';') {
+      mrb_funcall(M, io, "ungetc", 1, mrb_str_new(M, ";", 1));
+    } else {
+      mrb_funcall(M, io, "ungetc", 1, mrb_str_new(M, ":", 1));
+    }
   }
 
   mrb_value byte_array(size_t len) {

Marshal.dump crashing

I have a build of mRuby using this gem that crashes on calls to Marshal.dump.

> o = [:one, :two, :three]
  => [:one, :two, :three]
> m = Marshal.dump(o)
Segmentation fault: 11

It crashes in line 27 of marshal.cpp, right in this code:

bool operator!=(mrb_value const& lhs, mrb_sym const sym) {
  return !mrb_symbol_p(lhs) || mrb_symbol(lhs) != sym;
}

I'm building for iOS and macOS, and the code has been working; I had not tried using dump until now.

Hitting "_load" results in triggering assert

I'm loading classes from files, that are dumped from irb. The classes have _load method.
Upon marshal.load I hit the assert in line #L578.

If I remove the assert everything runs correctly.

Example:
Example class:

Ruby
class A
    def initialize
        @a = 0
    end

    def self._dump(arg = 0)
        [@a].pack('L')
    end

    def self._load(s)
        @a = s.unpack('L')
    end
end

For pack and unpack I'm using iij/mruby-pack, but I don't think the problem is there.

Marshaling of arrays

Since major/minor versions match, I believe this is supposed to be interoperable with Ruby?

puts Marshal.dump(['test']).bytes.inspect

Ruby 2.2.1

[4, 8, 91, 6, 73, 34, 9, 116, 101, 115, 116, 6, 58, 6, 69, 84]

Mruby

[4, 8, 91, 6, 34, 9, 116, 101, 115, 116]

Consequentially, trying to load an array serialized with Ruby in Mruby fails.

Error when built

I got the error when I build it with android toolchain.

mrbgems/mruby-marshal/src/marshal.cpp:10:10: fatal error: 'cassert' file not found #include <cassert> ^ 1 error generated.

Can you recommend the way to solve the error? Thank you.

Marshaling of Structures

Given MyStruct = Struct.new(:field):

Marshal.dump(MyStruct.new) crashes mruby;
Marshal.dump(MyStruct.new([0.0, 0.0])) in ruby, try loading in mruby: Invalid link ID: 2 (table size: 2) (ArgumentError)

Dumping object issue

So, I have the following code:

class B
  def initialize
    @a = 3
  end
end
a = B.new
Marshal.dump(a)
f = Marshal.load(a)

This hits assert(false) in the symbol function after the load function

mruby bytes

> Marshal.dump(B.new).bytes
 => [4, 8, 111, 58, 6, 66, 59, 0, 0]

ruby bytes

> Marshal.dump(B.new).bytes.to_a
=> [4, 8, 111, 58, 6, 66, 6, 58, 7, 64, 97, 105, 8]

I'm currently trying to figure out why this happens, but the Marshal module is new for me. Any help is welcome.

Use khash function for `Hash` keys.

mrb_hash_keys is a slow function.