macks / ruby-protobuf Goto Github PK

hi macks,

i've recognized, that ruby-protobuf needs about 58 secs to unserialize a message with one binary filed that has 100mb of data.

unserializing the same data with python takes around 1 second.

$ time ruby binary.rb 

real    0m58.321s
user    0m51.354s
sys     0m1.575s

$ time python binary.py 

real    0m0.817s
user    0m0.248s
sys     0m0.390s

this is the protocol i'm using:

package foo.bar;
message Message {
    required bytes data = 1;
}

let me know if i can help you by providing more data.

thanks in advance,
marc

I am using ruby_protobuf version 0.4.11 and ruby 2.0.0p247.

Given a proto file:

message Foo {
}

message Bar {
    optional Foo f = 1;
}

Foo.merge_from() will fail if the target's f field is unset:

$ cat foo.rb
require './foo.pb'

b1 = Bar.new
b2 = Bar.new
b2.f = Foo.new
b1.merge_from(b2)

$ ruby foo.rb
/localhome/aogail/.rvm/gems/ruby-2.0.0-p247/gems/ruby_protobuf-0.4.11/lib/protobuf/message/field.rb:622:in `merge_value': undefined method `merge_from' for nil:NilClass (NoMethodError)
    from /localhome/aogail/.rvm/gems/ruby-2.0.0-p247/gems/ruby_protobuf-0.4.11/lib/protobuf/message/field.rb:104:in `merge'
    from /localhome/aogail/.rvm/gems/ruby-2.0.0-p247/gems/ruby_protobuf-0.4.11/lib/protobuf/message/message.rb:256:in `block in merge_from'
    from /localhome/aogail/.rvm/gems/ruby-2.0.0-p247/gems/ruby_protobuf-0.4.11/lib/protobuf/message/message.rb:256:in `each'
    from /localhome/aogail/.rvm/gems/ruby-2.0.0-p247/gems/ruby_protobuf-0.4.11/lib/protobuf/message/message.rb:256:in `merge_from'
    from foo.rb:6:in `<main>'

When I try to compile the proto files from https://github.com/codeweaver/proto/tree/master/definitions I get

proto.y:156:in `_reduce_124': undefined method `unshift' for :bnet:Symbol (NoMethodError)
    from proto.y:208:in `scan'
    from racc/parser.rb:152:in `_racc_yyparse_c'
    from racc/parser.rb:152:in `yyparse'
    from proto.y:175:in `parse'
    from /Users/pcasaretto/.rvm/gems/ruby-1.9.2-p290/gems/ruby_protobuf-0.4.11/lib/protobuf/compiler/compiler.rb:23:in `block in create_message'
    from /Users/pcasaretto/.rvm/gems/ruby-1.9.2-p290/gems/ruby_protobuf-0.4.11/lib/protobuf/compiler/compiler.rb:22:in `open'
    from /Users/pcasaretto/.rvm/gems/ruby-1.9.2-p290/gems/ruby_protobuf-0.4.11/lib/protobuf/compiler/compiler.rb:22:in `create_message'
    from /Users/pcasaretto/.rvm/gems/ruby-1.9.2-p290/gems/ruby_protobuf-0.4.11/lib/protobuf/compiler/compiler.rb:13:in `compile'
    from /Users/pcasaretto/.rvm/gems/ruby-1.9.2-p290/gems/ruby_protobuf-0.4.11/lib/protobuf/compiler/compiler.rb:9:in `compile'
    from /Users/pcasaretto/.rvm/gems/ruby-1.9.2-p290/gems/ruby_protobuf-0.4.11/bin/rprotoc:40:in `<top (required)>'
    from /Users/pcasaretto/.rvm/gems/ruby-1.9.2-p290/bin/rprotoc:19:in `load'
    from /Users/pcasaretto/.rvm/gems/ruby-1.9.2-p290/bin/rprotoc:19:in `<main>'

protoc compiles these just fine.
It seems the problem is with the .bnet. and .D3. tags . If I remove them I get

/Users/pcasaretto/.rvm/gems/ruby-1.9.2-p290/gems/ruby_protobuf-0.4.11/lib/protobuf/message/message.rb:43:in `define_field': Field tag 1 has already been used in Google::Protobuf::FileDescriptorProto. (Protobuf::TagCollisionError)

In the .proto file, the package name cannot be 'message', otherwise, it throws parsing error

racc/parser.rb:347:in on_error': (Racc::ParseError) parse error on value :message ("message") from proto.y:194:inscan'
from racc/parser.rb:152:in _racc_yyparse_c' from racc/parser.rb:152:inyyparse'
from proto.y:175:in parse' from /Users/hzr/.rvm/gems/ruby-1.9.2-p180@goldfish/gems/ruby_protobuf-0.4.11/lib/protobuf/compiler/compiler.rb:23:inblock in create_message'
from /Users/hzr/.rvm/gems/ruby-1.9.2-p180@goldfish/gems/ruby_protobuf-0.4.11/lib/protobuf/compiler/compiler.rb:22:in open' from /Users/hzr/.rvm/gems/ruby-1.9.2-p180@goldfish/gems/ruby_protobuf-0.4.11/lib/protobuf/compiler/compiler.rb:22:increate_message'
from /Users/hzr/.rvm/gems/ruby-1.9.2-p180@goldfish/gems/ruby_protobuf-0.4.11/lib/protobuf/compiler/compiler.rb:13:in compile' from /Users/hzr/.rvm/gems/ruby-1.9.2-p180@goldfish/gems/ruby_protobuf-0.4.11/lib/protobuf/compiler/compiler.rb:9:incompile'
from /Users/hzr/.rvm/gems/ruby-1.9.2-p180@goldfish/gems/ruby_protobuf-0.4.11/bin/rprotoc:40:in <top (required)>' from /Users/hzr/.rvm/gems/ruby-1.9.2-p180@goldfish/bin/rprotoc:19:inload'
from /Users/hzr/.rvm/gems/ruby-1.9.2-p180@goldfish/bin/rprotoc:19:in `

In lib/protobuf/message/field.rb, there's some constants set up like this:

      INT32_MAX  =  2**31 - 1
      INT32_MIN  = -2**31
      INT64_MAX  =  2**63 - 1
      INT64_MIN  = -2**63
      UINT32_MAX =  2**32 - 1
      UINT64_MAX =  2**64 - 1

The problem is that with OS X's ruby (on Mountain Lion, anyway), we get this:

$ /usr/bin/ruby -e 'puts 2**63'
-9223372036854775808

Yep. Its ruby can't do math good. The ruby --version is ruby 1.8.7 (2012-02-08 patchlevel 358) [universal-darwin12.0].

It would be very nice if all these expressions were replaced by hard-coded integer literals.

We are using for our protobuf service layer a java server and the ruby app we're running is the client. When doing a request from the ruby client, calling the java RPC service, the java service returns bad request data. We also get a bad stack trace on the ruby side, saying invalid value: nil
Doing some grep-ing it is happening in this file: lib/protobuf/message/field.rb on 691

My ruby env is Ruby 1.9.1p243 (2009-07-16 revision 24175) [x86_64-linux]
We're using rprotoc 0.4.0, which we know is the experimental branch.
Unfortunately rprotoc 0.3.3 doesn't work with ruby 1.9.1 (having TCPSocket readbyte issues - invalid number of params)

Sample Ruby code:
rpc_request = Cerebro::UserAuthorizationParams.new
rpc_request.email = params[:email]
rpc_request.hash = Digest::SHA1.hexdigest(params[:password])
rpc_response = Cerebro::User.new

rpc_client = Protobuf::Rpc::Client.new('localhost', 7575)
rpc_client.call(:authorize , rpc_request , rpc_response) # throws the error of "TypeError - Invalid value: nil"

Java protobuf is reporting:
com.googlecode.protobuf.socketrpc.SocketRpcServer$Handler handleError
WARNING: BAD_REQUEST_DATA: Bad request data from client
com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length.
at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:49)
at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:691)
at com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:756)
at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:274)
at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:484)
at com.google.protobuf.GeneratedMessage$Builder.parseUnknownField(GeneratedMessage.java:271)
at com.googlecode.protobuf.socketrpc.SocketRpcProtos$Request$Builder.mergeFrom(SocketRpcProtos.java:371)
at com.googlecode.protobuf.socketrpc.SocketRpcProtos$Request$Builder.mergeFrom(SocketRpcProtos.java:1)
at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:244)
at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:182)
at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:202)
at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:683)
at com.googlecode.protobuf.socketrpc.SocketRpcServer$Handler.callMethod(SocketRpcServer.java:160)
at com.googlecode.protobuf.socketrpc.SocketRpcServer$Handler.run(SocketRpcServer.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

See ruby-protobuf/protobuf#16 for description and partial fixes. (Cross referencing.)

Calling .inspect on a message crashes.

NoMethodError: undefined method `name' for #
18:13:57.917176 15969 qanat.rb:72: /home/onespot/.bundle/ruby/1.9.1/gems/ruby_protobuf-0.4.0/lib/protobuf/message/message.rb:349:in `block in each_field'
/home/onespot/.bundle/ruby/1.9.1/gems/ruby_protobuf-0.4.0/lib/protobuf/message/message.rb:348:in `each'
/home/onespot/.bundle/ruby/1.9.1/gems/ruby_protobuf-0.4.0/lib/protobuf/message/message.rb:348:in `each_field'
/home/onespot/.bundle/ruby/1.9.1/gems/ruby_protobuf-0.4.0/lib/protobuf/message/message.rb:221:in `inspect'
/home/onespot/.bundle/ruby/1.9.1/gems/ruby_protobuf-0.4.0/lib/protobuf/message/message.rb:201:in `block in inspect'
/home/onespot/.bundle/ruby/1.9.1/gems/ruby_protobuf-0.4.0/lib/protobuf/message/message.rb:227:in `call'
/home/onespot/.bundle/ruby/1.9.1/gems/ruby_protobuf-0.4.0/lib/protobuf/message/message.rb:227:in `block in inspect'
/home/onespot/.bundle/ruby/1.9.1/gems/ruby_protobuf-0.4.0/lib/protobuf/message/message.rb:350:in `block in each_field'
/home/onespot/.bundle/ruby/1.9.1/gems/ruby_protobuf-0.4.0/lib/protobuf/message/message.rb:348:in `each'
/home/onespot/.bundle/ruby/1.9.1/gems/ruby_protobuf-0.4.0/lib/protobuf/message/message.rb:348:in `each_field'
/home/onespot/.bundle/ruby/1.9.1/gems/ruby_protobuf-0.4.0/lib/protobuf/message/message.rb:221:in `inspect'

Hi Macks, I mentioned an issue in my pull request. This error has been present for a while but is really pronounced in the eval() done to verify that the file created is able to be loaded. Basically the protobuf classes are not tolerant of being re-required.

http://gist.github.com/407742

What I think it should do is probably throw away the old definition if it is re-required but I wanted to get your opinion and input on the cleanest way to do that rather than go in with a machete myself.

I have a Ruby IDE for make game as client.Now I want to set up a server in java,After this,I want it can use google's protobuf to be a standard for communication?
How should I do?
help please.
Thanks a lot.

Many .proto files I'm trying to compile fail with:

`accept_message_visitor': NotImplementedError (NotImplementedError)

Looking in the code, it seems like this was just left unfinished....?

For an example .proto see

http://code.google.com/p/android-market-api/source/browse/trunk/AndroidMarketApi/proto/market.proto

module Com
  module Largesmall
    module Crawler
      module Model
        module Protobuf
          ::Protobuf::OPTIONS[:"java_package"] = "com.largesmall.crawler.model.protobuf"
          class PagePageInlink < ::Protobuf::Message
            defined_in __FILE__
            required :'com::largesmall::crawler::model::protobuf::PagePageLinkMetadata', :metadata, 1
          end

Notice the module is Com::Largesmall::... but the symbol is "com::largesmall::...". The case is wrong.

I have written a ruby client which makes a REST call to a java server using the AddressBook example. When the protobuf in being deserialized by java, I am getting the following exception even though 'id' property was set in ruby client:

: com.google.protobuf.UninitializedMessageException: Message missing required fields: id
at com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:550)
at com.symantec.AddressBookProtos$Person$Builder.build(AddressBookProtos.java:651)
at com.symantec.AddressBookProtos$Person$Builder.build(AddressBookProtos.java:1)
at com.symantec.ProtobufMessageBodyReader.readFrom(ProtobufMessageBodyReader.java:36)
... 39 more

For debugging purposes, I also serialized the same buffer in java and I realized that there is a small difference between the bytes serialized by ruby and the bytes serialized by java. The ruby generated bytes have an additional byte (2D) at the beginning and the rest is the same. Could this be the cause of the failure during deserialization?

Disregard this issue. It was made on the wrong project...