gcodeforks / protobuf-for-node Goto Github PK

I'm considering using NodeJS for an application which can't trust the data it 
receives. Invalid data crashing the VM is thus not really something that should 
happen.

Any chance that protobuf-for-node will support exceptions in error-cases in the 
future?

What steps will reproduce the problem?
1. var jane = {
    name: 'Jane Doe',
    id: 1235,
    email: '[email protected]',
    phone: [{ number: "111-2222", type: "HOME"}],
  };

  var serialized = Person.serialize(jane);


What is the expected output? What do you see instead?

I expect:
0a:08:4a:61:6e:65:20:44:6f:65:10:d2:09:1a:11:6a:64:6f:65:32:40:65:78:61:6d:70:6c
:65:2e:63:6f:6d:22:0c:0a:08:31:31:31:2d:32:32:32:32:10:01
(which is what the Java version generates)

I see:
0a:08:4a:61:6e:65:20:44:6f:65:10:d3:09:1a:11:6a:64:6f:65:32:40:65:78:61:6d:70:6c
:65:2e:63:6f:6d:22:0c:0a:08:31:31:31:2d:32:32:32:32:10:01
(the twelth byte is d3 in NodeJS, but d2 in Java)

This causes deserialisation (in Java) to fail, but NodeJS is tolerant of this.

com.google.protobuf.InvalidProtocolBufferException: Protocol message contained 
an invalid tag (zero).finished
    at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:68)
    at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
    at pbtest.AddressBookProtos$Person$Builder.mergeFrom(AddressBookProtos.java:1027)
    at pbtest.AddressBookProtos$Person$Builder.mergeFrom(AddressBookProtos.java:1)
    at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:300)
    at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
    at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:162)
    at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:716)
    at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
    at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:153)
    at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:709)
    at pbtest.AddressBookProtos$Person.parseFrom(AddressBookProtos.java:776)
    at pbtest.Main.main(Main.java:73)


The Java snippet is:

Person john =
              Person.newBuilder()
                .setId(1234)
                .setName("Jane Doe")
                .setEmail("[email protected]")
                .addPhone(
                  Person.PhoneNumber.newBuilder()
                    .setNumber("111-2222")
                    .setType(Person.PhoneType.HOME))
                .build();

        byte[] buf = john.toByteArray();

Would you folks mind adding protobuf-for-node to npm? ( http://npmjs.org )

What steps will reproduce the problem?
1. take a protobuf object with required fields
2. use .serialize() and don't include some of the required fields 
3. note that it creates the object anyway

What is the expected output? What do you see instead?
It should throw an exception or return null.

What version of the product are you using? On what operating system?
current version, OS X

Please provide any additional information below.
I have attached a patch

What steps will reproduce the problem?
1. Compile protoc v2.4.0rc1

What is the expected output? What do you see instead?
Compilation without errors.

This is the error message after generating the classes with protoc:
NODE_PATH=/Users/wizermil/local/lib/node PROTOBUF=/Users/wizermil/local 
node-waf configure clean build
Checking for program g++ or c++          : /usr/bin/g++ 
Checking for program cpp                 : /usr/bin/cpp 
Checking for program ar                  : /usr/bin/ar 
Checking for program ranlib              : /usr/bin/ranlib 
Checking for g++                         : ok  
Checking for node path                   : ok /Users/wizermil/local/lib/node 
Checking for node prefix                 : ok /Users/wizermil/local 
'configure' finished successfully (0.030s)
'clean' finished successfully (0.004s)
Waf: Entering directory 
`/Users/wizermil/Documents/Wizermob/protobuf-for-node/build'
[1/7] cxx: protobuf_for_node.cc -> build/default/protobuf_for_node_1.o
[2/7] cxx: addon.cc -> build/default/addon_2.o
[3/7] cxx: example/protoservice.pb.cc -> 
build/default/example/protoservice.pb_3.o
[4/7] cxx: example/protoservice.cc -> build/default/example/protoservice_3.o
../example/protoservice.cc: In function ‘void init(v8::Handle<v8::Object>)’:
../example/protoservice.cc:54: error: expected class-name before ‘{’ token
../example/protoservice.cc:64: error: cannot convert 
‘init(v8::Handle<v8::Object>)::<anonymous class>*’ to 
‘google::protobuf::Service*’ for argument ‘3’ to ‘void 
protobuf_for_node::ExportService(v8::Handle<v8::Object>, const char*, 
google::protobuf::Service*)’
../example/protoservice.cc:69: error: expected class-name before ‘{’ token
../example/protoservice.cc:90: error: cannot convert 
‘init(v8::Handle<v8::Object>)::<anonymous class>*’ to 
‘google::protobuf::Service*’ for argument ‘3’ to ‘void 
protobuf_for_node::ExportService(v8::Handle<v8::Object>, const char*, 
google::protobuf::Service*)’
Waf: Leaving directory 
`/Users/wizermil/Documents/Wizermob/protobuf-for-node/build'
Build failed:  -> task failed (err #1): 
    {task: cxx protoservice.cc -> protoservice_3.o}


What version of the product are you using? On what operating system?
MacOSX 10.6.5

Please provide any additional information below.
You will find below the patch that will fix the compilation issue with the next 
version of protoc (alias 2.4.0)

protobuf-for-node doesn't function with node v0.3.0, due to some API changes in 
the Buffer object.

A patch to fix these issues is attached (though, of course, this makes things 
fail on older versions of node).

What steps will reproduce the problem?
1. A proto file:
package testpkg;
message dt_in {
    required int32 arg_one = 1;
}

2. The Node.js file
var fs = require('fs');
var puts = require('util').putsw;
var Schema = require('protobuf_for_node').Schema;
var schema = new Schema (fs.readFileSync('./testpkg.desc'));
var dsresult = schema['testpkg.dt_in'];
var t1 = {arg_one : 23};
puts(JSON.stringify(t1,null,2));
var t1_a = dsresult.serialize(t1);
puts(JSON.stringify(t1_a,null,2));
var t2 = dsresult.parse(t1_a);
puts(JSON.stringify(t2,null,2));


3.The output 
{
  "arg_one": 23
}
{
  "length": 0
}
libprotobuf ERROR google/protobuf/message_lite.cc:123] Can't parse message of 
type "pgrpc.dt_in" because it is missing required fields: arg_one

node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
Error: Malformed message
    at Function.parse (unknown source)
    at Object.<anonymous> (/Users/sushil/work/learn/nodejs/c_test6/get_ds2.js:14:19)
    at Module._compile (module.js:402:26)
    at Object..js (module.js:408:10)
    at Module.load (module.js:334:31)
    at Function._load (module.js:293:12)
    at Array.<anonymous> (module.js:421:10)
    at EventEmitter._tickCallback (node.js:126:26)



What version of the product are you using? On what operating system?
I am running on OSX LION

What steps will reproduce the problem?
1. Serialize the proto-buffers using Java and store it as byte string
2. Read the byte string from NodeJS
3. Deserialize the byte String using the Feed.parse() method. Feed is given 
with the required schema assigned with the corresponding descriptor file 
compiled from the proto files. 

sample code snippet:
var Schema = require('protobuf').Schema;
var schema = new Schema(fs.readFileSync('example/test_proto.desc'));
var Feed = schema['proto.root1.root2'];
....
datanode.get("key", function(err, reply) {
   Buffer.prototype.toByteArray = function () {
       return Array.prototype.slice.call(this, 0)
    }

    console.log(reply);
    var buffer = new Buffer(new Buffer(reply).toByteArray()) ;
    aFeed = Feed.parse(buffer);   << error thrown here..


What is the expected output? What do you see instead?
The expected output would be the readable content
sample output:
{
    name: 'abc',
    id: 123,
    phone: [{ number: "111-1111", type: "WORK"}],
  };

Observed Output : 
Error: Malformed message
    at Function.parse (<anonymous>:1:70)
    at ..node_modules/npm/node_modules/protobuf/example/test.js:44:18
    at try_callback (..node_modules/npm/node_modules/redis/index.js:532:9)
    at RedisClient.return_reply (..node_modules/npm/node_modules/redis/index.js:614:13)
    at ReplyParser.<anonymous> (..node_modules/npm/node_modules/redis/index.js:266:14)
    at ReplyParser.EventEmitter.emit (events.js:95:17)
    at ReplyParser.send_reply (..node_modules/npm/node_modules/redis/lib/parser/javascript.js:300:10)
    at ReplyParser.execute (..node_modules/npm/node_modules/redis/lib/parser/javascript.js:203:22)
    at RedisClient.on_data (..node_modules/npm/node_modules/redis/index.js:488:27)
    at Socket.<anonymous> (..node_modules/npm/node_modules/redis/index.js:82:14)


What version of the product are you using? On what operating system?
OS : RHEL 5.4
Protobuf version : 2.5.0
protobuf-for-node : 0.8.6
NodeJS version : v0.10.7

Is anyone having the similar issue ? let me know if I am doing anything 
incorrectly here.. 

Thanks
Shunmugam M

What steps will reproduce the problem?
1.Compile protoc v2.4.1
2.
3.

What is the expected output? What do you see instead?
node.js:201
        throw e; // process.nextTick error, or 'error' event on first tick
              ^
Error: Unable to load shared library 
/var/www/node_modules/protobuf-for-node/build/Release/protobuf_for_node.node
    at Object..node (module.js:463:11)
    at Module.load (module.js:351:31)
    at Function._load (module.js:310:12)
    at Module.require (module.js:357:17)
    at require (module.js:368:17)
    at Object.<anonymous> (/var/www/node_modules/protobuf-for-node/test.js:6:14)
    at Module._compile (module.js:432:26)
    at Object..js (module.js:450:10)
    at Module.load (module.js:351:31)
    at Function._load (module.js:310:12)


What version of the product are you using? On what operating system?
centos 6

Please provide any additional information below.

What steps will reproduce the problem?

1. PROTOBUF=/usr/lib/protobuf node-waf configure clean build


What is the expected output? What do you see instead?

I see:
...
Checking for node prefix                 : ok /usr/local 
'configure' finished successfully (0.023s)
no such environment: default
Traceback (most recent call last):
  File "/usr/local/bin/node-waf", line 16, in <module>
    Scripting.prepare(t, os.getcwd(), VERSION, wafdir)
  File "/usr/local/bin/../lib/node/wafadmin/Scripting.py", line 145, in prepare
    prepare_impl(t, cwd, ver, wafdir)
  File "/usr/local/bin/../lib/node/wafadmin/Scripting.py", line 135, in prepare_impl
    main()
  File "/usr/local/bin/../lib/node/wafadmin/Scripting.py", line 188, in main
    fun(ctx)
  File "/usr/local/bin/../lib/node/wafadmin/Scripting.py", line 283, in clean
    bld.add_subdirs([os.path.split(Utils.g_module.root_path)[0]])
  File "/usr/local/bin/../lib/node/wafadmin/Build.py", line 981, in add_subdirs
    self.recurse(dirs, 'build')
  File "/usr/local/bin/../lib/node/wafadmin/Utils.py", line 634, in recurse
    f(self)
  File "/home/chris/protobuf/protobuf-for-node/wscript", line 39, in build
    obj = bld.new_task_gen('cxx', 'shlib')
  File "/usr/local/bin/../lib/node/wafadmin/Build.py", line 335, in new_task_gen
    ret = cls(*k, **kw)
  File "/usr/local/bin/../lib/node/wafadmin/Tools/ccroot.py", line 162, in __init__
    TaskGen.task_gen.__init__(self, *k, **kw)
  File "/usr/local/bin/../lib/node/wafadmin/TaskGen.py", line 118, in __init__
    self.env = self.bld.env.copy()
AttributeError: 'NoneType' object has no attribute 'copy'


What version of the product are you using? On what operating system?

NodeJS v0.4.12



Please provide any additional information below.

Worked around the issue by editing wscript and adding line at top of 

    def build(bld):
        bld.all_envs['default'] = bld.all_envs['Release']

What steps will reproduce the problem?
1. Install via npm

I see a long list of warnings when building protobuf - 
https://gist.github.com/hanxue/9659783/#file-node-protobuf-builderror-log

Is this an issue, and how can I remove the warnings?

What steps will reproduce the problem?
1. Try to install the software using NPM, either using `npm install 
protobuf-for-node` or `hg clone ... && npm link`

What is the expected output? What do you see instead?
Nothing of the sort happens.

What version of the product are you using? On what operating system?
hg default / Ubuntu 10.04

Please provide any additional information below.
I've hacked a sort-of package.json file, that almost works (turns out neither 
Ubuntu nor Debian ship the required 0.2.4 version of libprotobuf-dev, so it 
doesn't compile for me):

{
    "name": "protobuf-for-node",
    "version": 38,
    "engines": {
        "node": "~0.4"
    },
    "author": {
        "Matthias Ernst": "[email protected]"
    },
    "scripts": {
        "preinstall": "PROTOBUF=/usr/include/google/protobuf node-waf configure",
        "install": "node-waf clean build"
    }
}

What steps will reproduce the problem?
1. Create a field in a message using "bytes".
2. Parse a serialized message containing bytes that are not valid UTF-8.
3. Get malformed data on the bytes field.

What is the expected output? What do you see instead?
I would expect the field to contain a Buffer, not a String.

What version of the product are you using? On what operating system?
Using the latest version of the node3 branch.

I've attached a patch against the latest version of protobuf_for_node.cc to 
provide this functionality, and to encode Buffers to bytes fields as well. 
(Although it still accepts strings, as well.) This patch may (read: probably 
will) break anything that currently uses bytes fields in protobuf-for-node, but 
it seems to be the "correct" solution.

What steps will reproduce the problem?
1. Make a simple protobuf that has a variable like:

package feeds;
message Feed {
  optional string title_string = 1;
}

2. Make the desc file. 
3. Attempt to access it via serialize (and I assume parse)

What is the expected output? What do you see instead?
I'd expect to see a serialized Object, I'm getting an empty object.

What version of the product are you using? On what operating system?
Linux

Please provide any additional information below.

What steps will reproduce the problem?
1. Load a schema using int64/uint64 types
2. Try to parse a message using it

What is the expected output? What do you see instead?

The 64-bit int represented as two 32-bit halves or a string. Instead, I see an 
imprecise double float.

Using the latest libeio (which version do you guys use?) from cvs the call to 
eio_custom in protobuf_for_node.cc:562 uses the callback AsyncInvocation::Run 
which returns int but eio_custom requires it to return null.

Also it should be added to the documentation that libeio is required to build, 
do you expect to use the private version in nodejs or the downloaded version 
from their cvs?