Currently we don't treat oneof fields in any special way. Should we? I'm not sure.

There's potential ambiguity after introducing default value handling:

oneof test {
  int32 foo = 1;
  int32 bar = 2; 
}

If either foo or bar is set to 0, we will decode the message as {foo: 0, bar: 0}. However there will be no way to distinguish which of the fields is set when looking at the result.

One way to address this would be to decode all fields except the set one as undefined, or simply not to include them in the initial object.

cc @kjvalencik

It would be beneficial for adding a comment to the top of the generated javascript warning that the code is generated. Perhaps with the version of pbf that was used as well.

// GENERATED FILE. DO NOT MODIFY. pbf v2.0.1

Somewhat related to #44 since negative numbers are encoded as 64-bit numbers regardless of length.

I try to open a large pbf file (>2Go), but get an error:

var file = 'test.pbf';
var pbf = new Pbf(fs.readFileSync(file));

buffer.js:23
  const ui8 = new Uint8Array(size);
              ^
RangeError: Invalid typed array length

It seems nodejs buffers can't be bigger than 2Go:
nodejs/node#6560 (comment)

So my question:
Is there a way to use a stream instead of buffer as argument to access the file? Or maybe an other way to access file bigger than 2Go?

Thanks.

node version: 4.2.6

There are several pbf implementations in JS, so it would help to have a couple sentences in the readme describing the specifics of this particular library (small & low-level, doesn't read proto files etc.)

It seems to me that doing this:

const pbf = new PBF();
Response.write(myobject, pbf);

Is slow because it does a bunch of realloc internally, i.e. it creates a bigger a Uint8Array and copy the content inside.

If the length of your messages is similar or constant it seems better to do this:

var currentEncodingLength = 16;
function encode(message) {
  const pbf = new PBF(currentEncodingLength);
  Response.write(message, pbf);
  const result = pbf.finish();
  const length = Math.ceil(result.length / 16) * 16;
  currentEncodingLength = Math.max(currentEncodingLength, length);
  return result;
}

I got a big win by doing this.

The issue is that protocol-buffers-schema does not properly handle package information when importing. Regardless of the package, messages are always flattened.

This means .proto files are not exchangeable between javascript and other languages. Example:

syntax = "proto3";
package Foo.Enums;

enum E {
    A = 0;
    B = 1;
}

syntax = "proto3";
package Foo.Bar;

message Envelope {
    .Foo.Enums.E enum = 1;
}

This can be worked around by updating the getType method to skip namespaces when they aren't found.

function getType(ctx, field) {
    var path = field.type.split('.');
    return path.reduce(function(ctx, name) { return ctx && ctx[name]; }, ctx);
}

/* Becomes */

function getType(ctx, field) {
    var path = field.type.split('.');
    return path.reduce(function(ctx, name) { return ctx[name] || ctx; }, ctx);
}

The downside of this work around is that you could end up grabbing the wrong message. However, this could currently happen and right now .proto files are not interchangeable, so I think this would be a decent stop gap while waiting for an upstream fix.

Thoughts?

We would need support for older browsers, which are not supported by buffer.js. There's better support e.g. in the buffer package in npm. Could pbf use that instead of buffer.js?

The concrete issue with Firefox < 30 is that toString cannot be overridden by buffer.js, resulting in all strings deserialising as "[object Uint8Array]" instead of the actual data.

Another browser supported by buffer but not buffer.js is IE9.

Might be worth looking at this library:

https://github.com/lemire/simdcomp

and seeing if we can apply that to varint packing (or things like repeated uint32 geometry = 4 [ packed = true ];).

The conditional use of node's Buffer versus a shim based on Uint8Array leads to subtle incompatibilities between node and the browser. I just spent a couple hours debugging an issue in mapbox-gl-js that stems from the following difference:

var Pbf = require('pbf');
var pbf = new Pbf();
pbf.writeVarint(0);
console.log(pbf.finish().buffer.byteLength);

In the browser, this prints 16, as expected. In node, it prints 8192.

To eliminate this gotcha, pbf should use standard Uint8Array's everywhere, and drop usage of Buffer.

Maybe I'm missing something, but I can't get enums to work:

$ cat > example.proto
package figma;

enum ExampleType {
  A = 0;
  B = 1;
}

message Example {
  required ExampleType type = 1;
}
$ ./bin/pbf example.proto > example.js
$ node -e 'pbf = require("./index"), console.log(require("./example").Example.write({type: "A"}, new pbf()))'
pbf/example.js:16
    if (example.type !== undefined) pbf.writeMessage(1, writeExampleType, exam
                                                        ^
ReferenceError: writeExampleType is not defined
    at Object.writeExample [as write] (pbf/example.js:16:57)
    at [eval]:1:68
    at Object.exports.runInThisContext (vm.js:74:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:460:26)
    at evalScript (node.js:431:25)
    at startup (node.js:90:7)
    at node.js:814:3

It looks like there's code that should be handling enums, it's just never being called.

Example:

// omitted...
var uint8 = pbf.finish();
console.log('uint8.length:', uint8.length);
console.log('uint8.buffer.length:', uint8.buffer.byteLength);

and the result is:

uint8.length: 79
uint8.buffer.length: 128

The length of uint8 doesn't match the uint8.buffer's. This will result in extra bytes (0) in Node's Buffer if we use Buffer.from(uint8.buffer).

pbf is now super-fast and robust, so now we can easily make a tool for generating pbf encoding/decoding code from a proto file, which will make the library easier to use and more on par with official bindings and tools like protocol-buffers. The tool would be an alternative way to use the library and required through a separate file.

The generated code would look like this and would be usable out of the box.

Would you be able to provide an example of how I can use this to parse open street maps data and stream the data as it comes in?

https://github.com/pierrec/node-eval could be handy for this.

There are writeFloat, writeBoolean and readUInt64 methods but no counterparts.

Please, can you add the license of the code. ?

encode vector tile with pbf x 60.58 ops/sec ±1.17% (65 runs sampled)
native JSON.stringify x 202 ops/sec ±1.99% (88 runs sampled)

The current writing performance is pretty good, but could be much faster. The biggest performance hit comes from the fact that we need to create a new buffer for every embedded message, and reallocate size for each buffer often (and in practice, there are usually lots of messages of different sizes).

Solution I have in mind: change the writeMessage API so that it accepts a function that writes all the fields, rather than a Pbf object. Then we can write all the fields into the same single buffer, calculate how many bytes we wrote, and shift the data in the buffer by the amount necessary to write a length varint. It should be faster because 1) we don't create new buffers, 2) we don't do as many reallocs — a single global buffer will have a lot less because of the exponential growth.

We can optimize further by reserving single-byte varint, and not shifting data if the message length actually fits into single byte — there should be a lot of cases like this because in practice, small messages appear in the data more often than big messages.

It would be nice that the generated code would include a comment about it being auto-generated, so that people would not update it manually, but instead change their .proto-files.

Can we have a ES module versionimport pbf from 'https://unpkg.com/[email protected]/dist/pbf.mjs?module' please and a proto compiler --module flag that can output a module instead of a self thing. Life is to short to wait for a stable nodejs mjs release :)

Was hoping a bit of CI magic can generate a extra dist mjs automatically

Thanks

Proto file

message Person {
    required string user_name        = 1;
    optional int64  favourite_number = 2;
    message Data {
        optional double length = 6;
        optional double dbl = 7;
        repeated string text = 8;
    }
    repeated string interests        = 3; 
    repeated bool male        = 4; 
    optional int64  dt = 5;
}

And this the object I'm parsing

var obj = {
    "user_name": "Martin ",
    "favourite_number": 1337,
    "data" : {
        "length" : 2,
        "text" : ["str1", "str2"]
    },
    "interests": ["daydreaming", "hacking"],
    "male" : true,
    "dt" : (new Date("Mon Feb 26 2018 17:42:17 GMT+0530 (IST)")).getTime()
}

But when I parse and parse back

var pbf = new Pbf();
Person.write(obj, pbf);
var buffer = pbf.finish();

// read
var pbf_reader = new Pbf(buffer);
var outputObj = Person.read(pbf_reader);

But the output object is

{
    "user_name": "Martin ",
    "favourite_number": 1337,
    "interests": [
        "daydreaming",
        "hacking"
    ],
    "male": [],
    "dt": 1519647137000
}

Can you please tell me if I'm doing something wrong.

Protocol buffer parsers must be able to parse repeated fields that were compiled as packed as if they were not packed, and vice versa. This permits adding [packed=true] to existing fields in a forward- and backward-compatible way.

https://developers.google.com/protocol-buffers/docs/encoding#packed

I.e., the packed flag should only impact encoding and not decoding. This works expected when decoding an unpacked data on a packed field because the type field is validated before reading a packed field.

However, it does not work when reading a packed field as unpacked.

Unimplemented type: 4

This could easily be fixed by using the readPacked* variant on both packed and unpacked fields. However, this could have performance implications due to the extra function call and while loop.

@mourner Would you like me to put together a benchmark to evaluate the performance implications?

Currently the proto compiler doesn't properly resolve types like Feature.Type. Also, same-name types in different namespaces lead to broken code.

Resolving types properly will add some complexity in the compiler code, but it's necessary.

After updating to pbf 3.x I have issues reading packed values with a tag number greater than 15.

syntax = "proto3";
package ProtobufTest;

message Working {   
    repeated uint32 values = 15;
}

syntax = "proto3";
package ProtobufTest;

message NotWorking {    
    repeated uint32 values = 16;
}

The problem seems to be in readPackedEnd which guesses the type based on pbf.pos -1.

function readPackedEnd(pbf) {
    return (pbf.buf[pbf.pos - 1] & 0x7) === Pbf.Bytes ?
        pbf.readVarint() + pbf.pos : pbf.pos + 1;
}

As of now this library handles integers over 2^53 as numbers which causes them to loose precision. The safer way would be to convert them to strings or buffers. Do you plan to add this to this package? In our project we extend your library to handle varint64 in this way, do you need a pull request with the similar implementation?

To be able to require just the reader or just the writer for even smaller builds and better code organization.

Since upgrading protocol-buffers-schema, we now get proper parsing of map fields.

https://developers.google.com/protocol-buffers/docs/proto3#maps

Map fields are backwards compatible because they are identical to a message with key / value fields.

https://developers.google.com/protocol-buffers/docs/proto3#backwards-compatibility

message Envelope {
    map<string, string> obj = 1;
}

// Identical

message Envelope {
    message FieldEntry1 {
        string key = 1;
        string value = 2;
    }

    repeated FieldEntry1 obj = 1;
}

One approach could be:

1. Generate an embedded _FieldEntryN message for each map type. Prefixing with an _ will prevent collisions since this is an invalid message name.

   Envelope._FieldEntry1 = {};
   
   Envelope._FieldEntry1.read = function (pbf, end) {
       return pbf.readFields(Envelope._FieldEntry1._readField, {key: '', value: ''}, end);
   };
   
   Envelope._FieldEntry1._readField = function (tag, obj, pbf) {
       if (tag === 1) obj.key = pbf.readString();
       else if (tag === 1) obj.value = pbf.readString();
   };

2. Add a method to pbf for mapping an entry object. This feels wrong. Maybe there's a better place for this. This could also be done inline, but would require a temp variable.

   Pbf.prototype = {
       /* ... */
       mapFieldEntry: function(obj, msg) {
           obj[msg.key] = msg.value;
       }
   };

3. Add a map case for reading a field. Similar to repeated fields.

   Envelope._readField = function (tag, obj, pbf) {
       if (tag === 1) pbf.mapFieldEntry(obj.obj, Envelope._FieldEntry1.read(pbf, pbf.readVarint() + pbf.pos));
   };

4. Add a map case for writing a field. Similar to repeated, but iterating keys.

   Envelope.write = function (obj, pbf) {
       if (obj.obj) for (var keys = Object.keys(obj.obj), i = 0; i < keys.length; i++) pbf.writeMessage(3, Envelope._FieldEntry1.write, {key: keys[i], value: obj.obj[keys[i]]});
   };

While this is the most straightforward approach with the smallest number of changes, it will perform suboptimally. There are many additional allocations of temporary objects. However, there could later be a follow-up to handle maps as a special case that performs better.

Many of our protobuf objects end up being eventually serialized as JSON. Currently, message fields have default values of null, which gets serialized into the result.

This is problematic because most specs for defining schema in JSON (e.g., swagger) define optional fields as being omitted from the result. This forces the need to prune null values before serialization in order to adhere to spec.

What are your thoughts on changing the default value of these types to undefined? What about a compile time option for deciding?

pbf --default-defined ./thing.proto

These will always fail.

There are a few things about packed fields the protobuf spec mentions that we are not yet adhering to:

In proto3, repeated fields of scalar numeric types use packed encoding by default.

This needs a change in the compiler and is relatively straightforward.

Although there's usually no reason to encode more than one key-value pair for a packed repeated field, encoders must be prepared to accept multiple key-value pairs. In this case, the payloads should be concatenated.

Protocol buffer parsers must be able to parse repeated fields that were compiled as packed as if they were not packed, and vice versa. This permits adding [packed=true] to existing fields in a forward- and backward-compatible way.

This is more involved — we would probably need to change all readPacked* methods to optionally accept an existing array rather than returning a new one, and then the compiler would generate a code like this:

if (tag === 2) pbf.readPackedVarint(obj.tags); // new
if (tag === 2) obj.tags = pbf.readPackedVarint(); // old

Currently it's pretty hard to get how all the methods of the library relate to the real Protobuf language spec, unless you know how Protobuf works internally (most people don't) and study example implementation such as vector-tile-js.

So we need 1) better docs and examples; 2) hide all methods that are not supposed to be used directly (by prefixing with _ and not mentioning in the readme)

Just a quick heads-up:

• When using a numbered tag higher than 15, PBF will throw errors. Tags starting at 16 should be used to encode less frequently used fields, as the tag takes up two bytes instead of one.
• When using more than 127 repeatable elements, PBF will throw errors like "Uncaught Error: Unimplemented type: 7" or it will parse the fields incorrectly, seemingly at random, even with the exact same input.

Love the library other than that :)

Hi there,

I'm just trying to work out how to get this module to look directly a vector tile source rather than a local file

So for example this works

var data = fs.readFileSync(__dirname + '/data/25.Pbf');
var tile = new VectorTile(new Protobuf(data));

but this does not

var tile = new VectorTile(new Protobuf('http://localhost:7777/countries/tiles/5/17/25.pbf'));

I gather it's to do with making the xhr request properly, is anyone able to provide any sample code for how to easily do this in node?

Thanks in advance,
Rowan

At the moment the module lacks a round-tripping example as simple as:

const outPbf = new Pbf();
Line.write(kLineToWrite, outPbf);
const outBuf = outPbf.finish();
fs.writeFileSync(kLinePbfFile, outBuf);

const inBuf = fs.readFileSync(kLinePbfFile);
const inPbf = new Pbf(inBuf);
const roundtrip = Line.read(inPbf);
console.log(roundtrip);

for the compiled code.

The readme only shows:

var obj = Test.read(new Pbf(buffer)); // read
var buffer = Test.write(obj, new Pbf()); // write

which misses a concrete example and also the needed finish call.

/cc @karenzshea @miccolis

Example:

message Envelope {
    int32 type = 1;
    string name = 2;
}

/* ... */
var Envelope = compile(proto).Envelope;
var pbf = new Pbf();

Envelope.write({
    type: 'foo',
    name: 'test'
});

Envelope.read(new Pbf(pbf.finish()))

Result: Error: Unimplemented type: 4

It seems to be due to val being being coerced into NaN. I have not tested if other number types are affected.

A simple solution would be to force it to always be a number:

// Current
val = +val;

// Replacement
val = +val || 0;

But, maybe it would be better to if (isNaN(val)) throw new Error()?

Naming this library "PBF" is a bit confusing in the Geo-Context, because thats what OSMers call its Protobuf-based OSM data format. The German language weekly OSM report already erroneously reported this as a new library to read OSM files.

Here is the complete list supported by protobuf 3: https://developers.google.com/protocol-buffers/docs/proto3#json

I face an issue in compiling proto file with Timestamp field. So I was wondering if I'm using the wrong syntax or it is not supported.

message Signals {
    Timestamp start = 2;
    repeated sint32 signal = 1;
}

I'm trying to read a pbf with a packed repeated enum field (as defined here. The compiled pbf parser seems odd for that field:

Relation.read = function (pbf, end) {
    return pbf.readFields(Relation._readField, {id: 0, keys: [], vals: [], info: null, roles_sid: [], memids: [], types: 0}, end);
};
Relation._readField = function (tag, obj, pbf) {
    […]
    else if (tag === 10) obj.types.push(pbf.readVarint());
};

Shouldn't the default value for the types field be an empty array [] (instead of 0), and the data be read via something like pbf.readPackedVarint(obj.types); (instead of obj.types.push(pbf.readVarint());)?

There are no methods for reading/writing signed fixed integers.

Currently when using pbf with browsers not supporting ArrayBuffer.isView (such as some versions of IE11), they crash with TypeError: Object doesn't support property or method 'isView'. We should check for the presence of the function before using it.

Moving this to an issue from a bunch of TODOs over the code.

Do we really need to check bounds though? We can't cover all cases where the protobuf being read is invalid/corrupted, throwing a meaningful error all the time, so is it worth implementing? Maybe we should just assume that this library will always read valid Protobuf files. @kkaefer

This initially confused me quite a bit: in the Protobuf language guide, the fixed length integer types are called fixed32, fixed64, sfixed32, sfixed64, but corresponding pbf methods are named readUInt32 etc., while uint32/etc. types in the guide are actually varint.

For a less confusing API, I suggest renaming readUInt32 to readFixed32, etc.

Is anyone here opposed to maybe adding a more user-friendly API to this library.

Right now it takes a fair amount of boilerplate to do what is essentially JSON.parse(encoded) or JSON.stringify(obj).

Plus, there doesn't seem to be a way to get the same objects out on the decode end as were put into the encoding step. This means even more boilerplate code to translate certain fields from numbers into enum strings. Considering this is probably the most common use case it seems worth it to make this a simple one-step call like buffer = encode(schema, obj) and obj = decode(schema, buffer). And preferably, the input to encode and the output of decode should have deep equality.

Browserified pbf contains a huge Buffer polyfill which isn't actually necessary — we can ditch it since we only use a small subset of Buffer features that are not covered by Uint8Array.

i use google.protobuf.Any type, need import "google/protobuf/any.proto", when i pbf xxx.proto > xxx.js
Error: ENOENT: no such file or directory, open '........./google/protobuf/any.proto', how i to do? thanks

Found a critical bug that was not covered with our tests. Investigating.

var str = '{"Feature":"http://example.com/vocab#Feature","datetime":{"@id":"http://www.w3.org/2006/time#inXSDDateTime","@type":"http://www.w3.org/2001/XMLSchema#dateTime"},"when":"http://example.com/vocab#when"}';
var pbf = new Pbf();
pbf.writeString(str);
pbf.pos = 0;
pbf.readString();
// '"Feature":"http://example.com/vocab#Feature","datetime":{"@id":"http://www.w3.org/2006/time#inXSDDateTime","@type":"http://www.w3.org/2001/XMLSchema#dateTime"},"when":"http://example.com/vocab#when"}\u0000'