mapbox / pbf Goto Github PK
View Code? Open in Web Editor NEWA low-level, lightweight protocol buffers implementation in JavaScript.
License: BSD 3-Clause "New" or "Revised" License
A low-level, lightweight protocol buffers implementation in JavaScript.
License: BSD 3-Clause "New" or "Revised" License
Might be worth looking at this library:
https://github.com/lemire/simdcomp
and seeing if we can apply that to varint packing (or things like repeated uint32 geometry = 4 [ packed = true ];
).
Example:
message Envelope {
int32 type = 1;
string name = 2;
}
/* ... */
var Envelope = compile(proto).Envelope;
var pbf = new Pbf();
Envelope.write({
type: 'foo',
name: 'test'
});
Envelope.read(new Pbf(pbf.finish()))
Result: Error: Unimplemented type: 4
It seems to be due to val
being being coerced into NaN
. I have not tested if other number types are affected.
A simple solution would be to force it to always be a number:
// Current
val = +val;
// Replacement
val = +val || 0;
But, maybe it would be better to if (isNaN(val)) throw new Error()
?
There are several pbf implementations in JS, so it would help to have a couple sentences in the readme describing the specifics of this particular library (small & low-level, doesn't read proto files etc.)
There are a few things about packed fields the protobuf spec mentions that we are not yet adhering to:
In
proto3
, repeated fields of scalar numeric types use packed encoding by default.
This needs a change in the compiler and is relatively straightforward.
Although there's usually no reason to encode more than one key-value pair for a packed repeated field, encoders must be prepared to accept multiple key-value pairs. In this case, the payloads should be concatenated.
Protocol buffer parsers must be able to parse repeated fields that were compiled as packed as if they were not packed, and vice versa. This permits adding
[packed=true]
to existing fields in a forward- and backward-compatible way.
This is more involved — we would probably need to change all readPacked*
methods to optionally accept an existing array rather than returning a new one, and then the compiler would generate a code like this:
if (tag === 2) pbf.readPackedVarint(obj.tags); // new
if (tag === 2) obj.tags = pbf.readPackedVarint(); // old
Currently we don't treat oneof fields in any special way. Should we? I'm not sure.
There's potential ambiguity after introducing default value handling:
oneof test {
int32 foo = 1;
int32 bar = 2;
}
If either foo
or bar
is set to 0
, we will decode the message as {foo: 0, bar: 0}
. However there will be no way to distinguish which of the fields is set when looking at the result.
One way to address this would be to decode all fields except the set one as undefined
, or simply not to include them in the initial object.
cc @kjvalencik
Can we have a ES module versionimport pbf from 'https://unpkg.com/[email protected]/dist/pbf.mjs?module'
please and a proto compiler --module
flag that can output a module instead of a self
thing. Life is to short to wait for a stable nodejs mjs release :)
Was hoping a bit of CI magic can generate a extra dist mjs automatically
Thanks
It would be beneficial for adding a comment to the top of the generated javascript warning that the code is generated. Perhaps with the version of pbf
that was used as well.
// GENERATED FILE. DO NOT MODIFY. pbf v2.0.1
var pbfPart = pbf.readLazy();
...
pbfPart.readVarint();
There are no methods for reading/writing signed fixed integers.
This initially confused me quite a bit: in the Protobuf language guide, the fixed length integer types are called fixed32
, fixed64
, sfixed32
, sfixed64
, but corresponding pbf
methods are named readUInt32
etc., while uint32
/etc. types in the guide are actually varint.
For a less confusing API, I suggest renaming readUInt32
to readFixed32
, etc.
i use google.protobuf.Any type, need import "google/protobuf/any.proto", when i pbf xxx.proto > xxx.js
Error: ENOENT: no such file or directory, open '........./google/protobuf/any.proto', how i to do? thanks
encode vector tile with pbf x 60.58 ops/sec ±1.17% (65 runs sampled)
native JSON.stringify x 202 ops/sec ±1.99% (88 runs sampled)
The current writing performance is pretty good, but could be much faster. The biggest performance hit comes from the fact that we need to create a new buffer for every embedded message, and reallocate size for each buffer often (and in practice, there are usually lots of messages of different sizes).
Solution I have in mind: change the writeMessage
API so that it accepts a function that writes all the fields, rather than a Pbf object. Then we can write all the fields into the same single buffer, calculate how many bytes we wrote, and shift the data in the buffer by the amount necessary to write a length varint. It should be faster because 1) we don't create new buffers, 2) we don't do as many reallocs — a single global buffer will have a lot less because of the exponential growth.
We can optimize further by reserving single-byte varint, and not shifting data if the message length actually fits into single byte — there should be a lot of cases like this because in practice, small messages appear in the data more often than big messages.
pbf is now super-fast and robust, so now we can easily make a tool for generating pbf encoding/decoding code from a proto file, which will make the library easier to use and more on par with official bindings and tools like protocol-buffers. The tool would be an alternative way to use the library and required through a separate file.
The generated code would look like this and would be usable out of the box.
Somewhat related to #44 since negative numbers are encoded as 64-bit numbers regardless of length.
I'm trying to read a pbf with a packed repeated enum field (as defined here. The compiled pbf parser seems odd for that field:
Relation.read = function (pbf, end) {
return pbf.readFields(Relation._readField, {id: 0, keys: [], vals: [], info: null, roles_sid: [], memids: [], types: 0}, end);
};
Relation._readField = function (tag, obj, pbf) {
[…]
else if (tag === 10) obj.types.push(pbf.readVarint());
};
Shouldn't the default value for the types
field be an empty array []
(instead of 0
), and the data be read via something like pbf.readPackedVarint(obj.types);
(instead of obj.types.push(pbf.readVarint());
)?
Protocol buffer parsers must be able to parse repeated fields that were compiled as packed as if they were not packed, and vice versa. This permits adding [packed=true] to existing fields in a forward- and backward-compatible way.
https://developers.google.com/protocol-buffers/docs/encoding#packed
I.e., the packed flag should only impact encoding and not decoding. This works expected when decoding an unpacked data on a packed field because the type field is validated before reading a packed field.
Lines 397 to 400 in 258c8c1
However, it does not work when reading a packed field as unpacked.
Unimplemented type: 4
This could easily be fixed by using the readPacked*
variant on both packed and unpacked fields. However, this could have performance implications due to the extra function call and while loop.
@mourner Would you like me to put together a benchmark to evaluate the performance implications?
Naming this library "PBF" is a bit confusing in the Geo-Context, because thats what OSMers call its Protobuf-based OSM data format. The German language weekly OSM report already erroneously reported this as a new library to read OSM files.
Currently it's pretty hard to get how all the methods of the library relate to the real Protobuf language spec, unless you know how Protobuf works internally (most people don't) and study example implementation such as vector-tile-js.
So we need 1) better docs and examples; 2) hide all methods that are not supposed to be used directly (by prefixing with _
and not mentioning in the readme)
The conditional use of node's Buffer
versus a shim based on Uint8Array
leads to subtle incompatibilities between node and the browser. I just spent a couple hours debugging an issue in mapbox-gl-js that stems from the following difference:
var Pbf = require('pbf');
var pbf = new Pbf();
pbf.writeVarint(0);
console.log(pbf.finish().buffer.byteLength);
In the browser, this prints 16, as expected. In node, it prints 8192.
To eliminate this gotcha, pbf should use standard Uint8Array
's everywhere, and drop usage of Buffer
.
It would be nice that the generated code would include a comment about it being auto-generated, so that people would not update it manually, but instead change their .proto-files.
Browserified pbf contains a huge Buffer polyfill which isn't actually necessary — we can ditch it since we only use a small subset of Buffer features that are not covered by Uint8Array
.
Hi there,
I'm just trying to work out how to get this module to look directly a vector tile source rather than a local file
So for example this works
var data = fs.readFileSync(__dirname + '/data/25.Pbf');
var tile = new VectorTile(new Protobuf(data));
but this does not
var tile = new VectorTile(new Protobuf('http://localhost:7777/countries/tiles/5/17/25.pbf'));
I gather it's to do with making the xhr request properly, is anyone able to provide any sample code for how to easily do this in node?
Thanks in advance,
Rowan
Moving this to an issue from a bunch of TODOs over the code.
Do we really need to check bounds though? We can't cover all cases where the protobuf being read is invalid/corrupted, throwing a meaningful error all the time, so is it worth implementing? Maybe we should just assume that this library will always read valid Protobuf files. @kkaefer
At the moment the module lacks a round-tripping example as simple as:
const outPbf = new Pbf();
Line.write(kLineToWrite, outPbf);
const outBuf = outPbf.finish();
fs.writeFileSync(kLinePbfFile, outBuf);
const inBuf = fs.readFileSync(kLinePbfFile);
const inPbf = new Pbf(inBuf);
const roundtrip = Line.read(inPbf);
console.log(roundtrip);
for the compiled code.
The readme only shows:
var obj = Test.read(new Pbf(buffer)); // read
var buffer = Test.write(obj, new Pbf()); // write
which misses a concrete example and also the needed finish
call.
/cc @karenzshea @miccolis
To be able to require just the reader or just the writer for even smaller builds and better code organization.
We would need support for older browsers, which are not supported by buffer.js
. There's better support e.g. in the buffer
package in npm. Could pbf
use that instead of buffer.js
?
The concrete issue with Firefox < 30 is that toString
cannot be overridden by buffer.js
, resulting in all strings deserialising as "[object Uint8Array]" instead of the actual data.
Another browser supported by buffer
but not buffer.js
is IE9.
Just a quick heads-up:
Love the library other than that :)
Currently when using pbf with browsers not supporting ArrayBuffer.isView
(such as some versions of IE11), they crash with TypeError: Object doesn't support property or method 'isView'
. We should check for the presence of the function before using it.
Maybe I'm missing something, but I can't get enums to work:
$ cat > example.proto
package figma;
enum ExampleType {
A = 0;
B = 1;
}
message Example {
required ExampleType type = 1;
}
$ ./bin/pbf example.proto > example.js
$ node -e 'pbf = require("./index"), console.log(require("./example").Example.write({type: "A"}, new pbf()))'
pbf/example.js:16
if (example.type !== undefined) pbf.writeMessage(1, writeExampleType, exam
^
ReferenceError: writeExampleType is not defined
at Object.writeExample [as write] (pbf/example.js:16:57)
at [eval]:1:68
at Object.exports.runInThisContext (vm.js:74:17)
at Object.<anonymous> ([eval]-wrapper:6:22)
at Module._compile (module.js:460:26)
at evalScript (node.js:431:25)
at startup (node.js:90:7)
at node.js:814:3
It looks like there's code that should be handling enums, it's just never being called.
Currently the proto compiler doesn't properly resolve types like Feature.Type
. Also, same-name types in different namespaces lead to broken code.
Resolving types properly will add some complexity in the compiler code, but it's necessary.
Here is the complete list supported by protobuf 3: https://developers.google.com/protocol-buffers/docs/proto3#json
I face an issue in compiling proto file with Timestamp field. So I was wondering if I'm using the wrong syntax or it is not supported.
message Signals {
Timestamp start = 2;
repeated sint32 signal = 1;
}
As of now this library handles integers over 2^53 as numbers which causes them to loose precision. The safer way would be to convert them to strings or buffers. Do you plan to add this to this package? In our project we extend your library to handle varint64 in this way, do you need a pull request with the similar implementation?
It seems to me that doing this:
const pbf = new PBF();
Response.write(myobject, pbf);
Is slow because it does a bunch of realloc
internally, i.e. it creates a bigger a Uint8Array
and copy the content inside.
If the length of your messages is similar or constant it seems better to do this:
var currentEncodingLength = 16;
function encode(message) {
const pbf = new PBF(currentEncodingLength);
Response.write(message, pbf);
const result = pbf.finish();
const length = Math.ceil(result.length / 16) * 16;
currentEncodingLength = Math.max(currentEncodingLength, length);
return result;
}
I got a big win by doing this.
Would you be able to provide an example of how I can use this to parse open street maps data and stream the data as it comes in?
Example:
// omitted...
var uint8 = pbf.finish();
console.log('uint8.length:', uint8.length);
console.log('uint8.buffer.length:', uint8.buffer.byteLength);
and the result is:
uint8.length: 79
uint8.buffer.length: 128
The length of uint8 doesn't match the uint8.buffer's. This will result in extra bytes (0) in Node's Buffer
if we use Buffer.from(uint8.buffer)
.
Since upgrading protocol-buffers-schema
, we now get proper parsing of map
fields.
https://developers.google.com/protocol-buffers/docs/proto3#maps
Map fields are backwards compatible because they are identical to a message with key
/ value
fields.
https://developers.google.com/protocol-buffers/docs/proto3#backwards-compatibility
message Envelope {
map<string, string> obj = 1;
}
// Identical
message Envelope {
message FieldEntry1 {
string key = 1;
string value = 2;
}
repeated FieldEntry1 obj = 1;
}
One approach could be:
Generate an embedded _FieldEntryN
message for each map
type. Prefixing with an _
will prevent collisions since this is an invalid message name.
Envelope._FieldEntry1 = {};
Envelope._FieldEntry1.read = function (pbf, end) {
return pbf.readFields(Envelope._FieldEntry1._readField, {key: '', value: ''}, end);
};
Envelope._FieldEntry1._readField = function (tag, obj, pbf) {
if (tag === 1) obj.key = pbf.readString();
else if (tag === 1) obj.value = pbf.readString();
};
Add a method to pbf
for mapping an entry object. This feels wrong. Maybe there's a better place for this. This could also be done inline, but would require a temp variable.
Pbf.prototype = {
/* ... */
mapFieldEntry: function(obj, msg) {
obj[msg.key] = msg.value;
}
};
Add a map
case for reading a field. Similar to repeated fields.
Envelope._readField = function (tag, obj, pbf) {
if (tag === 1) pbf.mapFieldEntry(obj.obj, Envelope._FieldEntry1.read(pbf, pbf.readVarint() + pbf.pos));
};
Add a map
case for writing a field. Similar to repeated, but iterating keys.
Envelope.write = function (obj, pbf) {
if (obj.obj) for (var keys = Object.keys(obj.obj), i = 0; i < keys.length; i++) pbf.writeMessage(3, Envelope._FieldEntry1.write, {key: keys[i], value: obj.obj[keys[i]]});
};
While this is the most straightforward approach with the smallest number of changes, it will perform suboptimally. There are many additional allocations of temporary objects. However, there could later be a follow-up to handle maps as a special case that performs better.
Many of our protobuf objects end up being eventually serialized as JSON. Currently, message fields have default values of null
, which gets serialized into the result.
This is problematic because most specs for defining schema in JSON (e.g., swagger) define optional fields as being omitted from the result. This forces the need to prune null
values before serialization in order to adhere to spec.
What are your thoughts on changing the default value of these types to undefined
? What about a compile time option for deciding?
pbf --default-defined ./thing.proto
message Person {
required string user_name = 1;
optional int64 favourite_number = 2;
message Data {
optional double length = 6;
optional double dbl = 7;
repeated string text = 8;
}
repeated string interests = 3;
repeated bool male = 4;
optional int64 dt = 5;
}
And this the object I'm parsing
var obj = {
"user_name": "Martin ",
"favourite_number": 1337,
"data" : {
"length" : 2,
"text" : ["str1", "str2"]
},
"interests": ["daydreaming", "hacking"],
"male" : true,
"dt" : (new Date("Mon Feb 26 2018 17:42:17 GMT+0530 (IST)")).getTime()
}
But when I parse and parse back
var pbf = new Pbf();
Person.write(obj, pbf);
var buffer = pbf.finish();
// read
var pbf_reader = new Pbf(buffer);
var outputObj = Person.read(pbf_reader);
But the output object is
{
"user_name": "Martin ",
"favourite_number": 1337,
"interests": [
"daydreaming",
"hacking"
],
"male": [],
"dt": 1519647137000
}
Can you please tell me if I'm doing something wrong.
Is anyone here opposed to maybe adding a more user-friendly API to this library.
Right now it takes a fair amount of boilerplate to do what is essentially JSON.parse(encoded)
or JSON.stringify(obj)
.
Plus, there doesn't seem to be a way to get the same objects out on the decode end as were put into the encoding step. This means even more boilerplate code to translate certain fields from numbers into enum strings. Considering this is probably the most common use case it seems worth it to make this a simple one-step call like buffer = encode(schema, obj)
and obj = decode(schema, buffer)
. And preferably, the input to encode and the output of decode should have deep equality.
After updating to pbf 3.x I have issues reading packed values with a tag number greater than 15.
syntax = "proto3";
package ProtobufTest;
message Working {
repeated uint32 values = 15;
}
syntax = "proto3";
package ProtobufTest;
message NotWorking {
repeated uint32 values = 16;
}
The problem seems to be in readPackedEnd which guesses the type based on pbf.pos -1.
function readPackedEnd(pbf) {
return (pbf.buf[pbf.pos - 1] & 0x7) === Pbf.Bytes ?
pbf.readVarint() + pbf.pos : pbf.pos + 1;
}
The issue is that protocol-buffers-schema does not properly handle package
information when importing. Regardless of the package
, messages are always flattened.
This means .proto
files are not exchangeable between javascript and other languages. Example:
syntax = "proto3";
package Foo.Enums;
enum E {
A = 0;
B = 1;
}
syntax = "proto3";
package Foo.Bar;
message Envelope {
.Foo.Enums.E enum = 1;
}
This can be worked around by updating the getType
method to skip namespaces when they aren't found.
Lines 93 to 96 in e0b84b8
function getType(ctx, field) {
var path = field.type.split('.');
return path.reduce(function(ctx, name) { return ctx && ctx[name]; }, ctx);
}
/* Becomes */
function getType(ctx, field) {
var path = field.type.split('.');
return path.reduce(function(ctx, name) { return ctx[name] || ctx; }, ctx);
}
The downside of this work around is that you could end up grabbing the wrong message. However, this could currently happen and right now .proto
files are not interchangeable, so I think this would be a decent stop gap while waiting for an upstream fix.
Thoughts?
Found a critical bug that was not covered with our tests. Investigating.
var str = '{"Feature":"http://example.com/vocab#Feature","datetime":{"@id":"http://www.w3.org/2006/time#inXSDDateTime","@type":"http://www.w3.org/2001/XMLSchema#dateTime"},"when":"http://example.com/vocab#when"}';
var pbf = new Pbf();
pbf.writeString(str);
pbf.pos = 0;
pbf.readString();
// '"Feature":"http://example.com/vocab#Feature","datetime":{"@id":"http://www.w3.org/2006/time#inXSDDateTime","@type":"http://www.w3.org/2001/XMLSchema#dateTime"},"when":"http://example.com/vocab#when"}\u0000'
https://github.com/pierrec/node-eval could be handy for this.
These will always fail.
I try to open a large pbf file (>2Go), but get an error:
var file = 'test.pbf';
var pbf = new Pbf(fs.readFileSync(file));
buffer.js:23
const ui8 = new Uint8Array(size);
^
RangeError: Invalid typed array length
It seems nodejs buffers can't be bigger than 2Go:
nodejs/node#6560 (comment)
So my question:
Is there a way to use a stream instead of buffer as argument to access the file? Or maybe an other way to access file bigger than 2Go?
Thanks.
node version: 4.2.6
There are writeFloat, writeBoolean and readUInt64 methods but no counterparts.
Please, can you add the license of the code. ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.