When merging data from the same field (i.e. identical FieldInfo) we can avoid validation checks because the source must already have been validated. This will be especially beneficial for repeated fields.

I believe this error is occurring during serialization of my protobuf object:

Uncaught NoSuchMethodError : method not found: 'Symbol("length")'
Receiver: Instance of 'ByteBuffer'
Arguments: [] 

Potentially related bug: https://code.google.com/p/dart/issues/detail?id=10136

Mailing list discussion: https://groups.google.com/a/dartlang.org/forum/?fromgroups#!topic/misc/cUn_bl4qDnA

This bug was reported from a user:

GeneratedMessage.clearField() is broken for optional fields.

clearField (and consequently generated helpers that use it) set the field to the default value instead of clearing it. As a consequence, the following test fails:

var proto = new ProtoWithOptionalField();
proto.clearTheOptionalField();
expect(proto.hasTheOptionalField(), false); // Boom!

Similar to https://github.com/dart-lang/dart-protobuf/blob/master/lib/src/protobuf/protobuf_enum.dart#L43 , but by name rather than by value.

Or at least provide a pre-built version of the plugin within the DartEditor distribution?

I'm just curious about the design decision to throw in this case. Now you always have to do those checks:

if (foo != null) pbMessage.foo = foo;

Couldn't it simply return instead of throwing when value is null.

Wouldn't it make sense to always call .check() on a GeneratedMessage when it's generated with .fromBuffer() or .fromJson()? This just really got me and I don't quite see why you wouldn't want to run a check...

This is tricky because the "observe" package is experimental and I'd like to avoid hurting performance when it's not used.

Here's my use-case: I get the JSON for a protobuf inside a larger JSON, so I already have it in decoded (Map<String, dynamic>) form.
To deserialize the protobuf with MyGeneratedMessage.fromJson, I need to stringify that map first, then let GeneratedMessage.fromJson parse it again.

It would be nice to be able to feed the decoded JSON to fromJson.

To do this, I suggest adding the following method to both GeneratedMessage and the generated MyGeneratedMessage:
fromDecodedJson(Map<String, dynamic> value, ExtensionRegistry).

Any thoughts?

The following list of strong-mode errors prevent code that depend on package:protobuf from being compiled with DDC.

(ran pub get && dartanalyzer --strong find lib -name '*.dart' | sort -u with SDK version 1.16.0-dev.0.0)

[error] Invalid override. The type of PbList.setRange ((int, int, List<E>, [int]) → void) is not a subtype of List<E>.setRange ((int, int, Iterable<E>, [int]) → void). (./lib/src/protobuf/pb_list.dart, line 90, col 3)
[error] Invalid override. The type of PbList.setRange ((int, int, List<E>, [int]) → void) is not a subtype of ListMixin<E>.setRange ((int, int, Iterable<E>, [int]) → void). (./lib/src/protobuf/pb_list.dart, line 90, col 3)
[hint] The method '_getBaseFieldType' is not used (./lib/src/protobuf/generated_message.dart, line 1029, col 7)
[warning] Unsound implicit cast from List<dynamic> to Iterable<GeneratedMessage> (./lib/src/protobuf/builder_info.dart, line 207, col 48)
[warning] Unsound implicit cast from List<dynamic> to Iterable<int> (./lib/src/protobuf/generated_message.dart, line 292, col 29)
[warning] Unsound implicit cast from List<dynamic> to Iterable<int> (./lib/src/protobuf/generated_message.dart, line 378, col 27)
[warning] Unsound implicit cast from List<dynamic> to Iterable<int> (./lib/src/protobuf/generated_message.dart, line 678, col 27)
[warning] Unsound implicit cast from List<dynamic> to Iterable<int> (./lib/src/protobuf/generated_message.dart, line 708, col 27)
[warning] Unsound implicit cast from List<dynamic> to Iterable<int> (./lib/src/protobuf/unknown_field_set.dart, line 137, col 21)
[warning] Unsound implicit cast from dynamic to List<int> (./lib/src/protobuf/generated_message.dart, line 654, col 42)
[warning] Unsound implicit cast from dynamic to Map<String, dynamic> (./lib/src/protobuf/generated_message.dart, line 844, col 20)

Right now, only tag numbers are supported. It would be great if we would also support using the fieldnames for interoperability with parsing JSON-encoded protobufs from other languages (eg Java or Python). See eg https://github.com/dpp-name/protobuf-json 's output

We are storing both signed and unsigned 64-bit integers in as instances of Int64, even though the Int64 class is intended for signed values. It's up to the caller to treat the the value appropriately.

This is confusing because methods on Int64 such as isNegative() return the wrong thing for numbers from 1<<31 to the top of the range.

To fix this we would need to add another type to the fixnum library:
dart-lang/fixnum#20

The reified type on a PbList is no longer set now that we call GeneratedMessage.createRepeatedField to create lists [1]. It looks like DDC [2] won't work until we put it back.

[1] ca53737
[2] https://github.com/dart-lang/dev_compiler

Code like this has no effect:

var rec = new pb.Rec();
rec.msg.str = "hello";

The reason is that when getField() returns the default value of a message field, it doesn't store it in the map. As a workaround, you can explicitly initialize the message:

var rec = new pb.Rec();
rec.msg = new pb.Msg();
rec.msg.str = "hello";

Protobuf supports "oneof" since 1.6. Currently oneof fields are generated like any other field. In Java there is a getOneofNameCase method that returns which field is set. A method like this is missing for dart. This bug is related to #33

The extensionRegistry argument in CodedBufferReader.readGroup and CodedBufferReader.readMessage is ignored.

It looks like

message.mergeFromCodedBufferReader(this);

in these methods should be changed to

message.mergeFromCodedBufferReader(this, extensionRegistry);

When reading and writing protobufs to JSON, we currently only serialize extensions if they are registered. Unknown fields are ignored.

This is a problem for the JSON-encoded constants for metadata in .pb.dart files. Any extensions in the original protobuf descriptor get stripped. It would be good to encode them somehow, instead.

The decoding worked in Dartium, but when I used Dart2js deserializing a protocol buffer containing signed int32s it threw exceptions for values being out of bounds of signed int32.

Fox32 found the replacement function that works, I have tested and confirmed that it works in Dartium and in Chrome with Dart2js.

http://stackoverflow.com/questions/17427976/bitwise-operations-wrong-result-in-dart2js/17433485#17433485

Can this function replace the _decodeZigZag32 in coded_buffer_reader.dart or will users need to account for this issue themselves?

Rewriting forEachto for(E value in _wrappedList) will produce better dart2js code.
It will be a while before we can inline that.

  int get hashCode {
    int hash = 0;
    _wrappedList.forEach((E value) {
      hash = (hash + value.hashCode) & 0x3fffffff;
      hash = (hash + hash << 10) & 0x3fffffff;
      hash = (hash ^ hash >> 6) & 0x3fffffff;
    });
    hash = (hash + hash << 3) & 0x3fffffff;
    hash = (hash ^ hash >> 11) & 0x3fffffff;
    hash = (hash + hash << 15) & 0x3fffffff;
    return hash;
  }```

PbList and UnknownFieldSet override the equals operator, we should also define the hashCode function for them. Currently dart2js produces hints/warnings mentioning this.

Here is where equals is declared:
https://github.com/dart-lang/dart-protobuf/blob/master/lib/pb_list.dart#L13
https://github.com/dart-lang/dart-protobuf/blob/master/lib/unknown_field_set.dart#L164

I use protobuf entities in polymer elements. For automatic binding / updating those entities must implement the Observable interface.

protoc could automatically implement this interface but a probably easier and less invasive way would be if protoc allowed generated entities to extend from something different than GeneratedMessage.

This would reduce the changes to:

• importing another library
• extending from another class
• passing a symbol to the setField, getField, clearField super functions.
  (This would allow us to use ChangeNotifier)

If you think this is a good /acceptable approach I would try to implement this.

(I currently have a fork where the generated code implements ChangeNotifier in the protoc output and not the super class)

http://stackoverflow.com/questions/25868749/how-do-i-observe-objects-which-do-not-implement-observe-observable-in-polymer

The latest dart editor highlights a couple bugs in the unit tests.

dart:json has been deprecated and the reference is now causing a runtime error. Should be using dart:convert instead.

https://developers.google.com/protocol-buffers/docs/proto#oneof

Generating enum and being able to switch on which field is set (like in Java proto library) would be especially useful.

For some reason, field names in BuilderInfo are converted to lower camelcase from underscores. This makes it harder to deserialize protos generated by other JSON proto libraries. Eg the Java proto to json serializer or https://github.com/google/protobuf.

If there is an reason to keep the modified field name, it would be nice to provide the actual field name in a different field.

In the proto files I have the same message name but in different packages.
The generated Dart code has no indication which Class to instantiate

..a(1100, 'nominatedWorkNominations', GeneratedMessage.OM, () => new Nominations(), () => new Nominations())
..a(1200, 'nomineeNominations', GeneratedMessage.OM, () => new Nominations(), () => new Nominations())

message Property {
optional nominated_work.Nominations NominatedWork_Nominations = 1100;
optional nominee.Nominations Nominee_Nominations = 1200;
}

How can this be resolved?

Most proto RPC I've seen uses some kind of length-prefix encoding for the separation of the messages. Would it be a good place to provide such thing along the protobuf runtime? The following example uses a fixed 4-byte length prefix for each message on the stream.

class PacketPrefixEncoder implements StreamTransformer<List<int>, List<int>> {
  Stream<List<int>> bind(Stream<List<int>> stream) =>
      new Stream.eventTransformed(stream,
          (sink) => new _PacketPrefixEncoderSink(sink));
}

class PacketPrefixDecoder implements StreamTransformer<List<int>, List<int>> {
  Stream<List<int>> bind(Stream<List<int>> stream) =>
      new Stream.eventTransformed(stream,
          (sink) => new _PacketPrefixDecoderSink(sink));
}

class _PacketPrefixEncoderSink implements EventSink<List<int>> {

  final EventSink<List<int>> _outputSink;
  _PacketPrefixEncoderSink(this._outputSink);

  void add(List<int> event) {
    ByteData length = new ByteData(4);
    length.setUint32(0, event.length, Endianness.LITTLE_ENDIAN);
    _outputSink.add(new Uint8List.view(length.buffer, 0, 4));
    _outputSink.add(event);
  }

  void addError(errorEvent, [StackTrace stackTrace]) =>
      _outputSink.addError(errorEvent, stackTrace);

  void close() =>
      _outputSink.close();
}

class _PacketPrefixDecoderSink implements EventSink<List<int>> {

  final EventSink<List<int>> _outputSink;
  _PacketPrefixDecoderSink(this._outputSink);

  BytesBuilder _buffer = null;
  int _length;

  void add(List<int> data) {
    bool added = true;
    while (added) {
      added = false;

      if (_buffer == null) {
        _buffer = new BytesBuilder();
      }
      _buffer.add(data);

      if (_length == null && _buffer.length >= 4) {
        var byteData =
            new ByteData.view(new Uint8List.fromList(_buffer.toBytes()).buffer);
        _length = byteData.getUint32(0, Endianness.LITTLE_ENDIAN);
      }

      if (_length != null && _buffer.length >= _length + 4) {
        Uint8List base = new Uint8List.fromList(_buffer.toBytes());
        Uint8List msg = new Uint8List.view(base.buffer, 4, _length);

        if (_buffer.length == _length + 4) {
          _buffer = null;
        } else {
          var next = new Uint8List.view(base.buffer, 4 + _length);
          _buffer.clear();
          _buffer.add(next);
        }
        _length = null;

        _outputSink.add(msg);
        added = true;
      }
    }
  }

  void addError(errorEvent, [StackTrace stackTrace]) =>
      _outputSink.addError(errorEvent, stackTrace);

  void close() {
    if (_buffer != null) {
      _outputSink.addError(
          "Unprocessed buffer: ${_buffer.length} bytes.");
    }
     _outputSink.close();
  }
}

Assume the protobuffer test.proto:

syntax = "proto2";

message Simple {
  optional string name = 1;
  repeated int32 values = 2;
}

This will fail the '==' test (serialization is ok, however compare seems to differentiate between 'empty list' and 'list never initialized'). I would expect: if object type and serialization is identical, then objects should be identical:

library simple_test;

import 'package:unittest/unittest.dart';
import 'test.pb.dart';

void main() {
  Simple s1 = new Simple()
      ..name = 'Name'
      ;

  Simple s2 = new Simple()
      ..name = 'Name'
      ..values.clear() // redundant
      ;

  test('Json', () => expect(s1.writeToJson(), equals(s2.writeToJson())));
  test('==', () => expect(s1, equals(s2)));   // FAILS!
  test('Buffer', () => expect(s1.writeToBuffer(), equals(s2.writeToBuffer())));
}

For the case where an extension contains a message with a required field and the field's value is missing, it looks like check() will fail (which is correct) but the field won't be listed in the error message.

clone() method is generated by dart protoc plugin for all protos.
GeneratedMessage should declare: GeneratedMessage clone();, and the method should be overriden in generated classes

We would like a way to create read-only GeneratedMessages, using some kind build or freeze API.

An important use case is to storing read-only data in a cache and passing out references to it without corrupting the cache.

Some issues to consider:

• Is it recursive?
• Do we need to freeze in place or is creating a read-only clone acceptable?
• Do we need to unfreeze it?
• Do we want read-only types? That is, for message Foo, should we also have ReadonlyFoo?

I am looking for a way to retrieve all enum values from the fieldInfos.

I can easily get the default value, but the values() function is static and AFAIC there is no easy way (without mirrors) to call this static values() function from the default enum value.

Just filing this as a vote for some TODOs already in the code:

There are still a few getInt64/getUint64 accesses in GeneratedMessage _toString and _toMap. These work fine in Dartium but fail in dart2js, which makes it impossible to use mergeTo/FromJson for compiled code.

We'd really like to use to/fromJson as a standard approach for RPCs, and this is the only blocker. Thanks!

It would be really cool, if the comments added in the .proto files would be added as doctype comments in the generated dart files.

Anything would be helpful. Anything.

Hi! I'm trying to create a structure with a repeated field. At the moment, as far as i can see, the dart file generated by protoc doesn't export setters for repeated fields. Are there plans to add this in the future?

^

when trying to run in javascript, in chrome, an app using protobuf package, an exception occurs in "GeneratedMessage.fromBuffer" (I assume because I do not know how to debug when it is running in compiled javascript).

The exception is:
Uncaught CastError: Casting value of type ByteBuffer to incompatible type List js_helper.dart:1796

I can provide you with a test case if needed, may be it is not expected to use protobuf in javascript compiled program, but then it should be explained somewhere.

Many thanks for this project and for your help,
Adrien

Byte array fields are treated in some ways like repeated fields.

• They are stored in a mutable List
• hasField() returns false if they're empty.
• an empty byte array doesn't change the hashCode (versus null)

In other ways they're treated differently: you can use setField() to replace a byte array.
It seems like the corner cases need to be tested better, particular for events.

The current implementation of hashCode in generated_message.dart hashes the values of the fields of the protobuf – https://github.com/dart-lang/dart-protobuf/blob/f821faf77a3d273255a76ea723a46f390d1a89f8/lib/src/protobuf/generated_message.dart#L291.  This prevents its use as a map key or in a set since a protobuf is mutable.  Perhaps the hashCode getter should not be abused this way and this functionality should be moved into a regular method (e.g. "hash()") to avoid this issue.  In the current state, it only makes sense if the protobuf itself is immutable.

When will this be published to pub.dartlang.org?

For each protobuf class there is a default instance, which should be read-only. Mixins may add extra fields and a way to set them. Any mixin with a setter should throw an exception if the instance is read-only.

Currently the _isReadOnly getter is private. We should make it public. In the meantime, a mixin can implement the check like this:

if (this is ReadonlyMessageMixin) {
  throw new UnsupportedError(
    "attempted to set <somefield> on a read-only message (${info_.messageName})");
}

I am not sure if this feature request is worth it, but I wanted to share the idea, so here it is:

What if we take generated messages and make a constructor that takes as arguments the actual fields from the proto buff? for example:

message A {
  required someField = 1;
  optional otherField = 2;
}

generates:

class A extends GeneratedMessage {
  A.filled(this.someField, {this.otherField});
}

I just often see patterns where developers have all the data they need to fill in, and end up writing code like:

  new A()
      ..someField = a
      ..otherField  = b

instead of:

  new A(a, otherField: b)

Protobuf library: Every generate class that extends GeneratedMessage has a clone method, how about putting clone() into GeneratedMessage?
////////////////////////////////////////////////////////////////////////////////////
Editor: 1.5.0.dev_04_02 (2014-06-06)
OS: Windows 8 - amd64 (6.2)
JVM: 1.7.0_51

projects: 5

open dart files: 2

auto-run pub: true
localhost resolves to: 127.0.0.1
mem max/total/free: 1778 / 1209 / 552 MB
thread count: 36
index: 1063521 relationships in 180908 keys in 1616 sources

SDK installed: true
Dartium installed: true

I think it would be worth to create JSON-Codec and Buffer-Codec for the generated messages, as it would reduce some of the boilerplate in the client's code. Maybe as an option in the generated output? A simple example is the following, the extension could be passed in too:

class _Message__BufferEncoder extends Converter<Message, List<int>> {
  List<int> convert(Message input) =>
      input.writeToBuffer();
}

class _Message__BufferDecoder extends Converter<List<int>, Message> {
  Message convert(List<int> input) =>
      new Message.fromBuffer(input);
}

class Message__BufferCodec extends Codec<Message, List<int>> {
  Converter _decoder = new _Message__BufferDecoder();
  Converter _encoder = new _Message__BufferEncoder();
  Converter<List<int>, Message> get decoder => _decoder;
  Converter<Message, List<int>> get encoder => _encoder;
}

https://github.com/dart-lang/protobuf
should be
https://github.com/dart-lang/dart-protobuf

The following object with a circular dependency throws a Stack Overlfow:

Person child = new Person()
      ..firstName = "Eliott"
      ..lastName = "Garnier Capbert"

Person person = new Person()
    ..firstName = "Nicolas"
    ..lastName = "Garnier"
    ..children.add(child);

child.children.add(person);

I realize circular dependencies are not allowed but a graceful detection and error would be nice as this could happen in real life code and it would be best to have a way to catch these errors.

It looks like this could be triggered by a repeated field that contains protobufs that have a mixin that implements Map. It should check for GeneratedMessage first, then Map.

https://github.com/dart-lang/dart-protobuf/blob/master/lib/src/protobuf/utils.dart#L8