[proposal] Make an option for lazy-deserialization of complex types about scalapb HOT 3 CLOSED

Thanks for the clear write-up. This seems like a very interesting idea which I'd like us to explore!

Few thoughts:

1. I briefly looked up LazyField in protobuf-java, but I wasn't able to tell when it's being utilized. I can't recall ever seeing it in generated code. I haven't used the lite implementation much and it seems to be related. Do you have more information on the choices made by protobuf-java?
2. There are cases to watch for when the user combines this with other ScalaPB customization (no_box, custom types, custom collection types), and so on. This can lead to additional complexity in generator or cases where the user need to supply their own ByteStringCodec instance via an import option.
3. Methods such as getField and PMessage processing need to wrap/unwrap properly.

In summary, I'm interested in seeing this in PR and landing ultimately as an optional feature, though let's consider the complexity it adds to the code base vs. the benefits once there's a PR.

from scalapb.

I briefly looked up LazyField in protobuf-java, but I wasn't able to tell when it's being utilized. I can't recall ever seeing it in generated code. I haven't used the lite implementation much and it seems to be related. Do you have more information on the choices made by protobuf-java?

Actually, simplest examples are the way they treat String values in generated code:

private java.lang.Object name_ = "";
/**
  * <code>string name = 1;</code>
  * @return The name.
  */
public java.lang.String getName() {
  java.lang.Object ref = name_;
  if (!(ref instanceof java.lang.String)) {
    com.google.protobuf.ByteString bs =
      (com.google.protobuf.ByteString) ref;
    java.lang.String s = bs.toStringUtf8();
    name_ = s;
    return s;
  } else {
    return (java.lang.String) ref;
  }
}

You may note here that they are also throw away the underlying ByteString (which is good for the memory footprint), but I'm not sure if it better to store it for later serialization or not. Anyway, this code looks hacky and follows "at least once" deserializations.

There are cases to watch for when the user combines this with other ScalaPB customization (no_box, custom types, custom collection types), and so on. This can lead to additional complexity in generator or cases where the user need to supply their own ByteStringCodec instance via an import option.

I suppose this is true, but I think it may be relaxed a bit by thinking about specification of types in advance and a 'little bit' of refactoring in generator. I'm on the PoC PR for this issue and the type derivation is one of the most painful places indeed.

Methods such as getField and PMessage processing need to wrap/unwrap properly.

That's not a problem since it will use the same implicit conversion as the regular user code, but in generated context.

In summary, I'm interested in seeing this in PR and landing ultimately as an optional feature, though let's consider the complexity it adds to the code base vs. the benefits once there's a PR.

I'm on it, and I'll try to measure it in benchmarks.
But also there may be another caveats:

1. Lazy deserialization makes any protocol level errors boom on field access instead of parseFrom. Possibly unwrap of such fields may return an Either of error and value, or we just leave this on to user responsibility when they enable such generator option.

from scalapb.

Closing due to inactivity. Feel free to comment if this is still needed.

from scalapb.