Tracking issue: Fully Lazy Extension decoding about protobuf HOT 7 CLOSED

There are edge cases where a message can shrink between populating the size cache and encoding it, which would result in Go Protobuf producing invalid wire format. This would be a terrible outcome, so we’ll prevent producing invalid wire format by adding checks to Go Protobuf’s encoder that detect the situation and error out instead.

This check was added in commit protocolbuffers/protobuf-go@94bb78c

For readers from the future: If you see this error triggering (e.g. size mismatch (see https://github.com/golang/protobuf/issues/1609): calculated=36, measured=37), the most likely reason for it is accidental sharing and concurrent mutation of a Protobuf message or submessage.

We fixed one case where a submessage was shared (defined in the server struct) between requests, but then each request handler would also modify that same submessage with per-request data. When you concurrently process requests, this results in modifying a protobuf message while marshaling (encoding) it, which is not permitted.

Instead of producing invalid wire format, Go Protobuf will error out. The fix is to change your code to not mutate Protobuf messages while marshaling (encoding).

from protobuf.

Hey @stapelberg. Thank you for your tips. I finally found that some inner messages were mutated in an external library and fixed that.

from protobuf.

After running into this error, I feel like I don't know anything about how a program should work at all. It is so frustrating. My service was working perfectly and after updating it's protobuf version, some rpcs cannot handle concurrent requests anymore. By emptying some random inner message which I'm hundred percent sure won't be mutated during runtime, the errors disappear. I don't remember that I have read anywhere that one should not reuse protobuf messages.

from protobuf.

Hey Mehdi. Sorry for the trouble you’re running into, I can understand it can be frustrating.

I have investigated a small handful of different programs running into this error, and in 100% of the cases it was a race / concurrency issue. They can be hard to spot, but in the end we always found the issue. I think most likely you’re also dealing with a hard-to-spot concurrency issue.

If the program you’re working on is Open Source and you could share a core dump (you can add a panic to internal/errors to get a good crash), I can take a look — often, the stacktrace already provides enough information. If the stacktrace isn’t sufficient, you can look at the output buffer to see how many bytes it contains and thereby identify where in the message hierarchy the problem is located.

Note that this new check prevents invalid wire format, so before this check, your program would occasionally produce invalid data. I think the new failure mode of erroring is better than silently producing corrupt data.

from protobuf.