Comments (7)
There are edge cases where a message can shrink between populating the size cache and encoding it, which would result in Go Protobuf producing invalid wire format. This would be a terrible outcome, so we’ll prevent producing invalid wire format by adding checks to Go Protobuf’s encoder that detect the situation and error out instead.
This check was added in commit protocolbuffers/protobuf-go@94bb78c
For readers from the future: If you see this error triggering (e.g. size mismatch (see https://github.com/golang/protobuf/issues/1609): calculated=36, measured=37
), the most likely reason for it is accidental sharing and concurrent mutation of a Protobuf message or submessage.
We fixed one case where a submessage was shared (defined in the server struct) between requests, but then each request handler would also modify that same submessage with per-request data. When you concurrently process requests, this results in modifying a protobuf message while marshaling (encoding) it, which is not permitted.
Instead of producing invalid wire format, Go Protobuf will error out. The fix is to change your code to not mutate Protobuf messages while marshaling (encoding).
from protobuf.
Hey @stapelberg. Thank you for your tips. I finally found that some inner messages were mutated in an external library and fixed that.
from protobuf.
After running into this error, I feel like I don't know anything about how a program should work at all. It is so frustrating. My service was working perfectly and after updating it's protobuf version, some rpcs cannot handle concurrent requests anymore. By emptying some random inner message which I'm hundred percent sure won't be mutated during runtime, the errors disappear. I don't remember that I have read anywhere that one should not reuse protobuf messages.
from protobuf.
Hey Mehdi. Sorry for the trouble you’re running into, I can understand it can be frustrating.
I have investigated a small handful of different programs running into this error, and in 100% of the cases it was a race / concurrency issue. They can be hard to spot, but in the end we always found the issue. I think most likely you’re also dealing with a hard-to-spot concurrency issue.
If the program you’re working on is Open Source and you could share a core dump (you can add a panic to internal/errors to get a good crash), I can take a look — often, the stacktrace already provides enough information. If the stacktrace isn’t sufficient, you can look at the output buffer to see how many bytes it contains and thereby identify where in the message hierarchy the problem is located.
Note that this new check prevents invalid wire format, so before this check, your program would occasionally produce invalid data. I think the new failure mode of erroring is better than silently producing corrupt data.
from protobuf.
Related Issues (20)
- How to properly define protobuf `oneof` fields? HOT 2
- How can I import a pre-compiled proto from another package? HOT 4
- protodesc: feature "message_encoding" (in Protobuf editions) triggers proto2 group validation checks HOT 11
- timestamppb.AsTime is incorrectly implemented for nil/zero values HOT 2
- missing replace in go.mod HOT 1
- protojson: fail to unmarshal JSON with reserved fields HOT 19
- Parsing protoc plugin option containing ',' HOT 5
- go_features.proto has incorrect package and error-prone file path HOT 5
- feature: Generate test client / `bufconn` constructors HOT 4
- v1.33.0 breaks bazel build HOT 14
- Reflection: protopath parsing and path+message access HOT 7
- Proposal: Set go.mod Go language version to oldest supported Go version (1.21 currently) HOT 8
- Proposal: Add `Override` mode to `proto.Merge` HOT 2
- editions: maps are incorrectly serialized if file has default message encoding of DELIMITED HOT 8
- protodesc: too strict about proto3 field names when creating descriptors from FileDescriptorProto
- undefined: descriptorpb.Default_FileOptions_PhpGenericServices HOT 2
- `ProtoReflect().GetUnknown()` Not Working HOT 4
- Editions language feature: (pb.go).struct_tag HOT 11
- encoding/protojson: handling google.protobuf.Empty HOT 15
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from protobuf.