Comments (5)
proto.Size
will return the size of the wire-encoding of a message.
In the case of an RPC service accepting size-limited requests, the right place to put the size check is before you've parsed the request. If you're calling proto.Size
on a value you read from the network, it's already too late; you already spent the time and memory on decoding the value.
You should also be aware that the in-memory representation of a parsed message can be substantially larger than the on-wire representation of the message. Exactly how much more depends a lot on the structure of the message. If you're worried about malicious inputs, it's worth testing edge cases (for example: a message containing a deeply-nested set of map values) to ensure that your system behaves acceptably.
from protobuf.
@neild when you say
In the case of an RPC service accepting size-limited requests, the right place to put the size check is before you've parsed the request.
I'm assuming what you mean here is that by the time the RPC handler, which implements the underlying gRPC service definition, receives the request struct, then you've already fully decoded the message into memory. Is that correct? And if so, to my understanding, the only way to control that at the server layer is to limit the maximum message size the server will receive (e.g. https://pkg.go.dev/google.golang.org/grpc#MaxRecvMsgSize).
So checking a size constraint on the value of the context input within the RPC handler is a bit of a superfluous check since you've already afforded the decoding step and it's already in-memory. Any further size calculations would just incur even further impact to the server.
from protobuf.
I realize the conversation above is more specific to gRPC than protobuf, but I was also wondering at the proto layer if there is a more efficient way to check the size as compared to proto.Marshal
, and it seems that proto.Size
is for sure the better way. I also benchmarked this and it is noticeably faster.
Thanks for entertaining the more gRPC related question(s) as well 👍
from protobuf.
I'm assuming what you mean here is that by the time the RPC handler, which implements the underlying gRPC service definition, receives the request struct, then you've already fully decoded the message into memory. Is that correct?
Yes, exactly. If you're trying to reject messages that are too big, you want to do it before you invest a lot of effort in parsing the message.
In particular, if your definition of "too big" is proto.Size
, then you really should be applying that limit before decoding. You know how large the encoded message is (because you have the encoded message bytes right there), so decoding it only to then spend more effort computing the encoded size is a very long way to go to get information you already had.
from protobuf.
This question looks answered, so I’ll close.
from protobuf.
Related Issues (20)
- Support for optional fields HOT 2
- Latest version of google.golang.org/protobuf (v1.33.0) is no longer compatible with the latest version of github.com/golang/protobuf (v1.5.3) HOT 7
- Add option to JSON marshal / unmarshal: StripEnumPrefix / AddEnumPrefix HOT 10
- How to properly define protobuf `oneof` fields? HOT 2
- How can I import a pre-compiled proto from another package? HOT 4
- protodesc: feature "message_encoding" (in Protobuf editions) triggers proto2 group validation checks HOT 11
- timestamppb.AsTime is incorrectly implemented for nil/zero values HOT 2
- missing replace in go.mod HOT 1
- protojson: fail to unmarshal JSON with reserved fields HOT 19
- Parsing protoc plugin option containing ',' HOT 5
- go_features.proto has incorrect package and error-prone file path HOT 5
- Tracking issue: Fully Lazy Extension decoding HOT 7
- feature: Generate test client / `bufconn` constructors HOT 4
- v1.33.0 breaks bazel build HOT 14
- Reflection: protopath parsing and path+message access HOT 7
- Proposal: Set go.mod Go language version to oldest supported Go version (1.21 currently) HOT 8
- Proposal: Add `Override` mode to `proto.Merge` HOT 2
- editions: maps are incorrectly serialized if file has default message encoding of DELIMITED HOT 8
- protodesc: too strict about proto3 field names when creating descriptors from FileDescriptorProto
- undefined: descriptorpb.Default_FileOptions_PhpGenericServices HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from protobuf.