Comments (6)
The reason for the discrepancy is that gob and json (and probably bson) decoders internally do read buffering. The concern with that is that it can read beyond the actual gob and json data in the input stream. Both gob and json point this out explicitly in their docs.
See
http://golang.org/pkg/encoding/gob/#NewDecoder
http://golang.org/pkg/encoding/json/#NewDecoder (json mitigates this somewhat by providing a Buffered method that returns the io.Reader which may have already read past the json data requested).
For my codec library, I wanted to support the ability to have multiple data in an inputstream. E.g. use binc decoder to decode a struct, then read a boundary, then read a gif image right after that.
Folks can pass a buffered reader directly to the NewDecoder call, and get the same benefits of buffering. At this point, they are managing the buffering themselves, and all is well. Binc/Msgpack will not internally read past its data.
For RPC, since the assumption is that the RPC owns the connection, then RPC can create a buffered reader/writer around the connection and pass that to the codec. I've implemented that now, and the numbers are now as you would expect.
BenchmarkEndToEndGob 50000 32450 ns/op
BenchmarkEndToEndMsgPack 50000 34325 ns/op
BenchmarkEndToEndBinc 50000 33911 ns/op
BenchmarkEndToEndJSON 50000 52476 ns/op
It took me a while to figure out because I had to hunt down why gob/json was faster, which is unexpected because the encoding of binc/msgpack is consistently faster.
I will push the change out within the next few hours and close this issue when I am done.
Please let me know if this makes sense.
from go.
Thanks for the detailed response! Yes, the added buffer speeds up MsgPack-RPC and Binc-RPC a lot. Thx!
Here are my results:
pcdummy@ThinkPad-T410 ~/Projekte/golang/src/rpc_test $ GOMAXPROCS=2 go test -test.bench="." bson_bench_test.go
PASS
BenchmarkEndToEndBSON-2 50000 46872 ns/op
BenchmarkEndToEndMsgPack-2 50000 35437 ns/op
BenchmarkEndToEndGob-2 50000 32116 ns/op
BenchmarkEndToEndBinc-2 50000 37007 ns/op
BenchmarkEndToEndJSON-2 50000 49059 ns/op
ok command-line-arguments 12.071s
pcdummy@ThinkPad-T410 ~/Projekte/golang/src/rpc_test $ go test -test.bench="." bson_bench_test.go
PASS
BenchmarkEndToEndBSON 50000 66116 ns/op
BenchmarkEndToEndMsgPack 50000 45132 ns/op
BenchmarkEndToEndGob 50000 39081 ns/op
BenchmarkEndToEndBinc 50000 45951 ns/op
BenchmarkEndToEndJSON 50000 59995 ns/op
ok command-line-arguments 15.419s
from go.
Thanks.
Added one last optimization in the latest update that should narrow the difference further.
Since io.Reader may now be a buffered reader, it is also most likely a ByteReader with a specialized ReadByte() method. I now call ReadByte if available, instead of calling Read([]byte) when needing to read just one byte.
In my testing, it brought the times vs gob to be about the same.
BenchmarkEndToEndGob 50000 32226 ns/op
BenchmarkEndToEndMsgPack 50000 32261 ns/op
BenchmarkEndToEndBinc 50000 31841 ns/op
BenchmarkEndToEndJSON 50000 51524 ns/op
Please test again on your end and let me know if you are seeing the same results.
from go.
Can't go get, getting errors.
pcdummy@ThinkPad-T410 ~/Projekte/golang/src $ go get -u github.com/ugorji/go/codec
github.com/ugorji/go/codec
github.com/ugorji/go/codec/binc.go:28: undefined: encdecHandle
github.com/ugorji/go/codec/binc.go:225: undefined: charEncoding
github.com/ugorji/go/codec/binc.go:295: undefined: charEncoding
github.com/ugorji/go/codec/binc.go:303: undefined: charEncoding
github.com/ugorji/go/codec/decode.go:179: undefined: Handle
github.com/ugorji/go/codec/decode.go:189: undefined: Handle
github.com/ugorji/go/codec/encode.go:44: undefined: charEncoding
github.com/ugorji/go/codec/encode.go:46: undefined: charEncoding
github.com/ugorji/go/codec/encode.go:178: undefined: Handle
github.com/ugorji/go/codec/encode.go:198: undefined: Handle
github.com/ugorji/go/codec/encode.go:198: too many errors
from go.
That is probably a problem with your environment. You will have to do some more digging there. The problem is not in the commit.
Look at the commit description:
5a8bee9
It's all removal of comments, and a few short edits to use ReadByte and io.ReadAtLeast.
from go.
Yes it was a problem at my side.
New Results:
BenchmarkEndToEndBSON 50000 45805 ns/op
BenchmarkEndToEndMsgPack 50000 41245 ns/op
BenchmarkEndToEndGob 50000 39104 ns/op
BenchmarkEndToEndBinc 50000 42863 ns/op
BenchmarkEndToEndJSON 50000 61911 ns/op
Thanks a lot, will use MsgPack at my project!
from go.
Related Issues (20)
- omitempty handling for arrays disagrees with the standard encoding/json HOT 2
- codec.MsgpackHandle.RawToString is confusingly named HOT 2
- Go 1.20 - invalid memory address or nil pointer dereference HOT 5
- Msgpack deserialization with codecgen produces wrong result HOT 3
- Unexpected encoding with interface{} key using Canonical and MapKeyAsString.
- Decoding a slice of structs encoded as arrays into a slice of structs without the same number of fields HOT 2
- Duplicate key inside of a map triggers a "cannot decode unsigned integer" msgpack decode error HOT 1
- Efficient direct conversion between supported formats? HOT 2
- Expose Handle.Name() in some form through Encoder and Decoder for use by codec.Selfer HOT 1
- `io.ErrUnexpectedEOF` should be returned instead of `io.EOF` when only partial data is present
- Add `DecodeOptions.IntType` and `DecodeOptions.FloatType` ? HOT 1
- Docs do not seem to mention case-sensitive field handling
- mem leak? HOT 1
- Tests use rand.Intn(1) for boolean rand, but always returns 0
- Decoder accepts indefinite-length CBOR strings containing chunks with mismatched major types HOT 1
- UTF-8 validation isn't applied to CBOR text strings nested inside an indefinite-length text string HOT 2
- Add support for github.com/go-json-experiment/json / UnmarshalerV2 ? HOT 1
- runtime error with golang master (furture go 1.22)
- Leading zeros is not allowed in rfc7159
- attr.go
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go.