Comments (13)
Why do you believe a streaming RPC is necessary? What use case would it solve? And how would you expect it to look like?
from go.
A typical use-case is retrieving 100s or 1000s of records of data from a database using SQL, in a loop. We may not know how many rows are going to be retrieved nor we would like to store all the records in memory and do the marshalling of the whole data, which may not be memory efficient.
It would be desirable to retrieve each row and marshall it out immediately in a loop, to emit data as a continuous stream
While data can be streamed out in a loop, provision to emit a header at the beginning and a footer at the end of stream is desirable.
Example:
var data = [ // header
{"a"=1, "b"=2}, // row 1
......
{"a"=3, "b"=4}, // row n
]; // footer
from go.
This was why I asked.
You can already do this.
So I'd have to see a real use-case that shows that this isn't possible.
In some C++ libraries, they have different requirements, because they try to read the data, store it into a memory container (ie byte array, etc), and they decode that. So they need a separate streaming API, and separate ASYNC api, etc. However, Go library allows you decode directly from any inputstream (anything implementing io.Reader), and async is implemented under-the-hood and exposed via a sync API.
In theory, you can have a stream with 200 entities encoded in sequence, and create an encoder, and keep decoding from the stream until the stream is finished.
That's one reason why none of the Go encoding libraries (JSON, Gob, Msgpack, etc) have a separate streaming API, or net/rpc has a streaming API.
If you look at the msgpack rpc spec, there's no streaming API built into the spec. I don't see a way to even build it. I don't see a clear API that is needed to enable a use-case.
If you still think this is necessary, please provide a use-case which is not currently supported by the current API, and what API you believe would be necessary to support it. That will help me see what you are alluding to.
from go.
Another use-case is to receive a request over RPC to transmit data (of file(s), which could be 1 or 2 GiB size). Obviously the data can not be sent in one response packet. So, we end up opening the network connection directly and pump out the contents. While doing so, RPC package is expected not to interfere, which is not perfect. Or we end up avoiding RPC in such scenarios.
For RPC, the receiver methods can possibly take io.Reader/Writer or chan as input argument
from go.
Hi @tejorupan it seems your comment was cut and didn't transmit completely. Can you retype it? I would love to see where you were going with it.
Thanks.
from go.
Another use-case is to receive a request over RPC to transmit data (of file(s), which could be 1 or 2 GiB size). Obviously the data can not be sent in one response packet. So, we end up opening the network connection directly and pump out the contents. While doing so, RPC package is expected not to interfere, which is not perfect. Or we end up avoiding RPC in such scenarios.
For RPC, the receiver methods can possibly take io.Reader/Writer or chan as input argument, to deal with the streaming. The exact naming of API we will get into later, once you are convinced.
from go.
A "suggested" naming of the API and how it would be used to solve a use-case more elegantly than the currently provided constructs will help make your case.
My RPC support just provides a codec binding for binc/msgpack that integrates with net/rpc. There is no way to do anything that is not natively supported by net/rpc.
Beyond that, the use-case you provided involves bundling RPC communication and a file for download on the same stream. I don't think that would be a good architecture for solving this problem.
Most folks that request streaming APIs for RPC, etc typically want the ability to stream an array of results, processing them one at a time instead of waiting for the whole array to be populated first. The RPC functionality currently doesn't allow that. And other codecs with non-dependent RPC support (e.g. avro, thrift) have traditionally had a very hard time justifying the need, designing an API, and implementing support, for streaming RPC.
See:
http://highperformancesystem.wordpress.com/2012/11/28/xml-json-protocol-buffer-thrift-and-avro/
https://issues.apache.org/jira/browse/AVRO-406
from go.
I did some more research to find out about any thought put into streaming RPC in the net/rpc library.
See:
https://groups.google.com/d/topic/golang-dev/q-xC7SnHYkY/discussion
https://groups.google.com/d/topic/golang-nuts/zQdDCkPfyuU/discussion
The summary of both is that streaming support for net/rpc was considered, but Rob thought it is very tricky to get right and hasn't seen it done right before. No support has been added.
I think you should consider opening up an issue on the Go issue tracker, because that is where streaming support should be. My codec library just creates a codec for binc/msgpack that integrates with net/rpc. The full usage model is controlled from the net/rpc package.
from go.
The latest commit 4f88b73 contains some code to alleviate the problem. You can build a custom streaming
solution that allows you perform both RPC and non-RPC functions on the same connection. We did this by
providing access to the buffered reader and writer.
It's not a full RPC streaming solution, but allows you construct a custom solution for your specific use-case.
from go.
You may find the following two interesting from the streaming RPC perspective
https://github.com/flynn/rpcplus which is forked from https://github.com/youtube/vitess/tree/master/go/rpcplus
They use chan.
from go.
I looked at the vitess code. They forked net/rpc and added streaming support to it.
That's not the job of the codec library. Its the job of the layer that calls the codec library.
You may want to file an issue with the go net/RPC library to push them to add streaming support.
from go.
Thanks for your feedback.
I will file an enhancement feature request with go net/RPC.
from go.
I opened a new issue: 6569 ( https://code.google.com/p/go/issues/detail?id=6569 ) for this.
Kindly provide your support to steer it forward.
Thank you.
from go.
Related Issues (20)
- codec.MsgpackHandle.RawToString is confusingly named HOT 2
- Go 1.20 - invalid memory address or nil pointer dereference HOT 5
- Msgpack deserialization with codecgen produces wrong result HOT 3
- Unexpected encoding with interface{} key using Canonical and MapKeyAsString.
- Decoding a slice of structs encoded as arrays into a slice of structs without the same number of fields HOT 2
- Duplicate key inside of a map triggers a "cannot decode unsigned integer" msgpack decode error HOT 1
- Efficient direct conversion between supported formats? HOT 2
- Expose Handle.Name() in some form through Encoder and Decoder for use by codec.Selfer HOT 1
- `io.ErrUnexpectedEOF` should be returned instead of `io.EOF` when only partial data is present
- Add `DecodeOptions.IntType` and `DecodeOptions.FloatType` ? HOT 1
- Docs do not seem to mention case-sensitive field handling
- mem leak? HOT 1
- Tests use rand.Intn(1) for boolean rand, but always returns 0
- Decoder accepts indefinite-length CBOR strings containing chunks with mismatched major types HOT 1
- UTF-8 validation isn't applied to CBOR text strings nested inside an indefinite-length text string HOT 2
- Add support for github.com/go-json-experiment/json / UnmarshalerV2 ? HOT 1
- runtime error with golang master (furture go 1.22)
- Leading zeros is not allowed in rfc7159
- attr.go
- [Question] Big Endian architecture
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go.