[DESIGN PROPOSAL] Generate a gRPC request/response based on a GraphQL query at build time about rejoiner HOT 15 OPEN

Guys!!! checkout this one that I built, https://github.com/Just4Ease/graphrpc

I was thinking deeply about using GraphQL as my API contract for RPCs.
Although, my underlying data transfer uses NATS.io ( nats.io ).

Let me know what you think, we could improve that or start a new project entirely focussed on this subject.

PS: GraphRPC is still a wip, but I'm using it in a backend app, so as to force me to get it to be stable and all working and neat.

from rejoiner.

Hello Nick,

We are actually also exploring building a tight integration between GRPC and GraphQL at my company. We are still very early in the process and we won't get to it for a few months, but I'm very curious to follow how your proposal develops. Hopefully we can contribute some ideas back.

On that topic, it would helpful if you could clarify a few elements. You mention "The GraphQL API explorer has been updated to include a proto preview, which can be seen in the above screen shots." but I can find any public reference to that. Is that internal to Google?

Also the "Request message" section seem to talk about the response only. Are "Request message" and "Response message" titles switched?

Finally could you clarify a bit the first rejected alternative? What part of the GraphQL spec could were you not able to support?

Thanks a lot,
J

from rejoiner.

Hey Julien,

Here is more information about the rejected alternative:

The previous attempt at offering a proto based GraphQL API didn’t support the full GraphQL spec. It created a static proto based on the GraphQL schema for a GraphQL server. This resulted in a standard looking proto response, but didn’t support the full GraphQL spec. Since the response proto was generated from with schema without knowledge of any query it could not support field name aliasing. Since there is no aliasing it also means it didn't support calling a method multiple times with different parameters. Also see the test file.

from rejoiner.

Great thanks.

The current approach does make sense a lot of sense now. The aliasing spec of GraphQL is indeed going to require request-specific proto generation. That's a little unfortunate because I have a hunch that it's going to add a bit of friction when developing against a GraphQL/GRPC endpoint in IDEs. We'll most likely need to recompile a target each time the GraphQL query is modified in order to avoid type errors and get a accurate type completion.

The query is parsed into an AST, which is then transformed via a QueryTraversal into a DescriptorProto. Additional type information supplied from the Schema during query traversal. The same heuristics are used as those in the client when creating a proto from the query.

Just one random thought. The need to have the same heuristics between the client and server introduces additional coupling. Down the road, it will probably make sense to fully specify that heuristics so we can generate GraphQL/GRPC clients using other libraries / languages.

What's the status of the project? What type of contributions are you looking for?

from rejoiner.

One more thought. One of the use cases we were hoping to support is to have one canonical GraphQL schema aggregating / stitching multiple downstream GraphQL servers via GRPC so we use binary proto over the wire and save one JSON serialization / deserialization cycle. It would be interesting to use the the main canonical GraphQL server generate the proto description at runtime in order to deserialize the dynamic protobuf received by downstream GraphQL / GRPC servers.

EDIT: I originally missed the solution that I described.

from rejoiner.

Currently I've tested a few of the technical aspects of the proposal to check that it will actually work. I've been able to dynamically create a proto definition and populate it on the server. I've started the proto preview feature which would be used or referenced for creating the RelayJs compiler plugin. Relay just released plugin support, so nothing should be blocking development.

I'd like to get a end to end demo running to discover any issues with this approach. For this demo it might make sense to fake the Relay plugin since it seems the most straight forward.

What aspects of the project would you be interested in contributing to? The main parts are the relay compiler plugin and the servers dynamic response generation, but any contributions are welcome.

from rejoiner.

small update

a variation to this approach is to have the client send a (text) version of the proto descriptor in the request so that the client has control over field numbers and can maintain backwards compatibility. This would allow the GraphQL client to send the protos it receives from the GraphQL to other services.

from rejoiner.

@siderakis and @jnak, thank you for all your work on this. I like your approach. I was also wondering why the client wouldn’t send a descriptor along rather than trying to figure it out/compute it with heuristics.

Additionally, why send in text when descriptors are themselves protos, and could be encoded in binary? Just asking, I am sure you have your reasons.

Thank you for your hard work on this project. We’re really excited about it

from rejoiner.

I forgot to say, we would also like to help if you are looking for PRs, or testers :)

from rejoiner.

Does grpc support fieldmasks for repeated fields?

from rejoiner.

Just throwing an idea out there: while creating a response in graphql-json, much time is spent waiting for ALL dataloaders to finish and then construct the whole tree.
A grpc response could possible not contain an actual tree, but a stream of pieces as they become available. Each piece would be a subtree and a path where it would fit in the result.
The client would then be responsible for piecing the tree together, if required, or actually consuming the pieces as they come down the line.
I have no idea how this would be implemented given graphql-java at the moment ;-)

from rejoiner.

from rejoiner.

Hi Nick, what I am proposing is indeed similar to @stream and @defer, except it wouldn't need any directives, it would simply be the default mode of operation of the underlying grpc channel.

In the client side you would decide whether to wait for the whole tree to be constructed, like now, or to subscribe to (updates to) subtrees.

It could actually take memory load off the server because whenever a future in a dataloader is completed, it's data is sent over the wire and could become eligible for garbage collection.

This means that a protobuf structure generated for a particular query would usually not be sent over the wire in one piece. It might be constructed on the client side if the programmer chooses that mode of operation. Otherwise, the programmer would programmatically identify paths and callbacks/futures which would receive protobuf structures for fragments of the query.

This would also nicely extend to include @LiVe updates or subscriptions.

from rejoiner.

@siderakis we've just adopted Rejoiner internally and we'd love to help pick this up, however, we're not in a React environment, so RelayJS may not work for us.

i'm wondering if there are components from Google's universe for this? if you know of anything that might make GraphQL usable via Closure, let me know, otherwise, we'll add it to the lists of things to explore and/or write.

from rejoiner.

Is it really necessary to use a custom proto definition for each query? It's relatively easy to create a generic proto for a json-like dom structure. The overhead would be a few bytes for specifying types that are not strictly necessary.
After that there are many query->client generators that could be adapted to use this json-over-protobuf format.
So, apart from the performance, are there any other reasons to use a custom proto?

from rejoiner.