Comments (19)
@lulian7 - IIUC you should be able to use the workaround suggested here for Java gRPC clients: https://github.com/soheilhy/cmux#limitations
from cmux.
Any ideas how to match grpc-go requests then?
from cmux.
@dfawley Works. I missed that.
from cmux.
Testing with gRPC-Go 1.19.1 I can't get this to work even with
grpcL := mux.MatchWithWriters(cmux.HTTP2MatchHeaderFieldSendSettings("content-type", "application/grpc"))
Has anything changed in 1.19 to make this even harder? I tried debugging by sending non-matched entries to another listener and printing the output and this is what I got:
���c+��]
80�l
or as a byte slice:
[22 3 1 0 217 1 0 0 213 3 3 86 90 234 191 121 47 47 94 124 149 27 35 62]
I'm not sure what format this is. Any ideas? @menghanl @dfawley
from cmux.
@johanbrandhorst No, this behavior didn't change in 1.19.
I also don't recognize the bytes...
Does it work with 1.18? Is the client a simple client?
from cmux.
I haven't tried 1.18 yet, but yes it is a simple grpc-go client. I'll try 1.18 and below.
from cmux.
Hm, everything is working as expected when serving gRPC without TLS. I guess this is a false alarm. What's the best way to use cmux
and still encrypt the connection?
from cmux.
For future reference;
I got the following working:
- Use
tls.Listen
to create the listener. - Use
(*grpc.Server).Serve
withoutgrpc.Creds
. - Use
grpc.WithTransportCredentials
as usual in the client.
from cmux.
@johanbrandhorst I finally took some time to sketch a code snippet to make our use case work. It did, but I don't know if it's going to work after reading this commit (as we didn't upgraded to that grpc-go version yet). I want to expose grpc-gateway + pure grpc in the same port without security; It's a microservice never exposed directly to the outside so it's ok. But I don't want to commit to a solution that cannot be extended in the future by adopting an approach like BeyondCorp. This means that our microservice should be able to be exposed publicly by, obviously, using TLS. The mentioned code allows addition TLS over the whole cmux but I don't know if you encountered further problems with this approach?
from cmux.
There should be no reason you can't do TLS termination before any packets reach the service.
from cmux.
Correct, thanks for confirming. Although we found another issue with the combo grpc-gateway, grpc, port-sharing, cmux and persistent connections kept by a reverse proxy (envoy in this case). It seems that cmux
is not very useful if we're behind a proxy which keeps persistent connection without doing any differentiation between the underlying protocol being used.
Take this sequence as an example:
As stated by cmux
, it matches the connection when it's accepted, so it's possible this to happen:
- Someone makes a gRPC call
- Envoy creates a new http connection
- cmux sees it's a http2 connection and the content-type is gRPC it then creates a new connection against
grpc.Serve
grpc.Serve
replies to the gRPC request succesfully. While Envoy decides not to close the fresh connection for performance reasons- Someone makes a REST call
- Envoy decides to use the same connection created before
- As the connection was previously matched
grpc.Serve
is called again and it fails as it's not a gRPC call
WDYT @johanbrandhorst?
PS.: I don't really know which is the right forum for this question as there are lots of little components taking part in the overall problem...
from cmux.
This is probably not the place, no. Maybe raise an issue against Envoy?
from cmux.
This is probably not the place, no. Maybe raise an issue against Envoy?
Already asked in Envoy and it seems like nothing could be done there as it keeps one and only connection pool for everything no matter it's plain or gRPC.
UPDATE: It can be done but cannot because we're using Contour to manage Envoy.
@mpuncel wrote:
Envoy uses the same connection pool for gRPC and non-gRPC. You could probably accomplish what you want with 2 different clusters and set up a routing rule that matches on gRPC to go to 1 and everything else goes to the other
from cmux.
There's a question to be asked about why you're using cmux behind a proxy like envoy anyway. Envoy can do the traffic splitting for you, so why not just remove the cmux?
from cmux.
This is probably not the place, no. Maybe raise an issue against Envoy?
There are several reasons:
- We want to have one and only one container for both (gRPC and gRPC-Gateway) as it simplifies everything from management/operation to understanding of how the microservice works by the team.
- The operational cost for having multiple ports is bigger than having only one.
- Although Envoy supports multiplexing connections based on more advanced L7 rules (like header content based ones),
HeptioVMware Contour doesn't support it yet. Which means that currently it's not even possible.
Did you have this problem previously @johanbrandhorst? How do you currently operate services that publish both gRPC + REST in your environment?
Thanks!
from cmux.
I think we can move this discussion to the #grpc channel on Gophers slack. We've taken up this issue for long enough.
from cmux.
TLS: net/http uses a type assertion to identify TLS connections; since cmux's lookahead-implementing connection wraps the underlying TLS connection, this type assertion fails. Because of that, you can serve HTTPS using cmux but http.Request.TLS would not be set in your handlers.
do I understand this correctly if I assume that I can still serve http2 and http1(grpc server+ grpc gateway server) via cmux on the same port and have TLS for the clients and I only have to enable tls on the listener that I provide to cmux and disable tls for the servers the cmux is routing to? ie. I still can serve secure grpc and grpc gateway via cmux since both servers are linked to the cmux only via memory and not network, hence no security issue with tls "termination" on cmux level?
from cmux.
That is correct, TLS is terminated by cmux, but it's still in-memory, so it's about the same level of security.
from cmux.
Ran into this issue. Can we at least update the documentation for this?
from cmux.
Related Issues (20)
- Upgrade the x/net version in dependencies?
- metrics for mux usage
- cmux is not usable with GRPC+TLS
- Add benchmarks
- badly-behaved client can deadlock server stopping HOT 3
- Cannot get Matchers to mux h2c gRPC and HTTP2(REST) requests HOT 4
- It's hard to extend protocol HOT 3
- how to write client codeto gprc connect HOT 2
- Doesn't work with grpc-go's hello world example HOT 2
- Pull In Downstream Fixes HOT 1
- Cannot check closed error with errors.Is HOT 1
- Multiplexing gRPC and Prometheus metrics services using Google Cloud Run
- Matching based on URL HOT 1
- Cmux example HOT 1
- Close not working properly on gRPC graceful stop HOT 1
- Multiplexing gRPC & HTTP handlers with H2C+Prior Knowledge HOT 4
- socks5 matcher not working HOT 2
- http2 matcher not working with net/http HOT 2
- I don't understand why calling source.Read() might block?
- HTTP1Fast matcher omits PATCH HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cmux.