Design discussion about go-libp2p-daemon HOT 15 CLOSED

from go-libp2p-daemon.

i'm feeling pretty good about HTTP/JSON API with a unix socket based stream manager, where the client polls the configured directories in the filesystem for new incoming sockets

from go-libp2p-daemon.

Per discussion with bigs on zoom:

• We discussed a symmetric unix domain socket stream protocol
  • the daemon listens to a unix socket, and the client initiates streams by opening connection and issuing a protocol header
  • the client listens to a unix socket to provide stream handlers; the daemon connects back on stream open and issues a protocol header
• The daemon exports the control interface through http/json, as is the simplest and most flexible approach to begin with. We can later implement a binary control protocol over unix socket as well.

The protocol header must contain at minimum:

• a disambiguator, to allow later implementing the control protocol multiplexed in the daemon
• the peer ID and multiaddr
• the protocol for the stream

We settled on using delimited protobuf for the protocol header, as this saves the need to write custom serializers.

from go-libp2p-daemon.

Before going into design considerations too much, let's flesh out our motivations so we get on the same page: #3

Notes on the current discussion:

• Serialization Format: I'd like to make a somewhat bold proposal: no JSON. JSON is lacking a "bytes" type and this has caused no end of trouble. How do you feel about mandating CBOR?
• Multi-Tenet from day 1: We'd like much of this daemon to eventually move to the kernel (or, at least, a system daemon) and it turns out that adding in multi-user support later is rather tricky.
• 100% libp2p: I'd like to at least entertain the idea of going 100% libp2p; that is, no HTTP API. This would make everything we do network transparent (in theory). We may need to have some service expose an HTTP API for simplicity but we should at least consider a micro-kernel like architecture where that's a separate daemon.

from go-libp2p-daemon.

If we are going JSON-less, let's not add another format -- we can use protobufs for the control protocol and multiplex in the UNIX socket.

from go-libp2p-daemon.

For multi-tenant applications we might have the issue of who's handling the streams -- there can be only a single stream handler for each protocol.
I think it makes more sense for each application to run its own daemon; the application can also be composed of multiple processes.

from go-libp2p-daemon.

Couple of points here:

1. Another way to view what we're doing is stream virtualisation.

2. Supposing that the daemon is exposing streams over unix sockets and SHM, we should publish lightweight client bindings/protocol SDKs in different languages. We really don't want users (re)implementing the plumbing across the board to attach their apps to our our local, virtualised libp2p transports.

3. I echo @Stebalien's third point about keeping it 100% libp2p. I'd advocate for the daemon to be started with the listen multiaddr for the control plane, that only accepts local transports, e.g. --listen /unixsocket/[/var/unix/...], along with an option to enable the --http layer, outputting a warning that interface is only for development/testing/admin.

For the multi-tenant mode, the master control plane could accept only two commands: new, attach.

• With new an app starts a new session, does an encryption handshake, and receives a private socket/shm assignment for its app control plane (which is encrypted for that app only), along with a token to re-attach in the future.
• With attach, an existing app could reattach by providing the token.

Just some initial brainstorming, really.

from go-libp2p-daemon.

I think that requiring clients to implement yamux/secio is a huge burden for bindings implementors (speaking as one :)

from go-libp2p-daemon.

Agree. Bindings should not perform multiplexing, that's precisely what the daemon does for them, tunnelling each stream onto a physical mapping atop a local transport. So there'd be a 1:1 mapping between a local resource (e.g. shm, socket) and a backing stream.

Regarding secio, if we want multitenancy and isolation in the future, I guess we'll need an encryption mechanism to avoid apps cross-reading streams. But yeah, that complicates binding implementations. Alternatively, we could leverage OS resources like cgroups to provide the isolation.

from go-libp2p-daemon.

I'd advocate for the daemon to be started with the listen multiaddr for the control plane, that only accepts local transports

Technically, we don't even need to do that as long as we can whitelist (although we may want to anyways).

I think that requiring clients to implement yamux/secio is a huge burden for bindings implementors (speaking as one :)

My thinking here is that we'd use a super special local-only transport. That is, we build a SHM/unix domain socket transport that does 90% of the work on the server side. We could even have multiple: a simple one that uses a new file descriptor per stream and a fancy one that uses memory mapping and a single socket for control information. We also don't need to do any secio/encryption as it's all local and privacy/authentication can be enforced by the kernel. The real tricky part here would be key management because we currently expect all "peers" to be identified by a public key. We could do an authentication round using signatures but that feels like overkill.

The only tricky part here would be peer IDs but, in theory,

from go-libp2p-daemon.

Let's not have so many words and no code!
Initial implementation: #4

from go-libp2p-daemon.

I'm sorry, but is SHM transport is really needed?
In situation where

client -> shm -> libp2p-daemon -> network

libp2p-daemon become a bottleneck. SHM will allow clients to send data in much bigger speeds then libp2p-daemon will be able to handle and send to network, because network is much more slower then SHM.

In such case libp2p-daemon will needs to hold big buffers to keep all the incoming packets, or lock incoming clients until it will be able to send data over network.

from go-libp2p-daemon.

@cheatfate the spec has actually been formalized (in its current state) in SPEC.md. shm was thrown around as a more direct, efficient, method of IPC. we're not approaching it at the moment.

from go-libp2p-daemon.

Yeah, as @bigs says SHM is just on the radar, but not an immediate priority. We're aware of the complexity, and it'll warrant much experimentation.

I think your remark boils down to needing a mechanism for backpressure. Unix domain sockets inherently provide this (I believe). With SHM, it'll need to be part of the protocol agreed between both processes.

I have a lot of investigating to do before I can provide better answers, but for now SHM is just in the wishlist ;-)

from go-libp2p-daemon.

at this point, implementation details are starting to settle a bit. i'm going to close this conversation for now.

from go-libp2p-daemon.