P2P sharding network(s) about diamond_drops HOT 13 OPEN

I don't think we'll use Swarm as data will be stored on the blockchain, but it could also be done with Swarm as a complementary storage service.

i) How will users running a node behind a NAT traverse it in order to establish connections to peers in our implementation of libp2p's IPFS?

For discovery mDNS-discovery looks like a low priority since it's designed for LANs. random-walk seems like it may be a good candidate. Due to the partial trust with bootstrap list, that is another low priority / enhancement alternative.

For relay when it is not feasible to reach a peer through a NAT, https://github.com/libp2p/specs/tree/master/relay would be useful, however rust-libp2p is unfortunately not listed as an implementor of it.

from diamond_drops.

Or have one network...

https://ethresear.ch/t/torus-shaped-sharding-network/1720/6

from diamond_drops.

Please review the following questions that I've prepared after reading through the libp2p spec at https://github.com/libp2p/specs. Try to challenge/answer the questions collectively as a team:

1. What libp2p protocol categories/roles and associated properties, security models, and respective network size are we using on each layer of our stack, and what are scenarios where multiple protocols would run simultaneously?

• References:
  • https://github.com/libp2p/specs/blob/master/2-state-of-the-art.md#22-categorizing-the-network-stack-protocols-by-solutions
  • https://github.com/libp2p/specs/blob/master/9-references.md

2. What collection of transport protocols will we be running libp2p protocol over?

• Reference: https://github.com/libp2p/specs/blob/master/3-requirements.md#31-transport-agnostic

3. i) What port(s) will we use on the libp2p network layer to perform operations for each of the transport protocols that we’ll be using?
   ii) What is an example scenario where we’ll be protocol multiplexing multiple transport protocols (i.e. UTP, UDT, WebSockets, SCTP, WebRTC, TCP Remy) through a single port?
   iii) What is an example scenarios where we'll be multiplexing reliable stream(s) or unreliable datagram(s) in a connection from an IPFS node to other node(s) on the a transport protocol in the network layer of the communication model?
   iv) What protocol(s) would we be mounting on top of libp2p with the connection?
   v) What type of connection would we be using?
   vi) What platforms and network setups will we need to libp2p to support?
   vii) How will we build a multiaddr-based network package module like go-multiaddr-net in Rust, for IPFS to delegate the transport dialing to, and for scoping the implementation of other transport protocols.
   viii) What portable format encodings do we need to use for protocol messages (i.e. protobuf)?
   ix) What are scenarios when we will use multistream, multistream-select, and Stream Multiplexing? What default multiplexor will we use with Stream Multiplexing (i.e. HTTP/2, SPDY, QUIC, or SSH)?
   x) How will we use Multicodecs for each IPFS Protocol that we use?

• References:
  • https://github.com/libp2p/specs/blob/master/3-requirements.md#32-multi-multiplexing
  • https://github.com/libp2p/specs/blob/master/7-properties.md#72-ports---constrained-entrypoints
  • https://github.com/libp2p/specs/blob/master/7-properties.md#73-transport-protocols
  • https://github.com/libp2p/specs/blob/master/7-properties.md#755-portable-encodings
  • https://github.com/libp2p/specs/blob/master/7-properties.md#751-protocol-multiplexing
  • https://github.com/libp2p/specs/blob/master/7-properties.md#757-protocol-multicodecs

4. i) How do we need to encrypt communications on libp2p for security in our implementation? Under what scenarios would users not need to operate with encrypted communications?
   ii) Is libp2p's application of cyphers with only a minimal portion of the whole TLS standard used sufficient for our implementation?

• References:
  • https://github.com/libp2p/specs/blob/master/3-requirements.md#33-encryption
  • https://github.com/libp2p/specs/blob/master/7-properties.md#756-secure-communications

5. i) How will users running a node behind a NAT traverse it in order to establish connections to peers in our implementation of libp2p's IPFS?
   ii) Will we use libp2p's ICE-like protocol for full NAT traversal or will we use a different NAT traversal library instead of libnice, such as libwebrtc or natty?
   iii) How will we check if our implementation of NAT traversal is interoperable?
   iv) How will we implement optional fallback to connection relaying communication as a transport that may be turned off by users to ensure a full connectivity graph?

• References:
  • https://github.com/libp2p/specs/blob/master/3-requirements.md#34-nat-traversal
  • https://github.com/libp2p/specs/blob/master/3-requirements.md#35-relay

6. What network topologies will we be using for our system's routing mechanisms, peer discovery, and message propagation?

• References:
  • https://github.com/libp2p/specs/blob/master/3-requirements.md#36-enable-several-network-topologies
  • https://github.com/libp2p/specs/blob/master/6-interfaces.md#6-interfaces

7. i) What information will records (IPFS Linked Data) in our network store for discoverability?
   ii) How will we use libp2p to locate resources in the network and side channels?
   Reference: https://github.com/libp2p/specs/blob/master/3-requirements.md#37-resource-discovery

8. What libp2p messaging protocols will we let users use after they open a new stream and register a stream handler and why?

• References:
  • https://github.com/libp2p/specs/blob/master/3-requirements.md#37-resource-discovery
  • https://github.com/libp2p/specs/blob/master/7-properties.md#71-communication-model---streams

9. How will we adopt naming conventions in our implementation of the network?

• References:
  • https://github.com/libp2p/specs/blob/master/3-requirements.md#39-naming
  • https://github.com/libp2p/specs/blob/master/4-architecture.md#46-naming

10. What is a scenario where different libp2p subsystem interfaces (i.e. Peer Routing, Swarm, Distributed Record Store, and Discovery) would use each other in our system implementation?
    Reference: https://github.com/libp2p/specs/blob/master/4-architecture.md#4-architecture

11. What Peer Routing subsystem mechanism will we implement and why?
    Reference: https://github.com/libp2p/specs/blob/master/4-architecture.md#41-peer-routing

12. What are different Swarm scenarios that we need to handle (where we are using a stream)?

• Reference: https://github.com/libp2p/specs/blob/master/4-architecture.md#42-swarm

13. What Distributed Record Store system will we use and why?

• Reference: https://github.com/libp2p/specs/blob/master/4-architecture.md#43-distributed-record-store

14. i) What Discovery Protocols will we use and for what scenarios?
    ii) Will our Discovery Protocol need encrypted discovery beacons (i.e. mDNS)?
    iii) Would we use a Bootstrap-List Discovery Protocol and make it use configurable?

• Reference: https://github.com/libp2p/specs/blob/master/4-architecture.md#44-discovery

15. Which of the libp2p modules (transports) listed at the following reference link will we be using with the Rust implementation of libp2p https://github.com/libp2p/rust-libp2p?

• References:
  • https://github.com/libp2p/specs/blob/master/8-implementations.md#8-implementations
  • https://github.com/libp2p/specs/blob/master/6-interfaces.md#61-libp2p

16. How will we use the Swarm Dialer?

• Reference: https://github.com/libp2p/specs/blob/master/8-implementations.md#811-swarm-dialer

from diamond_drops.

libp2p/rust-libp2p#187

from diamond_drops.

https://github.com/libp2p/go-floodsub/issues/77

https://ethresear.ch/t/topology-on-libp2p-libp2p-design/2028/3?u=jamesray1

WIP on answering the above questions, thanks for asking.

from diamond_drops.

rustlib-p2p has implemented relay-0.1.0: https://github.com/libp2p/rust-libp2p/tree/master/relay.

from diamond_drops.

I'll probably implement gossipsub since it is preferable to floodsub.

from diamond_drops.

#94

from diamond_drops.

"Exploring Ethereum on libp2p" libp2p/libp2p#33 (comment)

from diamond_drops.

I just want to clarify that we don't need to design a libp2p implementation from scratch, a lot of work has already been done with rust-libp2p, plus other Rust modules. I've created a wiki page with your questions here, and will try to answer them in due course.

from diamond_drops.

I've answered about half of the questions, might come back to them some time later after writing gossipsub and looking at other priorities.

from diamond_drops.

@jamesray1 hey, I'll be joining the meetings in Berlin later this week for the libp2p team and starting to prep for it -- is the current state of the questions in the comment above or on the wiki page you mentioned?

from diamond_drops.

@lgierth hi, it's on the wiki page. I haven't bothered to answer all of them yet as I was tending to repeat myself by referring back to rust-libp2p, which Luke didn't seem to be aware of.

Right now I'm reading through tokio (a Rust crate for async networking), and before that was reading up on libp2p, rust-libp2p, Go, gossipsub, etc. Will start developing gossipsub soon.

from diamond_drops.