ssbc / epidemic-broadcast-trees Goto Github PK

It would be nice to have a Documentation on the Wire Protocol. Doing some simple debugging I see that the vector clock is first exchanged then the sequences are sent. But how is backpressure handled?

trying to add a feature, I found my self forgetting lots of stuff about how the protocol works.
such as: when peers connect, one is the client (who calls), the server is expected to send vector clock (notes) first. the place where this happens looks like this:

  //client should wait for the server notes, so that stream
  //can error before a peer sends a massive handshake.
  if(peer.replicating == null) return state

kinda difficult to understand, not very obvious that peer.replicating being unset means means...
in connect:

    //if we are client, wait until we receive notes to send code.
    //this is a weird way of doing it! shouldn't we just have a bit of state
    //for wether we have received a vector clock
    replicating: ev.client ? null : {}

now I recall, this was a JGIW (just get it working)...

There is a lot of non-obvious things going on in fixSeq and the comment just roughly outlines the what it is leaning towards.

IIRC it consolidates the remote and local notes which then decides if there are messages to send.

if you connect to multiple peers at once, it's possible that they send the same messages. Especially in the case of the initial sync, this could result in transmitting many more messages than expected.

The most obvious solution would be to subscribe to activate transmit on only the first peer, but the others in lazy mode. however, then if the first peer just sends messages very slowly (including possibly so slowly it looks like not sending anything) you may get stalled, because currently you only switch peers when you receive a note from another peer.

so if A has {a: 5, b:5} and B has {a:5, b:5} and we are connected to both and have A sending a,b to us, but it pauses at a:3 and b:3, then we are just stuck, even though we could just ask B for those feeds.

I think the solution is to track the latest time we received a message from a peer, and if this is greater than some threashold, switch peers (randomly?) if there is some other peer more up to date than us.

It might also be a good idea when connected to peers to request from each peer randomly too.

I found a couple of links and mentions about this but I couldn't find a clear explanation altough it sounds simple.. Is it just leaving out unchanged feed in the note/vector clock updates?

According to @dominictarr using request(<feedId>, false) is not a good way to pause replication of a feed.

Context from this ssb message: "On that note, I've just realized that you can't just flipflop replicate.request, because when you do replicate.request(id, false) it will send a "I don't want id signal" (-1) and that will screw with with request-skipping, because if you disconnect with the remote during the -1 state, they will remember that you don't want that feed and not mention it next time. (this may break eventual consistency - there are loads of tests in ebt to catch things like this)

Instead, we want to send a the signal that we do what that feed, but not right now! This is already sent if you are getting it faster from another peer. See the code in epidemic-broadcast-trees/v3.js (and v2.js was the old way to encode that). for example, epidemic-broadcast-trees/events.js timeout (line 390) checks if we are expecting messages on a feed but havn't got them, and cycles to another peer for that feed. it calls setNotes(peer, feed, seq, rx) to tell peer that we do (rx=true) or do not (rx=false) want to receive feed from them, but that we are up to seq.

To pause a feed, we just need to be able to set all peers to rx=false for a given feed."

Yesterday my ssb pub was crashing repeatedly, with the following error. I poked around a little trying to figure out the root cause and settled on checking if state was valid in this function. Not sure if that's the preferred solution but it did stop the crash and get my pub running better again. Let me know if there's anything I can do to help

/home/pi/easy-ssb-pub/node_modules/epidemic-broadcast-trees/stream.js:84
      state.msgs.length || state.notes
           ^
TypeError: Cannot read property 'msgs' of undefined
    at EBTStream.canSend (/home/pi/easy-ssb-pub/node_modules/epidemic-broadcast-
    at EBTStream.resume (/home/pi/easy-ssb-pub/node_modules/epidemic-broadcast-t
    at Object.update (/home/pi/easy-ssb-pub/node_modules/epidemic-broadcast-tree
    at Timeout._onTimeout (/home/pi/easy-ssb-pub/node_modules/epidemic-broadcast
    at ontimeout (timers.js:365:14)
    at Timer.unrefdHandle (timers.js:466:3)

After spending some time with the Epedemic Broadcast Trees (EBT) by Leitaõ and trying to run your example, I am wondering what "This module is loosely based on plumtree Epidemic Broadcast Trees EBT paper" exactly means. May it be possible that you explain in which way you differ from the paper?
Cheers!

See this

I'm seeing lots of these console.log() statements -- are these harmless?

Previous: fraction/oasis#12

@dominictarr can you shed some light on the _uptodate_ness of the notes.txt? In some places it looks a bit like copy pasta and this makes me doubt is accuracy somehow..

I'm especially interested in the meaning of negative numbers (https://github.com/dominictarr/epidemic-broadcast-trees/blob/master/notes.txt#L45) since I've only saw -1 in my experiemnts yet.

an example would be awesome too!

Was debugging and was wondering why I got those wierd numbers? :)

Running 6.3.4 I am now getting the following crash in my logs every few minutes when the pub starts syncing data. Not sure what would be causing this. The remote printed below is another pub if that helps chase it down.

 /home/pi/easy-ssb-pub/node_modules/epidemic-broadcast-trees/stream.js:40
  if(this.ended) throw new Error('write after ebt stream ended:'+this.remote)
                  ^
 Error: write after ebt stream ended:@XgC5wDA2EW++ufaDrjRDHXA7Dyd1ce5bTenCm2u6PZU=.ed25519
     at EBTStream.write (/home/pi/easy-ssb-pub/node_modules/epidemic-broadcast-trees/stream.js:40:24)
     at next (/home/pi/easy-ssb-pub/node_modules/push-stream-to-pull-stream/sink.js:31:21)
     at PacketStreamSubstream.weird.read (/home/pi/easy-ssb-pub/node_modules/muxrpc/pull-weird.js:33:7)
     at PacketStream._onstream (/home/pi/easy-ssb-pub/node_modules/packet-stream/index.js:204:12)
     at PacketStream.write (/home/pi/easy-ssb-pub/node_modules/packet-stream/index.js:135:41)
     at /home/pi/easy-ssb-pub/node_modules/muxrpc/pull-weird.js:56:15
     at /home/pi/easy-ssb-pub/node_modules/pull-stream/sinks/drain.js:24:37
     at /home/pi/easy-ssb-pub/node_modules/pull-goodbye/node_modules/pull-stream/throughs/filter.js:17:11
     at Object.cb (/home/pi/easy-ssb-pub/node_modules/packet-stream-codec/index.js:111:11)
     at drain (/home/pi/easy-ssb-pub/node_modules/pull-reader/index.js:39:14)

Looks like the new block code in 6.3.4 breaks test/block2 and test/block3 in sbot because replicate:finish is called twice.

@corlock wrote:

General documentation of EBT's makes sense, but what was unclear to me was where the actual networking and message propagation is handled. My current understanding is that this is connected to the createStream method, which isn't really documented in this module. It is referenced, but not made explicit what that means or how streams interface with EBT.

would an example clear it up? hmm, I see it just says

var stream = ebt.createStream('bob')
stream.pipe(remote_stream).pipe(stream)

that's a pattern that will be familiar if you know my earlier streams work, but isn't explained or an explaination linked to, not sure where that documentation should be. But it should be somewhere.

In the case where one nukes the local db and needs to resync.

I was running 6.0.1 and hit the following crasher bug. Not sure if this is still a bug in latest version.

/home/chrx/.ssb/node_modules/ssb-ebt/node_modules/epidemic-broadcast-trees/events.js:34
if((peer.clock[feed_id] || 0) > (state.clock[feed_id] || 0)) {
^

TypeError: Cannot read property '@6CAxOI3f+LUOVrbAl0IemqiS7ATpQvr9Mdw9LC4+Uv0=.ed25519' of null
at isAvailable (/home/chrx/.ssb/node_modules/ssb-ebt/node_modules/epidemic-broadcast-trees/events.js:34:21)
at Object.exports.timeout (/home/chrx/.ssb/node_modules/ssb-ebt/node_modules/epidemic-broadcast-trees/events.js:351:22)
at Timeout._onTimeout (/home/chrx/.ssb/node_modules/ssb-ebt/node_modules/epidemic-broadcast-trees/index.js:87:25)
at ontimeout (timers.js:478:11)
at Timer.unrefdHandle (timers.js:598:5)

/home/chrx/dev/scuttlebot/node_modules/epidemic-broadcast-trees/events.js:415
if(peer.clock[feed_id] || 0 > state.clock[feed_id] || 0) {
^
TypeError: Cannot read property '@f/6sQ6d2CMxRUhLpspgGIulDxDCwYD7DzFzPNr7u5AU=.ed25519' of null
at /home/chrx/dev/scuttlebot/node_modules/epidemic-broadcast-trees/events.js:415:20
at eachFrom (/home/chrx/dev/scuttlebot/node_modules/epidemic-broadcast-trees/events.js:75:8)
at Object.exports.timeout (/home/chrx/dev/scuttlebot/node_modules/epidemic-broadcast-trees/events.js:413:5)
at Timeout._onTimeout (/home/chrx/dev/scuttlebot/node_modules/epidemic-broadcast-trees/index.js:94:25)
at ontimeout (timers.js:478:11)
at Timer.unrefdHandle (timers.js:598:5)

//create a stream and pipe it to another instance
var stream = ebt.createStream('bob')
stream.pipe(remote_stream).pipe(stream)

I'm trying to understand where that remote_stream variable comes from. And more importantly what it's for.

Hello @dominictarr, when I run the example code a TypeError: listeners[i] is not a function error occurs. See code running here: https://runkit.com/carloslfu/58f98490b7c4ff00127fe01c.

Note: Thanks for sharing all these awesome projects, all those are exciting 😃 .

Seems all the dependencies are missing? At least push-stream is missing.

Reading the code to try and understand what it does, and found this line that is always false?

https://github.com/dominictarr/epidemic-broadcast-trees/blob/master/index.js#L17

Shouldn't it compare against min?