Networking: NOISE handshake and transport encryption about hypercore HOT 7 CLOSED

Now some more progress here, yeeha:

https://github.com/Frando/hypercore-protocol-rs

With the standard-dh branch of noise-protocol in Node.js, and two patches to snow ((1), (2)), I now have a mostly working (yet very basic) implementation of hypercore-protocol 7 (Hypercore 8 / Dat 2) in Rust: Pass the handshake, set up the transport encryption, open channels, verify capabilities and send and receive messages. Next would be to integrate with hypercore!

Also if someone would want to review the API and implementation of the repo linked above, I'd be very interested 😄

from hypercore.

There are some things I did in noise-protocol that are "non-standard" but that I want to fix:

• Use chacha20-poly1305 instead of xchacha20-poly1305. Not a requirement to be standard, but barely any other implementations have xchacha20. I also send a nonce over after the initial handshake which is also "non-standard". It's hard to talk about standards with noise tho, since it's a framework for writing protocol handshakes and not a strict standard
• I use the sodium kx APIs for Diffie-Hellman. I thought this was safer before I really learnt how the protocol works as this hashes the symmetric key derived from the DH function. Since everything in noise is "hash and mix" this doesn't matter, and just makes it harder to write an implementation in other languages.
• I never really considered transport encryption part of noise, but when I read the spec now I see that there's extensive instructions on what transport encryption is under the noise framework. I should include this in noise-protocol

The above points I am going to remedy soon, but these are also the only parts that are noise here. The rest is how hypercore has decided to use noise under various constraints.

I know that hypercore uses "dummy keys" and I think it uses "channel binding" as part of the new capabilities system. The application responsibilities also makes it more tricky to get different implementations to talk to each other: http://www.noiseprotocol.org/noise.html#application-responsibilities

Maybe it's worth it to consider using a pregenerated noise implementation from https://noiseexplorer.com and modify that to how hypercore uses noise?

from hypercore.

@emilbayes cool, thanks for this info.

With your standard-dh branch I got one step forward: Now the handshake between node and rust semi-completes - the initiator finishes correctly, the not-inititiator dies with a decrypt error.

I set up a repo to better test this: https://github.com/Frando/rust-node-noise-handshake

Feel free to chime in there or help to debug :-D could also be a good base for proper integration tests, it has a one-command runner that starts both server and client.

from hypercore.

This is all fixed and can be closed :)

See hypercore-protocol-rs. My PRs to snow got merged, and the new Hypercore 9 release switched the handshake algorithm. It also switched the handshake cipher from XChaChaPoly to ChaChaPoly, so one of the PRs to snow wasn't even needed in the end.

Anyway - the master branch of hypercore-protocol-rs can handshake, verify cabalities and exchange all messages with a nodejs hypercore now. Once snow gets a new release, I'll publish a first preview release.

from hypercore.

Some more investigations into the transport encryption after the handshake is complete:

The NOISE spec states with regard to transport messages:

A Noise transport message is simply an AEAD ciphertext that is less than or equal to 65535 bytes in length, and that consists of an encrypted payload plus 16 bytes of authentication data. The details depend on the AEAD cipher function, e.g. AES256-GCM, or ChaCha20-Poly1305, but typically the authentication data is either a 16-byte authentication tag appended to the ciphertext, or a 16-byte synthetic IV prepended to the ciphertext. (source)

and also

Applications must handle any framing or additional length fields for Noise messages (source)

(which in my tests was very true - the read_message method of the transport state in snow can only decrypt messages that are exactly the same messages as created by the write_message call on the other side, thus needing length-prefixes during transport)

hypercore-protocol in transport uses a streaming XSalsa20 cipher from libsodium, where the docs state:

The ciphertext is the message combined with the output of the stream cipher using the XOR operation, and doesn't include any authentication tag. (source)

So hypercore-protocol follows NOISE for the handshake, but does not use NOISE for the transport encryption, instead it uses XSalsa20 for streaming encryption with no authentication tags and no need for length-delimited messages for the decryption. This means that by-the-spec NOISE frameworks cannot be used for the transport phase of a hypercore-protocol stream.

from hypercore.

@mafintosh / @emilbayes: Could you check if these findings about hypercore-protocol are correct, and clarify if there's a reason why hypercore-protocol does not stick to the NOISE spec for the transport phase?

from hypercore.

I pushed a client/server example of how far I got with the handshaking, see here for details.

from hypercore.