Git Product home page Git Product logo

synapse's Introduction

Synapse

A bittorrent inspired, highly modular p2p data transfer system.

Notable Changes

  • Metainfo pieces list is actually a list, not a concatenated hex string, because we can select specific hashing algorithms to calculate the piece hashes.

Notes:

WebRTC Support

Bittorrent DHT

  • When saving data to a DHT it is SHA1 hashed as a limitation of bittorrent-dht. So when we use an infosig, it becomes SHA1(YOUR_SIG(YOUR_HASH(METAINFO))) or for infohash its SHA1(YOUR_HASH(METAINFO))

@hyperswarm/dht

  • Cannot use afaik, will not run in browser as it relies on sodium-native.

To Do:

  • Figure out a nice way to handle streaming chunks in order for consumers

  • Reduce memory pressure by having the piece buffer be backed by disk

  • Smart piece management

  • Rarest Piece first?

  • Retry logic

  • Multiple Peer handling

  • Metadata discovery

  • Become seeder after Metadata discovery

  • Select desired files, not just all files in torrent (Desired Bitfield vs Have Bitfield ?)

  • WebRTC

  • TCP

  • UDP

  • Browser compatibility

  • Node JS Server

  • New Metainfo Spec

  • DHT compatibility (Using Hyperswarm)

  • Signatures in Metainfo

  • Merkel Tree for file pieces hash

  • Nice API for accessing torrents on the fly Torrent.stream, yield Torrent.nextPiece()

  • [] Split Repo to reduce dependencies

Thoughts:

Mutable Torrents:

  • In the standard Mutable Torrent spec, it specifies that you should use 1 priv/pub key per torrent (seemingly).
    However, the issue with that in regards to this service is that users would need to store many many private/public keys per each file uploaded.
    This is a pain in the ass and although it should be possible, it should not be required.

  • Instead of 1 keypair per torrent, I propose using a signature of the infohash of the original torrent.
    eg. HASH(OriginalInfoSig) -> { NewInfoHash, SIG(NewInfoHash), PublicKey, OriginalInfoSig, seq }

    1. This would allow someone to look up the newest infohash from the original infosig.
    2. Validate that the NewInfoHash + SIG(NewInfoHash) is valid with the public key.
    3. Validate that the PublicKey published in the update is the same PublicKey associated with the OriginalInfoSig.
    4. Use seq to identify what iteration of the torrent we are looking at.

Application Structure:

  • Each added Torrent exists within its own scope/lifecycle. (TorrentManager.addTorrent)
  • Each Torrent remembers a list of peers (PeerManager) and which pieces it has along with the InfoHash associated
  • Observer Pattern, Torrent Manager sits in the middle and recieves events.
    1. For example, when a new peer is found, the Peer Manager alerts the TorrentManager, to which the torrent manager will ask what pieces they have.
    2. Or another example, when the UpdateScanner notices there is a new update for a given InfoSig, it will call back to the TorrentManager to get the latest metainfo file and validate which pieces have changed.
  • What do you need to seed a torrent?
    1. Metainfo
    2. Pieces
  • What do you need to leech a torrent?
    1. InfoHash/InfoSig
  • Search for Metainfo from InfoIdentifier (InfoID)
    1. Search DHT for Peers with the given InfoID
    2. Use ut_metadata extension to retrieve metainfo
    3. Start standard TorrentManager with no pieces
  • InfoID vs InfoHash vs InfoSig
    InfoID is the generic name for either the InfoHash or the InfoSig.
    If an InfoSig exists, use that instead of InfoHash.
    InfoSig implies a pseudonymous user has uploaded data that may be mutated in the future.
    InfoHash is an immutable store of a value.

With thanks to:

  • WebTorrent Creators
  • Bram Cohen
  • Bitping Team
  • Hyperswarm Team

synapse's People

Contributors

firaenix avatar dependabot[bot] avatar

Stargazers

Andrejs Agejevs avatar

Watchers

James Cloos avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.