vimpunk / cratetorrent Goto Github PK

Sometimes occurs when running tests/test_multi_connection_download.sh --size $(( 50 * 1024 * 1024 )).

[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Piece 761 intersects files: 0..1
[2020-11-14T19:20:55Z DEBUG cratetorrent::torrent] Disk write result Ok(BatchWrite { blocks: [BlockInfo { piece_index: 760, offset: 0, len: 16384 }, BlockInfo { piece_index: 760, offset: 16384, len: 16384 }, BlockInfo { piece_index: 760, offset: 32768, len: 16384 }, BlockInfo { piece_index: 760, offset: 49152, len: 16384 }], is_piece_valid: Some(true) })
[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Disk received command
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Saving torrent 0 block BlockInfo { piece_index: 761, offset: 16384, len: 16384 } to disk
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Saving block BlockInfo { piece_index: 761, offset: 16384, len: 16384 } to disk
[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Disk received command
[2020-11-14T19:20:55Z INFO  cratetorrent::torrent] Finished piece 760 download, valid: true, left: 38
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Saving torrent 0 block BlockInfo { piece_index: 761, offset: 32768, len: 16384 } to disk
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Saving block BlockInfo { piece_index: 761, offset: 32768, len: 16384 } to disk
[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Disk received command
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Saving torrent 0 block BlockInfo { piece_index: 761, offset: 49152, len: 16384 } to disk
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Saving block BlockInfo { piece_index: 761, offset: 49152, len: 16384 } to disk
[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Piece 761 is complete (65536 bytes), flushing 4 block(s) to disk
[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Piece hash: f1a938c768f7efbcc26b33a112bbf3ec381aa40b
[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Piece 761 is valid
[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Disk received command
[2020-11-14T19:20:55Z DEBUG cratetorrent::torrent] Disk write result Ok(BatchWrite { blocks: [BlockInfo { piece_index: 761, offset: 0, len: 16384 }, BlockInfo { piece_index: 761, offset: 16384, len: 16384 }, BlockInfo { piece_index: 761, offset: 32768, len: 16384 }, BlockInfo { piece_index: 761, offset: 49152, len: 16384 }], is_piece_valid: Some(true) })
[2020-11-14T19:20:55Z INFO  cratetorrent::torrent] Finished piece 761 download, valid: true, left: 38
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Saving torrent 0 block BlockInfo { piece_index: 762, offset: 0, len: 16384 } to disk
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Saving block BlockInfo { piece_index: 762, offset: 0, len: 16384 } to disk
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Creating piece 762 write buffer
[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Piece 762 expected hash 6292ef9ee99381ddd8fb524c64daa61047bbe88b
[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Piece 762 is 65536 bytes long
[2020-11-14T19:20:55Z TRACE cratetorrent::storage_info] Returning files intersecting piece 762
[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Piece 762 intersects files: 0..1
[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Disk received command
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Saving torrent 0 block BlockInfo { piece_index: 762, offset: 16384, len: 16384 } to disk
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Saving block BlockInfo { piece_index: 762, offset: 16384, len: 16384 } to disk
[2020-11-14T19:20:55Z DEBUG cratetorrent::disk::io] Disk received command
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Saving torrent 0 block BlockInfo { piece_index: 762, offset: 32768, len: 16384 } to disk
[2020-11-14T19:20:55Z TRACE cratetorrent::disk::io] Saving block BlockInfo { piece_index: 762, offset: 32768, len: 16384 } to disk
[2020-11-14T19:20:55Z WARN  cratetorrent::peer] [Peer 172.17.0.2:51413] timeout after 14935 ms (timeouts: 13)
[2020-11-14T19:20:55Z TRACE cratetorrent::peer] Making requests to peer 172.17.0.2:51413
[2020-11-14T19:20:55Z DEBUG cratetorrent::peer] Session 172.17.0.2:51413 starting new piece download
[2020-11-14T19:20:55Z TRACE cratetorrent::piece_picker] Picking next piece
[2020-11-14T19:20:55Z TRACE cratetorrent::piece_picker] Could not pick piece
[2020-11-14T19:20:55Z DEBUG cratetorrent::peer] Could not pick more pieces from peer 172.17.0.2:51413
[2020-11-14T19:20:55Z INFO  cratetorrent::peer] [Peer 172.17.0.2:51413] download rate: 34311 b/s (peak: 780141 b/s, total: 4128768 b) queue: 1, rtt: 616 ms (~0 s)
[2020-11-14T19:20:55Z WARN  cratetorrent::peer] [Peer 172.17.0.3:51413] timeout after 14951 ms (timeouts: 1)
[2020-11-14T19:20:55Z TRACE cratetorrent::download] Canceling request for piece 762 block BlockInfo { piece_index: 762, offset: 49152, len: 16384 }
[2020-11-14T19:20:55Z TRACE cratetorrent::download] Canceling request for piece 763 block BlockInfo { piece_index: 763, offset: 0, len: 16384 }
< snip>
[2020-11-14T19:20:55Z TRACE cratetorrent::download] Canceling request for piece 799 block BlockInfo { piece_index: 799, offset: 49152, len: 16384 }
[2020-11-14T19:20:55Z TRACE cratetorrent::peer] Making requests to peer 172.17.0.3:51413
[2020-11-14T19:20:55Z DEBUG cratetorrent::peer] Peer 172.17.0.3:51413 trying to continue download 784
[2020-11-14T19:20:55Z TRACE cratetorrent::download] Picking 1 block(s) in piece 784 (length: 65536, blocks: 4)
[2020-11-14T19:20:55Z DEBUG cratetorrent::download] Picked 1 block(s) for piece 784: [BlockInfo { piece_index: 784, offset: 0, len: 16384 }]
[2020-11-14T19:20:55Z DEBUG cratetorrent::peer] Peer 172.17.0.3:51413 trying to continue download 791
[2020-11-14T19:20:55Z INFO  cratetorrent::peer] Requesting 1 block(s) from peer 172.17.0.3:51413 (0 pending)
[2020-11-14T19:20:55Z INFO  cratetorrent::peer] [Peer 172.17.0.3:51413] download rate: 2593852 b/s (peak: 2593852 b/s, total: 45858816 b) queue: 1, rtt: 14627 ms (~14 s)
[2020-11-14T19:20:55Z TRACE cratetorrent::peer::codec] Decoder has 16371 byte(s) remaining
[2020-11-14T19:20:55Z TRACE cratetorrent::peer::codec] Read buffer is 16371 bytes long but message is 16393 bytes long
[2020-11-14T19:20:55Z TRACE cratetorrent::peer::codec] Decoder has 16397 byte(s) remaining
[2020-11-14T19:20:55Z DEBUG cratetorrent::peer] Received message 172.17.0.3:51413 from peer Some(Block)
[2020-11-14T19:20:55Z INFO  cratetorrent::peer] Received block from peer 172.17.0.3:51413: BlockInfo { piece_index: 784, offset: 0, len: 16384 }
[2020-11-14T19:20:55Z TRACE cratetorrent::download] Received piece 784 block BlockInfo { piece_index: 784, offset: 0, len: 16384 }
thread 'tokio-runtime-worker' panicked at 'supplied instant is later than self', library/std/src/time.rs:271:48
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[2020-11-14T19:20:56Z DEBUG cratetorrent::torrent] Torrent running for 26s

Technically here we should reject bitfield messages where the overhang part contains non-zero bits.

Originally posted by @mandreyel in #34

https://ryhl.io/blog/temporary-shared-mutation/

We should deterministically generate the test environment (test files, torrent files, Transmission seed dir structure and config files, etc), rather than saving them in git.

This will be important once we test downloads with larger files in which case we won't want to blow up the git repo in size, and it would also help to avoid bugs where a required test file was forgotten to be added.

The problem is that as far as I can tell it is not possible to generate a metainfo in an automated way. Maybe with docker exec.

Some module level docs would be nice.

Originally posted by @mandreyel in #34

Currently the disk module has tests to tests its high-level API, but it would be ideal to test each component of disk::io too for higher coverage.

Unit and integration tests.

Currently unsure how to do this with the container based approach. Maybe it's possible to block container links for a short time?

As for unit tests, PeerSession is not currently structured in such a way as to allow for easy unit testing of its logic. That is probably a larger issue, not related to only this.

The periodic (1s) torrent tick needs to do the following:

• remove finished peer sessions (#64)
• request more peers from tracker (#63)
• should connect peers? (#64)
• collect run times:
  • total
  • downloading
  • seeding
• send alerts to API user (#52)
• unchoke peers (#60)
• save resume data (TBD)
• request rate limit (TBD)

Currently the torrent doesn't do much with the peer sessions it has. This is partly because there is no existing infrastructure for a peer to report its state to its torrent.

This is crucial for a few reasons: if torrent requests peers from its tracker, the peers returned are not guaranteed to be seeds, so the torrent may end up in a situation where it is stuck. Therefore it needs to periodically check (in its "torrent tick") whether we need to disconnect idle peers and request new ones to drive forward the download.

We can start at 1.0.0-alpha.1 (https://codeblog.jonskeet.uk/2019/10/20/why-i-dont-start-versions-at-0-x-any-more/).

Currently the disk::io::Piece::write method takes a path and reopens the file with each write. We should open file once and keep around its handle and use it for writing all its pieces.

Currently cratetorrent unchokes a new peer unconditionally. This needs to change as upload slots need to be capped. Otherwise cratetorrent might take up most of host's upload bandwidth.

Currently it's enforced that all blocks be 4 KiB, but this may not be the case for the last block in the torrent if it's total download length is not a multiple of 4 KiB.

• inactivity
• hasn't become interested in too long

TODO: expand list

This issue tracks potential ways to speed up downloads.

• Request pipelining. Done to a basic extent (fixed pipeline size).
• Self-adjusting request queue size. We should always keep the download pipeline at capacity to ensure we don't waste bandwidth.
• TCP like slow-start to reach optimal request queue size as fast as possible. The problem here is that if we linearly increment the request queue size, for fast links we would waste performance instead of ramping up the download rate exponentially.
• Batch disk writes. This is done for the most part but configurability and maybe self-adjustment would be great.
• Request timeouts, which are important when there are multiple peers with the same piece. If a peer has it but times out, we can try to download it from another peer faster. (#18)
• Share downloads. This way, a piece may be completed faster if downloaded from multiple peers, resulting in fewer unfinished pieces overall. It's also important for end game mode. (#45)
• End game mode: here for the last n pieces (for a small n, possibly n = 1), we request all blocks from all peers (that have it), so that we're not stalled on a single peer to finish the download. If we receive a block, we send a cancel request to other peers. (#70)

Currently these are stored in two places but they are very much related. So when we get a block, we find its corresponding request entry in the request queue and then find the corresponding piece.

Maybe:

downloads: HashMap<PieceIndex, Download>,

struct Download {
    piece: PieceDownload,
    requests: Vec<BlockInfo>,
}

This would protect the client against getting DoS-ed.

Only (non-positional) scatter-gather IO is supported in the std lib via the Write::write_vectored trait interface, but this is not good enough for our use case as we'll need to perform 2 syscalls for every write: seek and writev.

• Rust stdlib feature request: rust-lang/rust#71055
• Possible workaround until then: https://github.com/vasi/positioned-io/tree/positioned-io-preview [EDIT]: this doesn't work as it only supports positional IO, not scatter-gather IO.
• or: https://docs.rs/nix/0.17.0/nix/sys/uio/fn.pwritev.html

It looks like github actions are free? If so, we should set up one to run unit tests and a few integration tests (not all to save time as we'd presumably capped).

Because there is no real reason to do so: cratetorrent doesn't have much config files so there is nothing to contain as with transmission, and everything else is configurable (listen address, download directory, as of this writing).

However, there are downsides to maintaining the dockerfile (e.g. openssl seems to break from time to time).

Currently we check after every block, which is needlessly expensive.

We can use https://github.com/fdehau/tui-rs, it's incredible.

Place limit on in-memory write buffer to not cause OOM. This should be configurable.

We currently notify torrent that this occurred (so that after n failures it may stop torrent and further propagate failures to user), but we should also not erase the piece write buffer and retry writing later.

Currently not an issue but we shouldn't pull all of tokio in anyway.

instead of allocating a new vector every time

Record round trip type, downloaded bytes per second, etc.

Pre-alpha

• Perform a single in-memory download of a file with a single peer connection if given
  the address of a seed and the path to the torrent metainfo. No multiple
  torrents, no seeding, no optimizations, or any other feature you might expect
  from a full-fledged BitTorrent library.
• Extend previous addition with actually saving the downloaded file to disk after
  verification.
• Download a directory of files using a single peer connection.
• Download a torrent using multiple connections.
• Optimize download performance to use self-adjusting optimal request queue
  sizes and slow start mode for ramping up download throughput (see #40).
  • Add request timeouts (#18)
  • Share a piece download (#45)

Alpha release: 0.1.0

• Seed a torrent (#50)
• Get peers from an HTTP tracker (#51)
• Torrent's peer management (#64)
• Simplified notification system + public API (#52)
• Improved API docs (#56)
• Update dependencies (tokio 0.3 in particular -- [update]: can't update tokio yet, see #76)
• Decide whether to use simplified 0.x.y semver or with pre-release tag. Former is simpler, latter more correct (though the semver spec covers the 0.x.y use case) while being more complicated for little gain: we don't have parallel releases that rely on strict semver.
• Fix inconsistencies (import stye, namings, etc) and clippy lints
• Read cache upper bound (#22) or remove entirely for mvp
• Basic TUI for CLI app (#74)
• Use simple non-tui CLI binary for tests as it breaks seeding
• Cargo release
• Remove unused benches
• Link to docsrs in readme, and other fancy badges

0.1.x

• Make cross-platform (#88)
• UDP tracker support (#97)
• License files (#91)

Priority

• DHT
• magnet links

Future features not yet scheduled

• persisting torrent metadata to continue them after restart
• disk-network IO backpressure to optimize performance and stability
• UPnP
• Fast Extension
• stream encryption
• uTP sockets
• file padding
• BT2.0
• rate limiting
• file priorities
• piece picking strategies (sequential, rarest first, other)
• make storage backend system agnostic/generic

The goal is simply to exchange a handshake and process a few messages from the peer. The focus is not a full download, as saving the file is part of the next milestone.

• binary message protocol
• basic PeerSession struct that holds peer related information and peer protocol methods
• perform handshake with a peer
• handle other messages from peer and reply with our own messages
• basic manual testing to assert that client protocol is "roughly ok"
• request and receive blocks
• fully download torrent into memory (without verifying it or saving to disk)

Currently PeerSession::run just returns with an Ok(()).

Although currently there is no real 'engine' entity, we still want to communicate two things:

• torrent sending a "DownloadComplete" event to engine
• engine's ability to send a "Stop" command to torrent

These can be used to more cleanly implement stopping the torrent than how it is currently done.

Later of course this will expand to many more types of commands engine can send to torrent, and events torrent can send to engine.

It is a bit puzzling why this is happening. There are sometimes peers that keep sending not requested blocks.

It is not yet clear whether this is some mismanagement in cratetorrent or due to malicious peers. Probably the former.

Right now we necessitate peer sending us a bitfield as we only support downloading from seeds. However, once we support seeding, our peer may not have any pieces in which case it won't send a bitfield, and this would cause us to be stuck in the piece availability exchange state.

While this simplifies the logic a great deal, it uses up more resources without much need.

Instead of looping over all blocks every time the function is called.

For now, until seeding is supported.

Disk IO

• read specified block from disk
• read cache, read cache lines (pull in more blocks with one read)
• handle concurrent requests for same block with "read faults"

Peer

• handle request messages to upload block data
• handle cancel message to cancel block read:
  • probably can't cancel in-progress disk IO
  • but shouldn't upload block that was cancelled.
  • use cancel buffer and check presence before uploading after successful disk read?
• inbound connections:
  • peer session inbound constructor
  • handle inbound handhsakes
  • remove all checks that peer is not a seed (previously only downloading from seeds was supported)
  • if we have pieces, send bitfield message
• send have message to non-seed peers

Torrent

• conditionally allocate torrent on disk
• accept new connections
• unchoking peers: simplified or full unchoke algorithm? -> out of scope, we unchoke unconditionally for now (#60)
• stop torrent alert (sent by engine) separate MR, out of scope (#62)

Engine & CLI

• parameter to seed a file: cli arg with corresponding option in engine
• 'torrent complete' alert from torrent to engine, stopping a download session separate MR, out of scope (#62)

Integration tests

Use cratetorrent for seeding and downloading as we already have test infrastructure to easily set up and verify cratetorrent downloads. As a next step we could have leech tests for e.g. transmission to ensure compatibility, but probably not before alpha release.

• set up 1 cratetorrent peer to seed a file
• set up 1 cratetorrent peer to download same file (can reuse existing test for the most part)