cosmicboots / phoenix Goto Github PK

The client needs to be able to seek to a custom position in a file, verify the data using the given chunk id (hash) and bundle it to be sent over the network.

• Add a message channel between every thread
• Store message channels in a vector
• Upon new file completion, send message to all threads

Options for structuring messages between each thread:

• There's a dedicated message struct for messages to each client connection thread
• The messages to the client connection threads are the same as the messaging messages and each client thread forwards the message to the client

File deletions need to propagate through the network.

This would improve lazy file transfers by allowing the server/client to reference a request ID rather than include the entire file metadata for each chunk requests. For the client to be able to find and send a specific chunk from the file system, it needs the file metadata and the chunk id. This is a lot of duplicated information being transmitted across the network when the client/server should already have the information needed.

• Client needs request state tracking
• Server needs request state tracking

This feature is subject to extreme scope modification. After some consideration, I would like to keep the client and server from having to keep a communication state as much as possible. I think the small sacrifice of adding extra context would be worth not undertaking the overhead of state tracking on the client/server.

Currently, chunks are sent from the client as soon as the file metadata is sent. This creates a situation where some chunks may not need to be sent to the receiver again. The solution is to request needed chunks anytime file metadata updates are sent across the system.

• Server needs to determine what chunks it needs to complete the file
• Server needs to request these through the network
  • The chunks need to have a file path associated with them
• The client needs to queue up the chunks to transfer
  • Needs to be able to check file changes between chunks being sent

• Server needs to calculate which chunks are new
• Properly update reference counter table to reflect update to chunk changes
• Request only updated chunks from client Moved to later feature

When a connection between the server and client is established, there needs to be a quick comparison of the current file tree states on either side. The client and server then need to be able to request/send the updated sections of the file tree to their peer.

• Create file tree FileList struct to the network abstraction layer
• Server needs to be able to generate a file tree
• Client needs to be able to generate the file tree
• Network protocol needs to be updated to transmit the file listings
• Client needs to be able to compare its file tree with the server's

This will need to be done using a separate table that hold the pending files. The main benefit of this would be keeping partial file transfers from being entered into the system.

• Server needs to have its own pending file table
• New file metadata should be inserted into the pending table
• Once the server has verified that all the file chunks have been inserted into the database, the file metadata should be moved to the file_table
• Add a cleanup routine on startup that drops the pending table
  • Some of the behavior from rm_file() will need to be duplicated to properly remove each chunk reference for each file in the pending table
  • Might need to bust chunk removal out into a helper function...?
• Update file update logic to properly handle both the file_table and the pending_table

Chunks from the server will need to travel in the form of a QualifiedChunk so the client knows what file the chunk is associated with.

TODO:

• Test chunks being sent out of order
• Determine when the file is completely downloaded so the blacklist entry can be removed.

The Noise protocol only allows 65535 byte messages (http://noiseprotocol.org/noise.html#message-format). Because of this, there needs to be a way to break up network messages.

Originally, I had been under the impression that file chunks can, and should, always fit inside this packet cap; however, I don't think limiting file chunks to ~60kb is a good idea.

To solve both these problems, the network layer needs to be able to send multi part messages so these larger pieces of data can fit inside individual noise packets.

The call to io::copy() is specifically taking forever. A faster method needs to be found.

The client needs to be able to request and process FileMetadata from the server.

To keep the filesystem watcher from triggering partial file transfers to be sent back to the server, a blacklist will need to be used. The blacklist should contain the paths for each file that's partially transferred.

Here's an example sequence diagram explaining the blacklist functionality: