cosmicboots / phoenix Goto Github PK
View Code? Open in Web Editor NEWA proof-of-concept file synchronization system
A proof-of-concept file synchronization system
The client needs to be able to seek to a custom position in a file, verify the data using the given chunk id (hash) and bundle it to be sent over the network.
Options for structuring messages between each thread:
messaging
messages and each client thread forwards the message to the clientFile deletions need to propagate through the network.
This would improve lazy file transfers by allowing the server/client to reference a request ID rather than include the entire file metadata for each chunk requests. For the client to be able to find and send a specific chunk from the file system, it needs the file metadata and the chunk id. This is a lot of duplicated information being transmitted across the network when the client/server should already have the information needed.
This feature is subject to extreme scope modification. After some consideration, I would like to keep the client and server from having to keep a communication state as much as possible. I think the small sacrifice of adding extra context would be worth not undertaking the overhead of state tracking on the client/server.
Currently, chunks are sent from the client as soon as the file metadata is sent. This creates a situation where some chunks may not need to be sent to the receiver again. The solution is to request needed chunks anytime file metadata updates are sent across the system.
When a connection between the server and client is established, there needs to be a quick comparison of the current file tree states on either side. The client and server then need to be able to request/send the updated sections of the file tree to their peer.
This will need to be done using a separate table that hold the pending files. The main benefit of this would be keeping partial file transfers from being entered into the system.
file_table
rm_file()
will need to be duplicated to properly remove each chunk reference for each file in the pending tablefile_table
and the pending_table
Chunks from the server will need to travel in the form of a QualifiedChunk
so the client knows what file the chunk is associated with.
TODO:
The Noise protocol only allows 65535 byte messages (http://noiseprotocol.org/noise.html#message-format). Because of this, there needs to be a way to break up network messages.
Originally, I had been under the impression that file chunks can, and should, always fit inside this packet cap; however, I don't think limiting file chunks to ~60kb is a good idea.
To solve both these problems, the network layer needs to be able to send multi part messages so these larger pieces of data can fit inside individual noise packets.
The call to io::copy()
is specifically taking forever. A faster method needs to be found.
The client needs to be able to request and process FileMetadata
from the server.
To keep the filesystem watcher from triggering partial file transfers to be sent back to the server, a blacklist will need to be used. The blacklist should contain the paths for each file that's partially transferred.
Here's an example sequence diagram explaining the blacklist functionality:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.