unixjunkie / daft Goto Github PK

process Sys.argv before giving control to the Arg module to extract
the command entered by the user in case there is no -i in the options

given a machines file, daft should be able to restore a working state in case
none of the former DS deleted its local datastore upon exit

get for a remote DS

if the user called daft with -z: set the boolean in case
the file is smaller after compression.
Otherwise, unset the bool and act accordingly.
This is an optimization so that we disable compression automatically
for files that do not compress.

this is discussed in Ludovic Courtes' thesis:
http://ethesis.inp-toulouse.fr/archive/00000544/01/courtes.pdf
This is a little complex and probably CPU hungry, however it allows to do file
deduplication: variable-size file chunks can be shared by several files.

and force all host options to be of the form host:port

put working even with a directory

this needs to work with socket_wrapper.ml:

• fix the compress flag to true in that way if --compress configure option
• fix the encrypt flag too if --encrypt
• fix the sign flag too if --sign

CLI should only have a partial vue of message types:
especially its send function should only know about
CLI_to_* types.
It's receive function should only know about
*_to_CLI types.

might be fun for the data transfer part

put for a remote DS

At send time:
unique_id = host ^ ":" ^ port ^ ":" ^ message_num
put it in a StringSet
at reception time:
look for unique_id into the StringSet
create a None and warn about it in case the message
was already seen
else give the message to the rest of the pipelin

an indirect chunk is a reference to a real chunk.
they will be used by files created with cat
and if we implement file chunk deduplication, some day

because having this env. vars setup will allow for much shorter command lines

we need to think if and how to fix this

• we need to continue increasing message numbers so that nonces are still OK
• we need to store the message number on disk upon exit

cat f1 f2 f3 dst_file <=> bash shell: cat f1 f2 f3 > dst_file
this operates only on metadata, it should be super fast

I start to see a way out: Huff + OTP
Use it if faster than AES(LZ4(cleartext))

crashes the local DS

If message signature was wrong for example
this allows all exe to ignore them

upon exit: destroy/delete the local data store in case the user gave this option

will be faster, plus is less stupid since clear text nonces can easily be faked by anyone

this suggests we should have a source-based installer script for it

after quit from the CLI, the remote DS was not shut down

i.e. except ls result and file chunks
--> then check it is OK for perfs

it crashes on files with > 1 chunk, the last one being smaller than the regular chunk size

get working even with a directory

do it before signing

then do less data copy when encoding and decoding

like we did for PAR: the wrapper could setup LD_LIBRARY_PATH
to the user's directory where ZMQ and LZ4 were installed

when a client want to wait for a file to appear in the system

turn it off if inflating message instead of shrinking

1. we need a data set
2. then a program doing random tests with it (choosing a command randomly then
   choosing more or less randomly the command parameters)

If it can run long enough, it will probably do a full code coverage or discover
bugs if stopping prematurely

to identify which chunks are here.
Use the bitv library from JCF.

encrypted or not
signed or not

"binprot will not crash on illegal input"
I just hope it will throw a nice exception we can catch in that case then

when the mds is started, it should start all DSs too since it has the machines file
this is a sequence of scp then ssh