VisTrails/VisTrails#755

Implementing the backend with MongoDB shouldn't be hard. This should probably be done and compared to the SQLite3 backend (and other SQL servers?)

Options should be available to control what print/query output: hashes, metadata, contents...

It is possible to craft a file that will have the same hash as a directory, by writing the text representation of the directory that is used for hashing internally.

Example:

TEMPROOT=/tmp/store.$$.$RANDOM
mkdir $TEMPROOT
cleanup(){
    rm -Rf $TEMPROOT
}
trap cleanup 0
STORE=$TEMPROOT/store
BIN="python -m file_archive"

# Adds directory
mkdir $TEMPROOT/testdir
echo foo > $TEMPROOT/testdir/foo
echo bar > $TEMPROOT/testdir/bar
$BIN $STORE create
$BIN $STORE add $TEMPROOT/testdir

# Adds carefully-crafter file: this will fail with:
# KeyError: 'This file already exists in the store'
cat > $TEMPROOT/testfile<<END
file foo f1d2d2f924e986ac86fdf7b36c94bcdf32beec15
file bar e242ed3bffccdf271b7fbaf34ed72d089537b42f
END
$BIN $STORE add $TEMPROOT/testfile

Currently it is assumed that we will never bump into the same file again, or that if we do, it will be the same file with the exact same set of metadata.

Something should probably be provided to handle the case when we have different metadata.

The possible operations are:

• Joining different sets of metadata (would occur went syncing?). I don't think this is what we want to do.
• Considering the file as different if it has different metadata (i.e. hash metadata+contents)

Right now the system only stores files.

There are two options for storing directories:

• Do the same thing we do for files, hash it recursively and copy it to filename objects/filehash
  • Pros:
    • Easy
    • Keeps file structure (the user needs the system to obtain the path, but then he can access it in place)
  • Cons:
    • File duplication is possible
• Do something similar to git, making a tree object that references other tree or blob objects
  • Pros:
    • No more duplication!
  • Cons:
    • Need for a GC (this can be made efficient by putting the SQL database to use)
    • Need to process the files to turn it back into exploitable format
    • Harder

file_archive path/to/store query age=str:two

If a file-like object is given to add_file, it will be read once, rewound using seek(0, SEEK_SET), and then read a second time. This is because it is hashed first and then copied to its destination.

I don't know if this is optimal. Maybe it should be read only once while both hashing it and writing it to a temporary file, and then moved to its destination.

viewer.py can probably be made to support both PyQt4 and PyQt5.

API is taking shape, but we should be able to add, remove and query files from the command line.

Maybe some special logic should be implemented for symlinks, right now the target would be copied. Make it an option? Show a warning?

Coverage as of 4cfe3e6:

Missing:

• Coverage for directories
• Tests for database module
• Tests for command line interface

https://github.com/spyder-ide/qtpy

The current storage scheme has the following problems:

• (C) Collisions: different entries (= sets of metadata) with the same content won't be accepted (issue #10)
• (F) File name selection: Matthias wants filenames to be selectable (or changeable)

Other options that come to mind are:

• (M) Changing metadata for an entry
• (D) Deduplication (store files once if two entries have the same content)

The schemas I have in mind are:

1

hash(content) <-> filename
               -> metadata

• can rename file
• can add/change metadata
• two files with the same hash will share metadata

2

hash(content||metadata) <-> filename
                         -> metadata

• can rename file
• can't change metadata
• no collisions
• no deduplication

3

uuid <-> filename
      -> metadata

• can rename file
• can change metadata
• no collisions
• no deduplication

4

uuid -> hash(content) <-> filename
     -> metadata

• can rename file (affects multiple entries)
• can change metadata
• no collisions
• deduplication

5

hash(content||metadata) -> hash(content) <-> filename
                        -> metadata

• can rename file (affects multiple entries)
• can't change metadata
• no collisions
• deduplication

6

hash(content||metadata) <-> filename
                         -> metadata

• can rename file
• can't change metadata
• no collisions
• no deduplication

Right now:

Legacy

hash(content) = filename
              -> metadata

• can't rename file
• can change metadata (not implemented)
• collisions possible

Probably related to this:
http://stackoverflow.com/questions/15834628/how-to-specify-explicit-python-packaging-dependencies-in-setup-py

Sending some of the files to another store needs:

• A way to select what to send (i.e. a query)
• Transfer the files (feeding thefilenames to rsync, scp, ftp...)
• Merge the databases

This works similarly to query, so is the correct behavior, however it is dangerous. A --force should probably be needed when no condition is passed (similar to rm's --preserve-root).

Add a way to select entries for which a certain metadata key, with a certain type, exists, regardless of its value.

• API support (31f2aec)
• test coverage (6672d8f)
• command-line support
• viewer support (28fa27f)