denismo / dynamofs Goto Github PK
View Code? Open in Web Editor NEWLinux FUSE file system implementation with AWS DynamoDB as the storage
License: GNU General Public License v3.0
Linux FUSE file system implementation with AWS DynamoDB as the storage
License: GNU General Public License v3.0
DynamoFS storage is quite expensive, while S3 has reasonable price.
S3 allows parallel uploads which can provide data transfers faster than DynamoFS fastest option.
Also, storing large files (GBs) via 64k blocks is very inefficient.
This feature will store data in S3:
This is to do with permitPrivilegedOnly. On one hand, one test executes chown under unprivileged user without changing ownership but expects EPERM. On the other hand, other test expects not to get EPERM is ownership does not change
This has primary advantage in that there are less blocks to read/write for any operations, which is easier to keep consistent. It'll also be faster as only 1 read is necessary for traversals, and for read/write operations the number of reads/write won't change.
By definition of truncate, it can set the file size to be large than before. The new bytes are filled with zeros. In our implementation, we don't allocate missing blocks so we need to make sure that read treats missing blocks as filled with zeros.
If we are moving a file while it is being updated by chmod, chown etc - simple attribute update - then it'll create 2 copies of the record as chmod's save will restore the deleted record,
At the moment Fuse is configured to make all request from one thread. This prevents some concurrency issues in FS implementation. Once those issues are resolved (e.g. #34) we can enable multi-threaded access.
The readme says "execited" instead of "executed" in the "Highly Concurrent" section.
At the moment the write performance is very slow. Copying a file of 90K takes about 30 seconds. This is due to the multiple write requests being performed by fuse library with 4K blocks. Every request requires read/write against Dynamo which is very slow.
At the moment the blocks are cached indefinitely which will cause memory overflow. Need to implement purging of blocks out of cache.
For example, chown only allows changing to the groups that the user belongs. If the user is in multiple groups, he can change between them. However, the fstest will fail (chown/00.t, 36) because FS can only get the number of the first group.
All previous tests were done with the version pjd-fstest-20080816.tgz, released on August 16, 2008.
There is a newer version which has better coverage: pjd-fstest-20090130-RC.tgz, released on January 30, 2009.
There is a corresponding issue against fusepy. This is just a reminder that one that issue is fixed something needs to be done here as well.
With the new design and improved concurrency properties we may not need the block cache anymore. Previously the first block had medata so that was one of the reasons to cache. Also, there were no concurrency checks.
At the moment the size of the file is supposedly stored in 32-bit integer.
Also, if the file is big, there will be many blocks however there is no pagination in Dynamo requests.
There are several operations (mostly rename, link) which update block with link count. It's done using +1 in code which may leave the blocks inconsistent if there is a concurrent update. Instead, they should put "increment" operation which can be executed in parallel without conflict.
Sometimes there are Fuse calls for release/lock against the file that we never saw before. LockManager fails in that case as there is no fileLock entry in dict. Need to either ignore failure (and create) or do nothing.
Also need to think about automatic purge of old fileLock entries.
More than half of the tests are failing. The major areas are rename, open, mkdir.
Due to whatever reason the operation succeeds but there is an error reported by Fuse:
root@ubuntu-VirtualBox:/mnt/pc/dynamo-fuse# ln -s /mnt/dynamo/aaa /mnt/dynamo/s_aaa
fuse: bad error value: 8
ln: failed to create symbolic link `/mnt/dynamo/s_aaa': Numerical result out of range
At the moment the table has to be pre-created manually, and it must have particular structure. It would be better if the application created (if absent) the table by itself with the right structure.
MVCC allows to implement reads/writes without read/write locks which is tempting as those are additional roundtrips on Dynamo
In truncate, there is a query for the tail blocks which uses GT(str(lastBlock)). As the range key is String, this will not produce the correct result. One workaround would be to format all numbers with head zeros up to the limit on the number of blocks
At the moment only the simplest mechanism via environment variables is supported. This needs to be extended to support all AWS mechanisms.
This is imagined to be a common use-case
It is obvious from the logs that Fuse performs some operations several times in a row. For example, getattr is called multiple times for the same file.
This presents significant opportunities for optimisation by caching, especially for any simple records like directory or file descriptor (but not data).
This will introduce a version field which will be incremented every time a write is made. It'll be verified against the current version to ensure that no concurrent modification happened. If it happens we need to figure out whether we can write automatic transaction retry mechanism.
Is this project still under active development, or has it been abandoned?
Documentation currently lists the use of python-fuse as requirement but the package python-fuse is not available under pip (RH6 based) Is this something that is specific to an OS distro? Documentation references yum, so the assumption is made that it is available for redhat based OS's but the libraries do not exist. Also what versions of python is required to run the software?
When hard-links are created they just inherit the blockId of the source. If the source is deleted it will delete the block as well eventhough there may be hard links.
At the moment there is no file locking mechanism
Throwing exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 448, in _wrapper
return func(_args, *_kwargs) or 0
File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 488, in rename
new.decode(self.encoding))
File "dynamofs.py", line 51, in call
ret = getattr(self, op)(path, *args)
File "dynamofs.py", line 166, in rename
item.hash_key = new
AttributeError: can't set attribute
If a client system which took a look on file dies no other system will be able to lock its files. Worse, if the system restarts the in-memory lockId will be gone and the system itself won't be able to unlock.
Need to implement lock recovery - if the system is identified to be dead the locks should be released. It can be done by some other system, based on "last client check-in" time.
Linux filesystems allow the files which are being written/read to be moved or even deleted without causing errors. This is probably because an open file creates a hidden hard link to the inode (not persisted) which keeps the inode up while the file name is removed.
We need to implement it in some way preferably without locking as I believe this mechanism is used by log rotators/archivers.
All the tests which are executed with specific user: fstest -u xxx -g xxx, - fail with EACCESS, but no call to the library is made
At the moment even though a symlink can be created, it is not being looked at by any other routines. Subsequently, reading or writing to a symlinked file may not work
There are several comments where pagination is mentioned. Need to implement it otherwise we won't support folders of many items or large files
A simple improvement - instead of incrementing it by one increment it by chunks (100) and use this amount within the same process until it expires.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.