sergey-dryabzhinsky / dedupsqlfs Goto Github PK
View Code? Open in Web Editor NEWDeduplicating filesystem via Python3, FUSE and SQLite
License: MIT License
Deduplicating filesystem via Python3, FUSE and SQLite
License: MIT License
PgSQL has really optimized storage format.
Checked via migration 13kk rows of Zabbix DB from MySQL InnoDB: 55Gb into 2Gb.
The goal of plan
is to keep first yearly snapshot, first interval
snapshot.
So if plan
is 2y,6m,8w,14d
then it must contain:
2 years
interval6 month
interval8 week
interval14 days
intervalAll of them cat intersect with each other.
Need to recalculate time-line to keep right amount of them.
If file was truncated down - blocks/index not deleted, block data not zeroed.
Indexes flushing only by defragmentation.
But by block data not truncated and block sizes not adjusted - subvol stats are incorrect (sparse sizes).
Need to check inode size change. If it possibly truncated - than recalc block count, last block size and truncate data to new size.
-O3
with -O2
-march=native
Add posibility to config block size per inode, except directory, special files.
To call snapshooting, vacuum, and other commands then FS is mounted.
The trick is to not mess with threads and fuse operations and Sqlite.
Try to:
do
commandNeed to add subvolume selection, or defragment & vacuum all subvolumes.
And store last defragment, vacuum date in subvol table.
Use latest stable version.
And this one can be build w/o legacy formats support.
Probably others too: 0.6.1, 0.4.7.
v0.3.6 must remain with legacy support.
Gzip tree, inode, block-index tables with gzip after snapshot creation.
This can make snapshooted data size be lesser by 75%.
This need to be supported by defragment, table open/close code.
Extract all snapshot data into other copy of dedupsqlfs like a new fs.
Just force data to flush on disk, keep it in memory cache.
Expire blocks from cache only on forget
, unlink
, rmdir
operations.
Or by inactivity time.
Or by cache size limits.
Change hashing algo for FS and recalculate all hashes with it.
There is two ways for handle sparce areas in files:
If FS is mounted or running some do
action - it must be locked.
Many incompatible changes in 0.42 version: https://github.com/python-llfuse/python-llfuse/blob/master/Changes.rst
Goes into 1.3 version of DDSQLFS.
For now it uses gzip program and .gz extension.
Need to support different programs and compression levels:
As soon as I update my python-zstd project
It was designed to just change files name for table storage.
Not table name inside that file.
So it can be copied back.
But sometimes things get worse: indexes may get doubled after making snapshot.
--compress
as --compress-custom
.all_
method - use last of them.all_fast
is all methods with fast
level activated.all_best
is all methods with best
level activated.--compress=zlib:9
or --compress=zlib:fast
.=
to :
It's needed for packaging.
Add options for host, port, db, user and pass.
For example lz4, zstd, lzo can be really fast.
And multi-process compression waste more time in inter-process communication.
Need to test if multi-threaded version will be faster.
Add:
Some parts/modules of DdSF can be compiled by cython:
This can be used by packaging.
Create "recompression" action in "do" app to "remove" not needed compression algo from fs. Don't forget to NULL all subvol stats after that.
Add possibility to install without it.
Do not waste resources if CPU is fast enough
Quicklz compression level is choosen at compilation time.
What need to do:
To prevent DB damage on mount or other action.
Sometimes it's just not needed.
Only if files big ehough - like tree,inode or index tables if:
What is means:
Like --cleanup-old-snapshots-by-plan=14d:8w=1:18m=1:3y
, so it will keep:
2017-05-21 05:05:17,751 - DedupFS - ERROR - Traceback (most recent call last):
File "/opt/rusoft/dedupsqlfs/dedupsqlfs/fuse/operations.py", line 765, in read
data = self.__get_block_data_by_offset(fh, offset, size)
File "/opt/rusoft/dedupsqlfs/dedupsqlfs/fuse/operations.py", line 1480, in __get_block_data_by_offset
block = self.__get_block_from_cache(inode, n + first_block_number)
File "/opt/rusoft/dedupsqlfs/dedupsqlfs/fuse/operations.py", line 1399, in __get_block_from_cache
self.getLogger().debug("-- db size: %s" % len(item["data"]))
TypeError: 'NoneType' object is not subscriptable
Do not collect garbage in hash+blocks - should be faster gc.
What need to be achieved:
There may be overhead by inter-process data movement.
File with base parameters to use without bunch of commands-line options.
Make mysqld server startup and stop separate commands or options for do/mkfs commands.
Goal:
It can save about 10-30 seconds on each operation due disabled start-stop of server.
Add posibility to config compression methods per inode, except directory, special files.
On some looong rsync processes filesystem consume memory and ... stops.
To gradualy drop caches we need to touch root of filesystem to provide any event (setattr).
It will trigger cache cleanup procedures.
Just create Thread that do os.utime(mountPoint, None)
every second.
Until fuse destroy.
Need to:
Should use distutils module distributed with python.
LZ4 & ZSTD modules affected.
Problem is that module not exists in old distros like Debian Wheezy, Ubuntu Lucid.
To use new compression levels.
And update statistics only if subvol FS was modified.
Current root subvol can gather stats on umount. Need option. If modified.
This is for snapshot statistics output speed up, subvolumes list stats speed up.
Stats:
If you move directory to other location of FS then all its content disappear!
http://www.sqlite.org/partialindex.html
If sqlite3.sqlite_version >= 3.8.0 - migrate to partial indexes.
They are smaller and faster (in theory).
Add --decompress-tryout=method1:method2,method3:method4,...
.
Enable instead --decompress-try-all
to speedup decompress on errors.
Behaviour:
Should be a little more faster. In some cases.
Don't use dict
to store 2-4 values in end leaf of caches.
Use list
. It 3 times less size in memory.
Not so pretty but can reduce memory footprint.
Need to change db structure:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.