sergey-dryabzhinsky / dedupsqlfs Goto Github PK

PgSQL has really optimized storage format.
Checked via migration 13kk rows of Zabbix DB from MySQL InnoDB: 55Gb into 2Gb.

The goal of plan is to keep first yearly snapshot, first interval snapshot.
So if plan is 2y,6m,8w,14d then it must contain:

• at least 2 yearly snapshots, and 1 oldest in 2 years interval
• at least 6 monthly snapshots, and 1 oldest in 6 month interval
• at least 8 weekly snapshots, and 1 oldest in 8 week interval
• at least 14 daily snapshots, and 1 oldest in 14 days interval

All of them cat intersect with each other.

Need to recalculate time-line to keep right amount of them.

If file was truncated down - blocks/index not deleted, block data not zeroed.
Indexes flushing only by defragmentation.
But by block data not truncated and block sizes not adjusted - subvol stats are incorrect (sparse sizes).

Need to check inode size change. If it possibly truncated - than recalc block count, last block size and truncate data to new size.

• Replace -O3 with -O2
• Remove -march=native
• Add basic fortification flags

Add posibility to config block size per inode, except directory, special files.

To call snapshooting, vacuum, and other commands then FS is mounted.
The trick is to not mess with threads and fuse operations and Sqlite.

Try to:

• start RPC-socket-server in separate thread
• use pipe/queue in main thread to get commands from RPC
• trigger queue run by trigger fuse operation like file creation or stat change
• use socket in do command

Need to add subvolume selection, or defragment & vacuum all subvolumes.
And store last defragment, vacuum date in subvol table.

Use latest stable version.

And this one can be build w/o legacy formats support.

Probably others too: 0.6.1, 0.4.7.
v0.3.6 must remain with legacy support.

Gzip tree, inode, block-index tables with gzip after snapshot creation.
This can make snapshooted data size be lesser by 75%.

This need to be supported by defragment, table open/close code.

Extract all snapshot data into other copy of dedupsqlfs like a new fs.

Just force data to flush on disk, keep it in memory cache.

Expire blocks from cache only on forget , unlink, rmdir operations.
Or by inactivity time.
Or by cache size limits.

Change hashing algo for FS and recalculate all hashes with it.

• List of directories/files that:
  • created
  • changed
  • deleted
• One file data chages in diff/rdiff format
• All data changes in diff/rdiff format

There is two ways for handle sparce areas in files:

1. If block full of zeroes - then don't hash, compress and write it.
   Still, write block size to db.
2. If block has many zeroes at block end, at least more than 1024b or 10% - then hash, compress and write only valuable bytes.
   Write "real", with zeroes, block size to db.

If FS is mounted or running some do action - it must be locked.

Many incompatible changes in 0.42 version: https://github.com/python-llfuse/python-llfuse/blob/master/Changes.rst

Goes into 1.3 version of DDSQLFS.

For now it uses gzip program and .gz extension.
Need to support different programs and compression levels:

• gzip / pigz - .gz
• bzip2 / pbzip2 - .bz2
• xz / pxz - .xz
• zstd / pzstd - .zst
• lzop, lz4, etc...

• add "isDeprecated" to compressors
• add option "--recompress-on-fly" to mount action
• check compression type for readed and writed block
  • if it's deprecated - mark block as "to write / update", compress again and save

As soon as I update my python-zstd project

It was designed to just change files name for table storage.
Not table name inside that file.
So it can be copied back.

But sometimes things get worse: indexes may get doubled after making snapshot.

1. Just use --compress as --compress-custom.
   If there will be all_ method - use last of them.
2. all_fast is all methods with fast level activated.
3. all_best is all methods with best level activated.
4. compression level can be defined as --compress=zlib:9 or --compress=zlib:fast.
   Delimiter symbol changed from = to :

It's needed for packaging.

Add options for host, port, db, user and pass.

For example lz4, zstd, lzo can be really fast.
And multi-process compression waste more time in inter-process communication.
Need to test if multi-threaded version will be faster.

Add:

• multi-threaded compression tool class
• new option to switch between multi-th/p versions

Some parts/modules of DdSF can be compiled by cython:

• dedupsqlfs/lib/cache
• dedupsqlfs/fuse

This can be used by packaging.

Create "recompression" action in "do" app to "remove" not needed compression algo from fs. Don't forget to NULL all subvol stats after that.

Add possibility to install without it.

Do not waste resources if CPU is fast enough

Quicklz compression level is choosen at compilation time.
What need to do:

1. Keep "old" module version for compatibility
2. Make two versions with different compilation options: quicklzf(ast), quicklzm(edium) & quicklzb(est)

To prevent DB damage on mount or other action.

Sometimes it's just not needed.
Only if files big ehough - like tree,inode or index tables if:

• many files on fs
• big files on fs

What is means:

• remove old snapshots
• keep only selected by some periods

Like --cleanup-old-snapshots-by-plan=14d:8w=1:18m=1:3y, so it will keep:

• 14 latest daily snapshots (distance <=1 day between)
• 8 weekly - Monday or any in week day if no other (distance <=1 week between)
• 18 monthly - 1 day or any day if no other (distance <=1 month between)
• 3 yearly - one in year (distance <=1 year between)

Do not collect garbage in hash+blocks - should be faster gc.

What need to be achieved:

• continue to work with in-memory data while cache flush/cleanup
• more smooth I/O

There may be overhead by inter-process data movement.

File with base parameters to use without bunch of commands-line options.

Make mysqld server startup and stop separate commands or options for do/mkfs commands.
Goal:

• start server and make fs
• mount fs and connect to started server
• operate with files, sync backups, etc.
• umount fs
• make snapshot
• print statistics
• stop server

It can save about 10-30 seconds on each operation due disabled start-stop of server.

• clean and only fuse operations
• helper for DB, Cache, other functions

Add posibility to config compression methods per inode, except directory, special files.

On some looong rsync processes filesystem consume memory and ... stops.

To gradualy drop caches we need to touch root of filesystem to provide any event (setattr).
It will trigger cache cleanup procedures.
Just create Thread that do os.utime(mountPoint, None) every second.
Until fuse destroy.

Need to:

• make two modules zstd01(old) and zstd03
• make migration to rename old zstd in DB into zstd01

Should use distutils module distributed with python.

LZ4 & ZSTD modules affected.

Problem is that module not exists in old distros like Debian Wheezy, Ubuntu Lucid.

To use new compression levels.

And update statistics only if subvol FS was modified.

1. Store statistics only for snapshots (readonly=True)
2. Recalculate on umount only if modified
3. Store in table only if modified, it is snapshot, no stats saved. On request stats, on snapshot creation.

Current root subvol can gather stats on umount. Need option. If modified.

This is for snapshot statistics output speed up, subvolumes list stats speed up.

Stats:

1. apparent size
2. unique size
3. sparse size
4. dedup size
5. compressed size
6. comp. uniq. size
7. comp. type. stats
   In json, one blob field.

If you move directory to other location of FS then all its content disappear!

http://www.sqlite.org/partialindex.html

If sqlite3.sqlite_version >= 3.8.0 - migrate to partial indexes.
They are smaller and faster (in theory).

Add --decompress-tryout=method1:method2,method3:method4,....
Enable instead --decompress-try-all to speedup decompress on errors.

Behaviour:

• if method1 fails - try method2
• if method2 fails - try method1
• else - raise error

Should be a little more faster. In some cases.

Don't use dict to store 2-4 values in end leaf of caches.
Use list. It 3 times less size in memory.
Not so pretty but can reduce memory footprint.

Need to change db structure:

• sizes in blocks index table
• subvolume ID in tree, inode, index tables