csirac2 / snazzer Goto Github PK

Currently the measurement reports are expensive to validate; PGP signatures are performed over the entire snapshot contents in parallel with sha512sum. It would make more sense to simply PGP sign the hashes. There was a time when I thought PGP sigs would be the only way to go but now I'm quite sure there is a vanishingly small population people even remotely interested in backup integrity at all, let alone integrity which involves PGP.

First, Snazzer is pretty exactly what I was looking for to backup my system, so thanks a lot for your great work :)

When I use snazzer to snapshot a subvolume in my home I would like to be able to explore it without being the root permission. But, the .snapshotz file have the following file permission: drwx------ 1 root root.

Is there a good reason for this? I think creating root folder in the user home is not a good practice don't you?

Mostly snazzer-send-wrapper, it seems

See #43 (comment)

expecting users to source the env on each invocation isn't terribly user-friendly

and perhaps a future debian post-install script could help do this semi-automagically.

Many people don't use nested subvols, instead manually mount btrfs subvols in the filesystem. Need to cope with this usage or at least error gracefully if we can't.

Related: #4

I recently adopted snazzer for keeping my btrfs snapshots.
Since I had several older snapshots that i created by hand, I wanted to import them into the snazzer .snapshotz directory.
These snaphots of course contain neither the .snapshotz directory nor .snapshot_measurements.exclude.

In principle, I could go through all the snapshots and add an empty .exclude file, but

• That would be a bit of a hassle because they are read-only snaps
• I'd rather not touch these, especially because they are already synced to other machines

Snazzer should be able to deal with a non-existent .exclude file (perhaps by replacing it with an empty dummy file...)

Alternatively, it would be nice to have a command snazzer import to properly import a manual snapshot.

I'm currently in the process of restoring a snazzer snapshot for each subvolume from my external backup disk to my internal main disk, as the filesystem got corrupted beyond repair.

So what will happen the next time I run snazzer-receive to get new snapshots from my internal disks, which has none of the old snapshots prior to the corruption? Will the old snapshots be deleted from the external disk, or does snazzer-receive never change snapshots already transfered, and I have to actually run snazzer --prune on the external hard disk as well, from time to time?

From my studies of snazzer-receive source code I now guess it's the second option, but I did not find any statements regarding this in the manpages.

@csirac2 I can get a PR ready improving the manpages as soon as this is clarified, if you want. :)

Idea: a snapshot /foo/.snapshotz/YYYY-MM-DDT000000Z, when sent, should have a file /foo/.snapshotz/.sent/YYYY-MM-DDT000000Z with the receiving hostname + datetime appended. Basically, we reinvent the archive bit...

... this would become useful in a future iteration of snazzer which could be smarter about pruning; such as pruning based on disk free or warning/erroring on pruning of snapshots which haven't been sent elsewhere yet.

Example:

• Remote host has a subvolume /var/log. Parent dir var is an ordinary directory on a btrfs root.
• Remote host then changes /var to be a subvolume itself, while retaining /var/log subvol.
• snazzer-receive could be smart enough to detect this change and change the var directory on the receive destination to a subvol.
• Also consider how snazzer deals with other changes (Eg. subvolume moves/renames).
• This probably calls for development of proper test infrastructure/hooks.

It should be possible to configure how many snapshots are kept.

Related bug: #10. user-submitted:

when doing
a dry-run on our system with no snapshots the snazzer-prune-candidates
command called by snazzer would throw an error about “Use of uninitialized
value” on line 512 (Which is my $top = $parsed->{ $sorted[0] }; - Once we
had created our first snapshots this problem went away but it’s something
you might want to know.

It'd be nice to delete a named snapshot wherever it exists across --all subvolumes (or whatever).

btrfs-progs 4.11 changed the options parser. Now snazzer throws errors.

line 178 in snazzer-receive throws: "ERROR: unable to resolve -p"
$SUDO btrfs send "$SNAP_ABSPATH" -p "$SNAP_PARENT_PATH" | \ $SUDO btrfs receive "$DEST_SUBVOL/.snapshotz/.incomplete"

has to do with some command line argument parsing.

man btrfs send sais this:
btrfs send [-ve] [-p <parent>] [-c <clone-src>] [-f <outfile>] <subvol> [<subvol>...]

so I put line 178 the other way arround:
$SUDO btrfs send -p "$SNAP_PARENT_PATH" "$SNAP_ABSPATH" | ....

...and it works.

btrfs-progs 4.11-1

with btrfs-progs 4.9.1 there is no problem.

or should I report to btrfs-progs ?

SNAZZER_SNAPSHOTZ_PERMS=0700 is not appropriate in all cases

This bug item is pretty much to address https://github.com/csirac2/snazzer/blob/master/doc/snazzer-prune-candidates.md#bugs-and-limitations "When some datetimes are close together, they mightn't be pruned"

I've considered adding a new retention variable, SNAZZER_RECENTS_TO_KEEP - a linear number of snapshots to retain. But this would still be a "minimum" type policy, just like the other _TO_KEEP variables. Instead, the user-submitted bug report probably desires a SNAZZER_MAX_SNAPSHOTS_TO_KEEP. Which is probably worth implementing, but I should fix the existing confusing behaviour first, because I suspect the desire for this feature is partly driven by the existing surprising behaviour (failure to prune snapshots when there's not many snapshots in the beginning of a deployment).

Because at some point monthlies are too frequent and yearlies are too course... default might be 8 quarters.

However, mount (at least on debian jessie) conveniently lists /path/to/img for each /mount/point, doing the same with findmnt looks to be a two-step process at least.

Change our assertions to use diff -u expected actual rather than [ "foo" = "$blah" ] so that bats will emit some useful/actionable information when tests fail.

Correct me if I'm wrong, but I think there's no option currently to exclude the root volume a.k.a. the mountpoint from being snapshotted. The exclude patterns file only applies to subvolumes, and as they don't have a leading / one couldn't specify the root to be excluded at all.

I like to have a somewhat flat subvolume layout, where a subvolume of the root volume is mounted as /, not the root volume itself, while the root volume is only used to contain the other subvolumes, but not mounted in the filesystem hierarchy.
See https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Layout for more details so I don't have to repeat them here. 😉

I'm not sure what to do here, though. Options that came to my mind, in no particular order:

• add a command line option like snazzer --ignore-mountpoint
• prefix all subvolumes listed by btrfs subvolume list -t with a /, so that one's able to specify / in the snazzer exclude file. This would require a change in all absolute paths in the exclude file, but might improve usability as people would probably throw in the leading / anyway if they're not familiar with the format of sudo btrfs subvolume list.
• Do nothing at all and just accept that there's an unneeded /.snapshotz full of snapshots that never change at all.

I'm currently using my own fork that has mounpoint snapshotting disabled in the code, but that's no general solution, of course. 😉

$ ./snazzer --help
[sudo] password for stefan:

--help should not ask for root permissions.

The only reproducible thing would seem to be to get the number of bytes at the end of the tar operation which feeds gpg2 and sha512sum.

uname -a = Linux 4.8.0-1-amd64 #1 SMP Debian 4.8.7-1 (2016-11-13) x86_64 GNU/Linux
Bash version 4.4.5(1)-release

Without perl-doc installed, when running snazzer --man, the source code for snazzer is displayed. After closing (is it less that's used?) a message is presented saying that "You need to install the perl-doc package to use this program".

Expected behaviour: I believe it would be good to, if the dependency isn't there, gracefully fail to the info message instead of opening the source code.

Even if it's just printing a list of commands the user needs to run for themselves... but hopefully we can do better than that.

When executing the new make test, the following test fails:

 ✗ sudo -n snazzer --list-snapshots '--all' 'foo=" some stuff "' 'hel'\\'' squot '\\''lo' 'asd " dquot " fgh' 'ap ple' ' bon'\\''squot'\\''jour' 'there'
   (in test file tests/snazzer-send-wrapper.bats, line 45)                                                                                                                                                          
     `[ "$status" = "0" ]' failed

I'm not sure what the problem is there, though.

Related: Did you try to run all tests and not only snazzer-prune-candidates via travis, maybe the beta image with Ubuntu 14.04 contains a new-enough btrfs for the tests to work?

Snazzer should show some kind of progress while the send/receive operation is running (at least while operating local).

Avoid forcing cron jobs to use cd

It would ease creating and maintaining snazzer packages if make install would also install the manpages to $INSTALL_PREFIX/share/man, preferably gzipped.

Refer to the AUR package shell script to see what package maintainers are currently doing.

I don't know how to achieve that, though. Add another find statement to make install / uninstall somehow?

This was requested by maintainer jhass @ https://aur.archlinux.org/packages/snazzer-git/

Because the usage of grep -f, each line inside /etc/snazzer/exclude.patterns currently behaves like it would have been surrounded by _-globs, making tmp essentially the same as _tmp*.

This is an issue e.g. because you can't exclude parent subvolume without excluding all child subvolumes as well: In my setup, /home/florian is a subvolume I want to snapshot, but /home is also a subvolume to be able to mount all home directories with a single mount command, but I don't want /home to be snapshotted. With the current globbing, it's impossible to exclude /home without excluding /home/florian as well.

user-submitted: won't run on systems with coreutils lower than 8.23 due to usage of df --output

On Fedora the default volume layout is

# btrfs subvolume list -t /
ID	gen	top level	path	
--	---	---------	----	
257	123693	5		root
258	123693	5		home
305	123659	257		var/lib/machines

snazzer --all --dry-run fails with ERROR: /root is not a btrfs subvolume and
snazzer --all --dry-run /home fails with /home/root is not a btrfs subvolume

mount /dev/sdb -o subvolid=5 /mnt and then
snazzer --all --dry-run /mnt will ignore ignores.
while snazzer --all --dry-run /mnt/root will fail with ERROR: /mnt/root is not a filesystem mountpoint same for /mnt/home

Because configuring sudoers, setting accounts on remote hosts, ssh shell wrappers, etc. is tedious and it might be useful to walk people through this at least once to make evaluation as painless as possible.