mfinifter / zfs-toolbox Goto Github PK

Per the discussion in #7, configuration should exist as properties on datasets. The zfs-auto-backup:backup-pools property specifies which backup pools the dataset should be backed up to. The property should be inherited by default.

Once implemented, this should resolve #6 and #7.

It seems like many of the issues we're running into are limitations of trying to parse command line results of the zfs commands. Perhaps a way worth exploring is using libzfs directly via Python bindings. I did find this project: http://sourceforge.net/projects/pyzfs/ I don't know how well it works or it it's stable, so it might be worth hacking up our own boost-python bindings for what we need.

Configuration should be able to specify a many-to-many relationship between local pools and backup pools. I may want to backup several local pools to the same backup pool, or a single local pool to multiple backup pools (e.g., for rotating off-site backups). Or multiple local pools to multiple backup pools.

Expected behavior is that a backup should happen for all possible combinations specified. For example, suppose I want both salmon_tank and tuna_tank to backup to both backup_ocean and backup_lake. The following backups should occur:

• salmon_tank to backup_ocean
• tuna_tank to backup_ocean
• salmon_tank to backup_lake
• tuna_tank to backup_lake

In the common case, only one of the backup pools will be available. This is fine; fail silently (but generate a log message) for any backup pool that isn't available.

Log to a configurable location. Use a proper logger with different logging levels. Add appropriate logging statements throughout.

The script borks if any dataset has a space in its name. Easy to work around with a few "zfs rename"s, but ideally, we would support this.

I think a basic documentation is needed for people to use this toolbox. Currently, a user has to read the code to see the usage and figure out the behavior by reverse engineering.
Thanks

I just had a crash while running the backup script (woo-hoo beta ZFS software). It managed to get through all the snapshots of a couple of my file systems, but not all of them. When my machine restarted, I went to run the backup script again, and it responded, "Nothing to do. Backup pool already has latest snapshot ..."

After some investigating, I realized that the problem here is that the script only checks the root dataset. In my case, the root dataset had successfully zfs send'd all of its snapshots, so it looked like the backup was up to date. Unfortunately, many of the child datasets had not successfully transferred, so this was an erroneous "up to date" message.

I'm not positive of the path forward here, but maybe we need to somehow check the latest snapshot for each dataset.

Actually, thinking about this now, this will probably be a problem in some none-error/crash cases. For example, if you take a manual snapshot on a non-root dataset, then the script will say you're at the latest snapsot, even if you're not up to date on some child dataset.

As discussed in #4:

At package install time for zfs-auto-backup (which doesn't yet exist), set up auto-snapshot configuration changes automatically by asking the user for the name of the backup pool and the local pool(s) to automatically backup. Use this information to change zfs-auto-snapshot config to add --default-exclude, and set the "com.sun:auto-snapshot" flag on all pools except the backup pool.

Currently, we have options for deleting all snapshots up to a given snapshot and for deleting all snapshots across datasets matching a specified snapshot. Another set of useful options might be ones that specify the deletion of all snapshots after a given snapshot or all snapshots between two given snapshots. These would be mutually exclusive with --upto.

I must be using subprocess.check_output incorrectly. It's raising a CalledProcessError even when the args look good.

We should create an overarching "zfs-toolbox" script that allows one to run any of these scripts. We should make sure to create bash-completion for it as well.

This complicates the incremental send/recv process because finding the incremental 'from' snapshot becomes more difficult. I think the expected behavior is that I should not have automatic snapshots for my backup pool.

So maybe this is a feature request for zfs-auto-snapshot to allow me to exclude pools? It may be that this feature already exists and I haven't read the zfs-auto-snapshot docs or code well enough. Yup: need to invoke zfs-auto-snapshot with --default-exclude and set the "com.sun:auto-snapshot" flag on datasets I want auto-snapshotted.

When you do a dry run of zfs-auto-backup, you get an error message prior to each command being executed:

cannot open 'foo/bar/baz': dataset does not exist

You get this for each dataset that it is trying to backup.

Because it is a dry run, it (obviously) does not create the new filesystems on the backup destination. However, then when get_latest_backed_up_zfsautobackup_snap is run, the parent dataset hasn't been created, so it fails with an error that there is no such dataset.

This is probably also a problem on incremental backups if there is a new dataset that hasn't been backed up yet, but I haven't tested this.

Another useful script would be a zfs-rollback script that rolled back all filesystems that are a child of the specified FS to a given snapshot name. That is, something like this:
zfs-rollback foo/bar snap1
would rollback foo/bar@snap1, as well as foo/bar/baz@snap1 and foo/bar/bazbar@snap1.

Currently, --verbose on zfs-auto-backup is not particularly consistent. There are some messages (such as zpool import failures) that ought to be default messages, but are only said on verbose, and there are others that are given that seem like they ought only to be verbose but are given by default. We should take a look at how various messages are output and whether they should be by default or verbose.

Related to this, I personally think that most messages should be given by default and we should create a --quiet option instead.

I should be able to backup only a subset of my datasets. Namely, if I have the datasets:
fishtank/ubuntu-1
fishtank/ubuntu-2
I should be able to specify just to backi up fishtank/ubuntu-1 (and all of its dependent datasets, of course).

Where to put this on the destination pool is another question. For example, if I have the backup pool fishhatchery, and I want to only backup fishtank/ubuntu-1, should the resultant backup be fishhatchery/fishtank/ubuntu-1 or fishhatchery/ubuntu-1? Propbably the former as that would seem to have fewer conflicts in the long run.

We need a script that 'installs' zfs-auto-backup, which means it just sets up the cron jobs correctly. Bonus if we make an actual ubuntu package.

I believe zfs-auto-snapshot deletes and rename snapshots as part of the daily/weekly/monthly snapshot generation. If that's the case, how will this script handle it? Will it? I feel like we're sitting on a problem here with how zfs send -I works once stuff has been deleted.

Configuration is currently hard-coded. It should be passed in as command-line arguments or stored in an external configuration file.