Git Product home page Git Product logo

btrfs-sxbackup's Introduction

btrfs-sxbackup

Btrfs snapshot backup utility

  • Push/pull support via SSH
  • Retention
  • Email notifications
  • Compression of transferred data
  • Syslog logging

System dependencies

Required

The following packages have to be available on both source and destination

  • bash
  • btrfs-progs

The system executing btrfs-backup requires

  • python3

Optional

  • ssh (for remote push/pull, not required for local operation)
    • bash has to be set as the default remote shell for the user running the backup
  • lzop (for compression support if desired)
  • pv (provides progress indication if installed)
  • sendmail (for email notifications if desired)

Installation

Setup

  • when using ssh, public/private key authentication should be set up

Known limitations

  • the destination filesystem has to be mounted without the subvol option, otherwise an error will occur on btrfs receive prompting you to remount with fs tree
  • some commands (like update) may not be available for backup jobs created with older versions of btrfs-sxbackup. in this case backup jobs can be recreated using destroy and init. existing snapshots will be kept as long as destroy is not invoked with --purge.

Usage examples

Initialize

Initialize a backup job pulling snapshots of subvolume / on remote host myhost.org to local subvolume /backup/myhost

Initialize a backup job pushing snapshots of local subvolume / to remote subvolume /backup/myhost on host mybackupserver.org

note that in the ssh:// URL, the username and port part are optional; if not specified, the defaults will be used.

Run

Run a backup job

Cron

Cronjob performing a pull backup job

Synopsis and options

init

run

update

info

purge

destroy

transfer

btrfs-sxbackup's People

Contributors

bhelm avatar crass avatar masc3d avatar morxa avatar sdaros avatar syntonym avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

btrfs-sxbackup's Issues

Incremental backup not working

Since the update to Fedora 31 I've got a problem with btrfs snapshot backups.
The first transfer runs without any problem as it is a complete backup of the source file system (machine 1) to the destination filesystem (machine 2).
The second run uses btrfs send -p to send the diff only.
On the receiver side (machine 2) btrfs receive is called. This aborts with an error:

ERROR source :: Command ['bash', '-c', 'ionice -c3 btrfs receive "/mnt/storage/"'] failed with error code 1 (ERROR: cannot open /mnt/storage//.temp.0c627bb6c073408fa8fbacffcc5491c1/o873662-31387-0: No such file or directory              ]
At snapshot .temp.0c627bb6c073408fa8fbacffcc5491c1)

Notice the double / in the path. For some reason btrfs receive gets that invalid path as destination for the new snapshot. In my understanding that path is provided by btrfs send.
Notice that the arguments of btrfs send do not contain any wrong path components:

DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'gilching1', 'ionice -c3 btrfs send -p "/mnt/storage-1/.sxbackup/sx-20191128-173739-utc" "/mnt/storage-1/.sxbackup/.temp.0c627bb6c073408fa8fbacffcc5491c1"']

In my oppinion the problem is not located in the code of btrfs-sxbackup as it provides correct arguments to btrfs send / receive. The problem might be located in btrfs-progs itself. So I updated these to the current version 5.4-1.fc31 with no effect.

I already reformatted the destination volume and did a complete initial backup run with success. However any further differential backup attempt failed.

Does anyone know about that issue?

Here is the complete debug log of the second run of btrfs-sxbackup (differential backup):

[root@machine2 storage]# btrfs-sxbackup -v run /mnt/storage
INFO btrfs-sxbackup v0.6.11
DEBUG ['bash', '-c', 'if [ -f "/mnt/storage/.btrfs-sxbackup" ]; then exit 10; fi']
DEBUG ['bash', '-c', 'cat "/mnt/storage/.btrfs-sxbackup"']
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'gilching1', 'cat "/mnt/storage-1/.sxbackup/.btrfs-sxbackup"']
INFO source :: url [ssh://gilching1:/mnt/storage-1/] container [.sxbackup/] retention [3] compress [False]
INFO destination :: url [/mnt/storage/] retention [30] compress [False]
INFO preparing environment
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'gilching1', 'if [ ! -d "/mnt/storage-1/.sxbackup/" ]; then btrfs sub create "/mnt/storage-1/.sxbackup/"; fi']
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'gilching1', 'btrfs sub show "/mnt/storage-1/.sxbackup/"']
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'gilching1', 'if [ -d "/mnt/storage-1/.sxbackup/.temp"* ]; then btrfs sub del "/mnt/storage-1/.sxbackup/.temp"*; fi']
DEBUG ['bash', '-c', 'if [ ! -d "/mnt/storage/" ]; then btrfs sub create "/mnt/storage/"; fi']
DEBUG ['bash', '-c', 'btrfs sub show "/mnt/storage/"']
DEBUG ['bash', '-c', 'if [ -d "/mnt/storage/.temp"* ]; then btrfs sub del "/mnt/storage/.temp"*; fi']
INFO source :: retrieving snapshots
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'gilching1', 'btrfs sub list -o "/mnt/storage-1/.sxbackup/"']
INFO destination :: retrieving snapshots
DEBUG ['bash', '-c', 'btrfs sub list -o "/mnt/storage/"']
INFO source :: creating snapshot
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'gilching1', 'btrfs sub snap "/mnt/storage-1/" "/mnt/storage-1/.sxbackup/.temp.0c627bb6c073408fa8fbacffcc5491c1" && sync']
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'gilching1', 'touch "/mnt/storage-1/.sxbackup/.temp.0c627bb6c073408fa8fbacffcc5491c1"']
DEBUG source :: updating property [ro] of [/mnt/storage-1/.sxbackup/.temp.0c627bb6c073408fa8fbacffcc5491c1] to [true]
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'gilching1', 'btrfs property set "/mnt/storage-1/.sxbackup/.temp.0c627bb6c073408fa8fbacffcc5491c1" ro true']
DEBUG ['bash', '-c', 'if [ -d "/mnt/storage/.temp.0c627bb6c073408fa8fbacffcc5491c1" ]; then exit 10; fi']
INFO source :: transferring snapshot
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'gilching1', 'ionice -c3 btrfs send -p "/mnt/storage-1/.sxbackup/sx-20191128-173739-utc" "/mnt/storage-1/.sxbackup/.temp.0c627bb6c073408fa8fbacffcc5491c1"']
DEBUG ['bash', '-c', 'type pv']
DEBUG ['bash', '-c', 'ionice -c3 btrfs receive "/mnt/storage/"']
ERROR source :: Command ['bash', '-c', 'ionice -c3 btrfs receive "/mnt/storage/"'] failed with error code 1 (ERROR: cannot open /mnt/storage//.temp.0c627bb6c073408fa8fbacffcc5491c1/o873662-31387-0: No such file or directory              ]
At snapshot .temp.0c627bb6c073408fa8fbacffcc5491c1)
ERROR source :: Command ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'gilching1', 'ionice -c3 btrfs send -p "/mnt/storage-1/.sxbackup/sx-20191128-173739-utc" "/mnt/storage-1/.sxbackup/.temp.0c627bb6c073408fa8fbacffcc5491c1"'] failed with error code -9 (At subvol /mnt/storage-1/.sxbackup/.temp.0c627bb6c073408fa8fbacffcc5491c1)
INFO destination :: removing subvolume [/mnt/storage/.temp.0c627bb6c073408fa8fbacffcc5491c1]
DEBUG ['bash', '-c', 'if [ -d "/mnt/storage/.temp.0c627bb6c073408fa8fbacffcc5491c1" ]; then btrfs sub del "/mnt/storage/.temp.0c627bb6c073408fa8fbacffcc5491c1"; fi']
INFO source :: removing subvolume [/mnt/storage-1/.sxbackup/.temp.0c627bb6c073408fa8fbacffcc5491c1]
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'gilching1', 'if [ -d "/mnt/storage-1/.sxbackup/.temp.0c627bb6c073408fa8fbacffcc5491c1" ]; then btrfs sub del "/mnt/storage-1/.sxbackup/.temp.0c627bb6c073408fa8fbacffcc5491c1"; fi']
ERROR Transferring snapshot failed
ERROR Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/btrfs_sxbackup/__main__.py", line 216, in main
    job.run()
  File "/usr/lib/python3.7/site-packages/btrfs_sxbackup/core.py", line 1028, in run
    raise e
  File "/usr/lib/python3.7/site-packages/btrfs_sxbackup/core.py", line 1024, in run
    compress=self.source.compress)
  File "/usr/lib/python3.7/site-packages/btrfs_sxbackup/core.py", line 333, in transfer_btrfs_snapshot
    raise e
  File "/usr/lib/python3.7/site-packages/btrfs_sxbackup/core.py", line 325, in transfer_btrfs_snapshot
    raise Error("Transferring snapshot failed")
btrfs_sxbackup.core.Error: Transferring snapshot failed

Retention info output spacing is off?

INFO destination :: url [/mnt/backup/srv-conversionready/] retention [2d: 1/d, 2w:3/w, 1m:1/w, 2m:none] compress [False]

Notice the space between 2d: 1/d expression. Is that normal? I would expect to be able to copy paste that as a base for a new expression.

Destination retention is not applied

My destination retention is set to 2w:daily, but destination snapshots are not deleted even after 6 weeks:

btrfs-sxbackup v0.6.11

INFO destination :: retrieving snapshots
   UUID                   0e58e454-b11f-492a-9a36-0947799e46a3
   Compress               False
   Source URL             /home
   Source info            Linux 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:59 UTC 2020 GNU/Linux, btrfs-progs v4.15.1
   Source container       .sxbackup
   Source retention       3
   Source snapshots       sx-20200318-203107-utc: Wed Mar 18 21:31:07 2020 (+0100)
                          sx-20200318-200438-utc: Wed Mar 18 21:04:38 2020 (+0100)
                          sx-20200318-200211-utc: Wed Mar 18 21:02:11 2020 (+0100)
                          sx-20200318-173009-utc: Wed Mar 18 18:30:09 2020 (+0100)
   Destination URL        ssh://root@beast1:/mnt/500gbext01/backup.kmadac.home
   Destination info       Linux 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 GNU/Linux, btrfs-progs v4.15.1
   Destination retention  2w:daily
   Destination snapshots  sx-20200318-203107-utc: Wed Mar 18 21:31:07 2020 (+0100)
                          sx-20200318-200438-utc: Wed Mar 18 21:04:38 2020 (+0100)
                          sx-20200318-200211-utc: Wed Mar 18 21:02:11 2020 (+0100)
                          sx-20200318-173009-utc: Wed Mar 18 18:30:09 2020 (+0100)
                          sx-20200317-173008-utc: Tue Mar 17 18:30:08 2020 (+0100)
                          sx-20200316-173008-utc: Mon Mar 16 18:30:08 2020 (+0100)
                          sx-20200315-173008-utc: Sun Mar 15 18:30:08 2020 (+0100)
                          sx-20200314-173008-utc: Sat Mar 14 18:30:08 2020 (+0100)
                          sx-20200313-184226-utc: Fri Mar 13 19:42:26 2020 (+0100)
                          sx-20200312-174706-utc: Thu Mar 12 18:47:06 2020 (+0100)
                          sx-20200311-173010-utc: Wed Mar 11 18:30:10 2020 (+0100)
                          sx-20200310-173007-utc: Tue Mar 10 18:30:07 2020 (+0100)
                          sx-20200309-173007-utc: Mon Mar  9 18:30:07 2020 (+0100)
                          sx-20200306-173007-utc: Fri Mar  6 18:30:07 2020 (+0100)
                          sx-20200305-173007-utc: Thu Mar  5 18:30:07 2020 (+0100)
                          sx-20200227-215206-utc: Thu Feb 27 22:52:06 2020 (+0100)
                          sx-20200226-173007-utc: Wed Feb 26 18:30:07 2020 (+0100)
                          sx-20200225-173008-utc: Tue Feb 25 18:30:08 2020 (+0100)
                          sx-20200224-173007-utc: Mon Feb 24 18:30:07 2020 (+0100)
                          sx-20200221-173008-utc: Fri Feb 21 18:30:08 2020 (+0100)
                          sx-20200220-173008-utc: Thu Feb 20 18:30:08 2020 (+0100)
                          sx-20200219-173007-utc: Wed Feb 19 18:30:07 2020 (+0100)
                          sx-20200218-173008-utc: Tue Feb 18 18:30:08 2020 (+0100)
                          sx-20200217-173007-utc: Mon Feb 17 18:30:07 2020 (+0100)
                          sx-20200216-173007-utc: Sun Feb 16 18:30:07 2020 (+0100)
                          sx-20200215-173007-utc: Sat Feb 15 18:30:07 2020 (+0100)
                          sx-20200214-173008-utc: Fri Feb 14 18:30:08 2020 (+0100)
                          sx-20200212-173008-utc: Wed Feb 12 18:30:08 2020 (+0100)
                          sx-20200211-173008-utc: Tue Feb 11 18:30:08 2020 (+0100)
                          sx-20200208-173008-utc: Sat Feb  8 18:30:08 2020 (+0100)
                          sx-20200207-173008-utc: Fri Feb  7 18:30:08 2020 (+0100)
                          sx-20200204-173008-utc: Tue Feb  4 18:30:08 2020 (+0100)
                          sx-20200203-101041-utc: Mon Feb  3 11:10:41 2020 (+0100)
                          sx-20200201-201353-utc: Sat Feb  1 21:13:53 2020 (+0100)

Limit bandwidth or I/O

Hello everybody,

We are running an application that is very I/O intensive. When pulling the first snapshots, there are moments when the VM freezes and btrfs-sxbackup needs to be stopped. The back-up is not to blame for this, but it adds just enough to freeze the server.
Is there any way to limit I/O or bandwidth so the back-up can be pulled at a slower pace?

btrfs-sxbackup assumes bash as remote shell

If the backup user's shell is not bash btrfs-sxbackup can not initialize a remote a backup job.

I did not test other commands than init. These are possibly affected as well.

Changing the remote command call to use a specific shell (/bin/bash?) could solve this, without breaking existing installations (assuming they have /bin/bash, which is safe to assume).

"update" command always activates compression, regardless of -c

BEFORE "update" -> Compress "False"

$ sudo btrfs-sxbackup info /srv/conversionready/
INFO btrfs-sxbackup v0.5.5
INFO source :: retrieving snapshot names
INFO destination :: retrieving snapshot names
   UUID                   688af26a-907a-4e64-820d-f09c59797bef
   Compress               False
   Source URL             /srv/conversionready
   Source container       .sxbackup
   Source retention       1h:1/h, 1d:12/d, 1w:7/w
   Destination URL        /mnt/backup/srv-conversionready
   Destination retention  1h:1/h, 1d:12/d, 1w:7/w, 1m:1/w, 2m:1/m, 3m:none

AFTER "update" -> Compress "True"

$ sudo btrfs-sxbackup update -sr "1h:1/h, 1d:12/d, 1w:7/w" -dr "1h:1/h, 1d:12/d, 1w:7/w, 1m:1/w, 2m:1/m, 3m:none" /srv/conversionready
INFO btrfs-sxbackup v0.5.5
INFO updating configurations
   UUID                   688af26a-907a-4e64-820d-f09c59797bef
   Compress               True
   Source URL             /srv/conversionready
   Source container       .sxbackup
   Source retention       1h:1/h, 1d:12/d, 1w:7/w
   Destination URL        /mnt/backup/srv-conversionready
   Destination retention  1h:1/h, 1d:12/d, 1w:7/w, 1m:1/w, 2m:1/m, 3m:none
INFO updated successfully

There was no -c specified for update.

documentation clarification

This is maybe more of a vague feature request. I read your docs, and had these observations: Where is this storing configuration about the jobs? Seems like something that someone considering using a utility like this would generally like to know. I see it's in the bottom, in the changelog. And, if I'm not supposed to interact with the config, how can Iist out the existing jobs? Regarding this line: "or more complex expression like ..." it implies there is further documentation elsewhere, but I don't see anything. Are those the only time intervals? Is there a way to specify unlimited number of backups?

Complex retention expressions do not seem to work

Nice to see so much progress on this project, thanks!
i have tried the advanced retention feature, but it does not seem to work for me.

here is my .btrfs-sxbackup file on destination:

root@backup:/home/backup/mariadb# cat .btrfs-sxbackup 
[Destination]
uuid = 9b7ea342-ac06-4162-b1f0-4d378d1daa86
source = ssh://[email protected]:/mariadb/
source-container = .sxbackup/
retention = 2w:24/d, 1m:4/d, 6m:daily, 7m:none
compress = True

and here is the log output from my last run:

2015-03-20T17:56:23.4655+01:00 []: btrfs-sxbackup[4447] INFO btrfs-sxbackup v0.5.4
2015-03-20T17:56:23.8235+01:00 []: btrfs-sxbackup[4447] INFO source :: url [ssh://[email protected]:/mariadb/] container [.sxbackup/] retention [1] compress [True]
2015-03-20T17:56:23.8237+01:00 []: btrfs-sxbackup[4447] INFO destination :: url [/home/backup/mariadb/] retention [2w:24/d, 1m:4/d, 6m:daily, 7m:none] compress [True]
2015-03-20T17:56:23.8238+01:00 []: btrfs-sxbackup[4447] INFO preparing environment
2015-03-20T17:56:24.5984+01:00 []: btrfs-sxbackup[4447] INFO source :: retrieving snapshot names
2015-03-20T17:56:24.7998+01:00 []: btrfs-sxbackup[4447] INFO destination :: retrieving snapshot names
2015-03-20T17:56:24.8485+01:00 []: btrfs-sxbackup[4447] INFO source :: creating snapshot
2015-03-20T17:56:27.6031+01:00 []: btrfs-sxbackup[4447] INFO source :: transferring snapshot
2015-03-20T18:11:55.0481+01:00 []: btrfs-sxbackup[4447] INFO source :: removing 1 snapshot due to retention [1]: sx-20150320-111915-utc
2015-03-20T18:11:56.3179+01:00 []: btrfs-sxbackup[4447] INFO backup sx-20150320-165624-utc created successfully in 00:15:32

it does not try to remove any snapshots from the destination. here is the full list of my destination snapshots:
http://www.pastebin.ca/2961717

i would expect that retention would remove all but one from the sx-20150219-* snapshots.

most of these snapshots were created by btrfs-sxbackup 0.3*. i have deleted all but these snapshots and did a btrfs-sxbackup init with the existing destination path.

Thank you!

Snapshot directory timestamps are constant, should be dynamic

.sxbackup $ ls -la
total 164
drwxr-xr-x 1 root root 426  8. juuni 08:00 .
drwxr-xr-x 1 root root 158  7. juuni 00:34 ..
-rw-r--r-- 1 root root 214  7. juuni 23:38 .btrfs-sxbackup
drwxr-xr-x 1 root root 158  7. juuni 00:34 sx-20150607-204433-utc
drwxr-xr-x 1 root root 158  7. juuni 00:34 sx-20150607-210208-utc
drwxr-xr-x 1 root root 158  7. juuni 00:34 sx-20150607-230005-utc
drwxr-xr-x 1 root root 158  7. juuni 00:34 sx-20150608-000004-utc
drwxr-xr-x 1 root root 158  7. juuni 00:34 sx-20150608-010004-utc
drwxr-xr-x 1 root root 158  7. juuni 00:34 sx-20150608-020004-utc
drwxr-xr-x 1 root root 158  7. juuni 00:34 sx-20150608-030005-utc
drwxr-xr-x 1 root root 158  7. juuni 00:34 sx-20150608-040004-utc
drwxr-xr-x 1 root root 158  7. juuni 00:34 sx-20150608-050015-utc

Everything seems to be created on June 7th on the same minute. The snapshot directory creation date should match the filename.

missing license header

The repository contains the GPLv2 license text, however the source files do not contain any statement of copying permission (license header). According to the GPL How-To, a statement of copying permission is also required, and it would avoid any confusion about the license.

Can you please add a copying statement to the source files?

This is part of an effort to package btrfs-sxbackup for Fedora.

Destination much bigger than source with btrfs and sxbackup 0.6.10

Hello,

I've just setup a backup using sxbackup 0.6.10 over btrfs on two Ubuntu 14.0.4 machines (source and destination).

My source has a size of 769 GB reported by df -h, and the destination is 1.1 T also reported by df -h. Those two machines have nothing more than the OS and data that I want to backup. So if we remove 5-6 GB for the OS from both machines we still have a huge difference of 340 GB. The data specific is that we are storing millions of small xml files (serveral KBs one).
Is this difference something normal or something went bad during the first backup.
Also, the second and third incremental backup went well.

Kind regards,

Dan

sxbackup uses pv when invoked by systemd timer.

sxbackup seems to use pv whenever it finds it installed, even when not running on a shell.

When running from a systemd timer-service setup, this will add a lot of CPU overhead as pv seems to misbehave under such conditions and runs at 100% CPU usage (i.e. fully loads a single core by itself) and causes transfer speeds to tank.

In any case, using an interactive progress indicator like pv makes no sense for a timed background job.

I would suggest not using pv unless specifically requested via commandline. Alternatively some kind of autodetection if sxbackup is running in an interactive shell or not, if that can be done reliably.

snapshots not getting purged

I've got the following snapshots:

drwxrwxrwx 1 root root 108 Jun 11 06:21 sx-20160609-133614-utc
drwxrwxrwx 1 root root 108 Jun 13 12:17 sx-20160613-161358-utc
drwxrwxrwx 1 root root 108 Jun 15 10:26 sx-20160615-142611-utc
drwxrwxrwx 1 root root 108 Jun 20 02:30 sx-20160620-063002-utc
drwxrwxrwx 1 root root 108 Jun 21 03:03 sx-20160621-063003-utc
drwxrwxrwx 1 root root 108 Jun 22 02:36 sx-20160622-063004-utc
drwxrwxrwx 1 root root 108 Jun 22 16:53 sx-20160622-193254-utc
drwxrwxrwx 1 root root 108 Jun 24 02:31 sx-20160624-063002-utc
drwxrwxrwx 1 root root 108 Jun 25 02:41 sx-20160625-063003-utc
drwxrwxrwx 1 root root 108 Jun 26 02:30 sx-20160626-063002-utc
drwxrwxrwx 1 root root 108 Jun 27 02:30 sx-20160627-063002-utc
drwxrwxrwx 1 root root 108 Jun 28 02:33 sx-20160628-063002-utc
drwxrwxrwx 1 root root 108 Jun 29 02:32 sx-20160629-063002-utc
drwxrwxrwx 1 root root 108 Jun 30 02:35 sx-20160630-063001-utc
drwxrwxrwx 1 root root 108 Jul 1 02:30 sx-20160701-063003-utc
drwxrwxrwx 1 root root 108 Jul 2 02:30 sx-20160702-063002-utc
drwxrwxrwx 1 root root 108 Jul 3 02:30 sx-20160703-063002-utc
drwxrwxrwx 1 root root 108 Jul 4 02:30 sx-20160704-063002-utc

When I run btrfs-sxbackup purge on this, I get:

btrfs-sxbackup purge /mnt/backuppool/
INFO btrfs-sxbackup v0.6.6
INFO source :: url [ssh://[email protected]:/mnt/storage/] container [.sxbackup/] retention [3] compress [False]
INFO destination :: url [/mnt/backuppool/] retention [2d: 1/d, 2w:3/w, 1m:1/w, 2m:none] compress [False]
INFO source :: retrieving snapshots
INFO destination :: retrieving snapshots
INFO destination :: removing 1 snapshot due to retention [2d: 1/d]: sx-20160622-193254-utc

Why is this not deleting more snapshots according to the retention policy?

purge command to manually clean out snapshots on source and/or destination

usage: btrfs-sxbackup purge [-h] [-sr SOURCE_RETENTION]
                             [-dr DESTINATION_RETENTION] [-c]
                             subvolume [subvolume ...]

positional arguments:
  subvolume             backup job source or destination subvolume. local path
                        or SSH url

optional arguments:
  -h, --help            show this help message and exit
  -sr SOURCE_RETENTION, --source-retention SOURCE_RETENTION
                        optionally override backup source retention.
  -dr DESTINATION_RETENTION, --destination-retention DESTINATION_RETENTION
                        optionally override backup job destination retention

Multiple source subvolumes

Hi, masc3d

is it possible to add multiple source volumes for single destination, I went trough this article [https://github.com//issues/9](Single Source & Multiple Destinations #9 ) and some similar hack would be very appreciated.

Thank you.

No way to disable compression

There is no inverse of the '-c' option, so you can't disable compression without deleting the Compression line out of the config file on the source and destination.

Will try to make a pull request when I get a chance

Can't destroy when destination unavailable

My old backup server died. I want to reinit backups from scratch, but I can't:

[root@okvmh1 ~]# btrfs-sxbackup destroy /home
INFO btrfs-sxbackup v0.5.9
ERROR could not read configuration [/media/btrfs-remote/okvmh1/.btrfs-sxbackup]

[root@okvmh1 ~]# btrfs-sxbackup init /home ssh://root@okdistrhw:/exports/okvmh1bak/okvmh1 -sr 0 -dr "1d:4/d, 1w:daily, 2m:none"
INFO btrfs-sxbackup v0.5.9
ERROR source is already initialized

Please add some option for destroy command to ignore missing remote.

I'm aware of the workaround:

btrfs subvolume delete /home/.sxbackup/sx-*
btrfs subvolume delete /home/.sxbackup

Allow to disable ssh compression

Disabling ssh's own compression (which might be on by default in users' settings) speeds things up considerably in certain scenarios. You might want to add -o Compression=no here.

Verify lzop is installed when doing compressed transfers

Dear

First of all, thanks a lot for the great software! It's amazing.

I had to reinstall my server and forgot to install lzop, which lead to some figuring out why the snapshot using compression didn't work.

Not a huge problem, just something I noticed

Kind regards, and keep up the good work!

Lieven

Single Source & Multiple Destinations

When trying to use a single destination for multiple sources, the second time you initialize the same source for a new job, it returns the error "ERROR source is already initialized". Is there a way of using the same source for multiple backup jobs to different locations?

Improve error output

Here is the error i had with btrfs-sxbackup. Can you tell the cause?

root@ns320047:~/btrfs-sxbackup# /usr/bin/flock -n /tmp/mariadb.lockfile /usr/bin/python3 /usr/local/bin/btrfs-sxbackup run /backup/mariadb/
INFO btrfs-sxbackup v0.5.8
INFO source :: url [ssh://[email protected]:/home/mysql/] container [.sxbackup/] retention [1] compress [True]
INFO destination :: url [/backup/mariadb/] retention [2w:4/d, 1m:daily, 3m:none] compress [True]
INFO preparing environment
INFO source :: retrieving snapshots
INFO destination :: retrieving snapshots
INFO source :: creating snapshot
INFO source :: transferring snapshot
INFO destination :: removing subvolume [/backup/mariadb/.temp.e1a73f6bf3104420bb36c353ca47572a]                                                                                                        <=>                                                                                                                  ]
INFO source :: removing subvolume [/home/mysql/.sxbackup/.temp.e1a73f6bf3104420bb36c353ca47572a]
ERROR Command '['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '[email protected]', 'btrfs send "/home/mysql/.sxbackup/.temp.e1a73f6bf3104420bb36c353ca47572a" | lzop -1']' returned non-zero exit status -9
ERROR At subvol /home/mysql/.sxbackup/.temp.e1a73f6bf3104420bb36c353ca47572a
2,49GiB 0:01:14 [34,3MiB/s] [                                                                                                                                                               <=>                                                                                                                             ]
root@ns320047:~/btrfs-sxbackup# 

The backup failed because of no free disk space on the receiving volume. Looking into the source, btrfs-sxbackup checks the sending return code first and throws the error above. in the "no free disk space on the receiving partition" case, it seems to me that send and receive side fail at the same time. btrfs receive prints an error, but btrfs-sxbackup only shows stderr on the sender side.

btrfs-sxbackup should report errors on both sender and receiver side if possible.

Compression not working

Hi,

lzop compression doesnt seem to work

btrfs-sxbackup v0.6.1

 Command ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '[email protected]', 'ionice -c3 btrfs send "/var/lib/lxd/containers/test/.sxbackup/.temp.8dc8849378ee4b0198d142d2b074980d" | lzop -1'] failed with error code 127 (bash: lzop: command not found
which lzop
/usr/bin/lzop

btrfs sxbackup issue

Hello,

After running the sxbackup script i received this error in var/log/syslog:

Dec 5 17:28:06 dx01 btrfs-sxbackup[1745] ERROR source :: Command ['bash', '-c', 'btrfs receive "/mnt/btrfs-root/"'] failed with error code 1 (At snapshot .temp.e5ff8c688cf046cca754cbb4b6a13de8#012ERROR: unexpected EOF in stream.)
Dec 5 17:28:06 dx01 btrfs-sxbackup[1745] ERROR source :: Command ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'root@dx02', 'btrfs send -p "/dxdata/.sxbackup/sx-20161204-073004-utc" "/dxdata/.sxbackup/.temp.e5ff8c688cf046cca754cbb4b6a13de8"'] failed with error code -9 (At subvol /dxdata/.sxbackup/.temp.e5ff8c688cf046cca754cbb4b6a13de8)

Also after running the script and encountering this error (the script is running on dx01 and making a backup of dx02), the btrfs filesystems are mounted as read-only. Can you please give me a hint about what`s causing the error?

Invalid script entry point, A callable suffix is required

I'm trying to install btrfs-sxbackup on RHEL7. It says:

Installation: pip install btrfs-sxbackup
Required: python3

I installed python34 from epel. There's no pip for it, so I installed it, like described here: http://stackoverflow.com/a/32771094/447503

pip3 builds my package, but can't install it:

[root@wc8rh7 ~]# yum -y install python34
[root@wc8rh7 ~]# curl https://bootstrap.pypa.io/get-pip.py | python3.4
[root@wc8rh7 ~]# pip3 install btrfs-sxbackup
Collecting btrfs-sxbackup
  Downloading btrfs-sxbackup-0.5.9.tar.gz
Building wheels for collected packages: btrfs-sxbackup
  Running setup.py bdist_wheel for btrfs-sxbackup ... done
  Stored in directory: /root/.cache/pip/wheels/42/28/92/3f60b95db806b9d2e1b3a1a91acdac263dadfaba17a88b1df9
Successfully built btrfs-sxbackup
Installing collected packages: btrfs-sxbackup
Invalid script entry point: <ExportEntry btrfs-sxbackup = btrfs_sxbackup.__main__:None []> for req: btrfs-sxbackup - A callable suffix is required. Cf https://packaging.python.org/en/latest/distributing.html#console-scripts for more information.

Pre/post-snapshot command for pull jobs

When pulling a backup it would be nice to be able to specify a command to be executed on the remote machine before taking the snapshot (to get a database in a consistent state for example).

v0.6.8 always transferring full snapshots

Hi.

Since upgrading from v.0.6.4 to v0.6.8, incremental backups do not work. Full snapshots are always sent without any warning message.

With a quick look it seems to me, that the culprit is commit 8e54abc.

It looks like that with this:
if self.destination is not None: [snip] else: source_parent_path = os.path.join(self.source.container_subvolume_path, str(self.source.snapshots[0].name))
...the source_parent_path is only set if there are no destination snapshots.

mutli destinations

Hi,

first of all I would like to thank for such a good script suite to create comfortable btrfs backups.
What I am currently miss is to use different backup destination. Assume in the backup maschines are different disk drives instead of raid, so it would be very nice to select the backupdestination by an argument or pass the config dir.

What is your opinion about his?

arcfour support for ssh

Hi,

I've seen slowdowns in the transfer of the snapshots between the backupserver and the client. Investigation into why I only got around 30 MB/s over a gigabit ethernet line resulted in the finding that the ssh server causes heavy CPU load. Using the arcfour parameter for ssh results in full gigabit speed transfers. Encryption is not important for this use case, since both computers communicate over a cross-over cable.

btrfs-sxbackup purge does nothing

Thanks for the great backup script, I'm using it for quit some time now. However, I still have to purge manually, but I don't understand, why:

# ls /.sxbackup/
sx-20150107-094126-utc  sx-20150211-220016-utc  sx-20150603-210021-utc  sx-20150909-210024-utc  sx-20151210-074830-utc  sx-20160302-220004-utc  sx-20160518-210025-utc  sx-20160810-210025-utc
sx-20150107-094931-utc  sx-20150218-220017-utc  sx-20150610-210021-utc  sx-20150916-210024-utc  sx-20151210-140054-utc  sx-20160309-220003-utc  sx-20160525-210024-utc  sx-20160811-094457-utc
sx-20150107-145211-utc  sx-20150225-220017-utc  sx-20150617-210023-utc  sx-20150923-210024-utc  sx-20151216-220003-utc  sx-20160310-081407-utc  sx-20160601-210024-utc  sx-20160817-210023-utc
sx-20150107-152803-utc  sx-20150304-220017-utc  sx-20150618-152353-utc  sx-20150930-210026-utc  sx-20151223-220028-utc  sx-20160316-220003-utc  sx-20160608-210025-utc  sx-20160824-210024-utc
sx-20150107-165956-utc  sx-20150311-220017-utc  sx-20150624-210024-utc  sx-20151007-210026-utc  sx-20151230-220026-utc  sx-20160323-220004-utc  sx-20160615-210026-utc  sx-20160825-115704-utc
sx-20150107-172849-utc  sx-20150318-220017-utc  sx-20150701-210025-utc  sx-20151014-210027-utc  sx-20160104-073210-utc  sx-20160330-210003-utc  sx-20160616-052336-utc  sx-20160831-210023-utc
sx-20150107-180218-utc  sx-20150325-220017-utc  sx-20150703-065022-utc  sx-20151015-075800-utc  sx-20160106-220024-utc  sx-20160331-070903-utc  sx-20160622-210025-utc  sx-20160907-210024-utc
sx-20150108-084917-utc  sx-20150401-210018-utc  sx-20150708-210025-utc  sx-20151021-210025-utc  sx-20160113-220024-utc  sx-20160406-210024-utc  sx-20160629-210025-utc  sx-20160908-074155-utc
sx-20150108-091317-utc  sx-20150408-210017-utc  sx-20150715-210027-utc  sx-20151028-220025-utc  sx-20160120-220022-utc  sx-20160407-073018-utc  sx-20160630-070522-utc  sx-20160914-210022-utc
sx-20150114-220017-utc  sx-20150415-210018-utc  sx-20150722-210028-utc  sx-20151104-220025-utc  sx-20160125-112524-utc  sx-20160413-210024-utc  sx-20160706-210026-utc  sx-20160915-071843-utc
sx-20150121-220017-utc  sx-20150422-210019-utc  sx-20150729-210028-utc  sx-20151111-220026-utc  sx-20160125-144931-utc  sx-20160414-094322-utc  sx-20160707-050921-utc
sx-20150128-220018-utc  sx-20150429-210018-utc  sx-20150805-210029-utc  sx-20151112-170603-utc  sx-20160203-220024-utc  sx-20160420-210024-utc  sx-20160713-210024-utc
sx-20150205-095830-utc  sx-20150506-210019-utc  sx-20150812-210030-utc  sx-20151118-220025-utc  sx-20160210-220024-utc  sx-20160421-122602-utc  sx-20160714-050147-utc
sx-20150205-101139-utc  sx-20150513-210019-utc  sx-20150820-094732-utc  sx-20151125-220025-utc  sx-20160217-220025-utc  sx-20160427-210024-utc  sx-20160720-210022-utc
sx-20150205-101833-utc  sx-20150520-210019-utc  sx-20150826-210023-utc  sx-20151202-220025-utc  sx-20160224-220026-utc  sx-20160504-210025-utc  sx-20160727-210023-utc
sx-20150205-105226-utc  sx-20150527-210020-utc  sx-20150902-210024-utc  sx-20151209-220026-utc  sx-20160225-064130-utc  sx-20160511-210021-utc  sx-20160803-210024-utc

# LC_ALL=C date
Thu Sep 15 10:26:56 CEST 2016


# btrfs-sxbackup purge /
INFO btrfs-sxbackup v0.6.10
INFO source :: url [/] container [.sxbackup/] retention [1w:2/d, 2w:daily, 1m:weekly, 2m:none] compress [False]
INFO destination :: url [/media/backups2/@/] retention [1m:weekly, 2m:monthly, 4m:2/m, 12m:none] compress [False]
INFO source :: retrieving snapshots
INFO destination :: retrieving snapshots

--- that's all, I get. Shouldn't the script remove backups from the source?

Sorry, I don't code Python usually, would have tried harder to fix it myself otherwise, but I have no idea, what's wrong.

Thank you again,

Stephan

btrfs send returned non-zero exit status -9 with latest HEAD

It seems to call btrfs send again after having deleted the temporary snapshot:

~ # btrfs-sxbackup -vvv run /var/backup/myhost.root
INFO btrfs-sxbackup v0.5.5
DEBUG ['bash', '-c', 'if [ -f "/var/backup/myhost.root/.btrfs-sxbackup" ] ; then exit 10; fi']
DEBUG ['bash', '-c', 'cat "/var/backup/myhost.root/.btrfs-sxbackup"']
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '[email protected]', 'cat "/.sxbackup/.btrfs-sxbackup"']
INFO source :: url [ssh://[email protected]:/] container [/.sxbackup/] retention [10] compress [False]
INFO destination :: url [/var/backup/myhost.root/] retention [10] compress [True]
INFO preparing environment
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '[email protected]', 'if [ ! -d "/.sxbackup/" ] ; then btrfs sub create "/.sxbackup/"; fi']
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '[email protected]', 'btrfs sub show "/.sxbackup/"']
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '[email protected]', 'if [ -d "/.sxbackup/.temp"* ]; then btrfs sub del "/.sxbackup/.temp"*; fi']
DEBUG ['bash', '-c', 'if [ ! -d "/var/backup/myhost.root/" ] ; then btrfs sub create "/var/backup/myhost.root/"; fi']
DEBUG ['bash', '-c', 'btrfs sub show "/var/backup/myhost.root/"']
DEBUG ['bash', '-c', 'if [ -d "/var/backup/myhost.root/.temp"* ]; then btrfs sub del "/var/backup/myhost.root/.temp"*; fi']
INFO source :: retrieving snapshot names
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '[email protected]', 'btrfs sub list -o "/.sxbackup/"']
INFO destination :: retrieving snapshot names
DEBUG ['bash', '-c', 'btrfs sub list -o "/var/backup/myhost.root/"']
INFO source :: creating snapshot
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '[email protected]', 'btrfs sub snap -r "/" "/.sxbackup/.temp.2df46184af934699946224a049785f5c" && sync']
DEBUG ['bash', '-c', 'if [ -d "/var/backup/myhost.root/.temp.2df46184af934699946224a049785f5c" ]; then exit 10; fi']
INFO source :: transferring snapshot
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '[email protected]', 'btrfs send -p "/.sxbackup/sx-20150323-142542-utc" "/.sxbackup/.temp.2df46184af934699946224a049785f5c"']
DEBUG ['bash', '-c', 'type pv']
DEBUG ['bash', '-c', 'btrfs receive "/var/backup/myhost.root/"']
INFO destination :: removing subvolume [/var/backup/myhost.root/.temp.2df46184af934699946224a049785f5c]
DEBUG ['bash', '-c', 'if [ -d "/var/backup/myhost.root/.temp.2df46184af934699946224a049785f5c" ]; then btrfs sub del "/var/backup/myhost.root/.temp.2df46184af934699946224a049785f5c"; fi']
INFO source :: removing subvolume [/.sxbackup/.temp.2df46184af934699946224a049785f5c]
DEBUG ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '[email protected]', 'if [ -d "/.sxbackup/.temp.2df46184af934699946224a049785f5c" ]; then btrfs sub del "/.sxbackup/.temp.2df46184af934699946224a049785f5c"; fi']
ERROR Command '['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '[email protected]', 'btrfs send -p "/.sxbackup/sx-20150323-142542-utc" "/.sxbackup/.temp.2df46184af934699946224a049785f5c"']' returned non-zero exit status -9
ERROR At subvol /.sxbackup/.temp.2df46184af934699946224a049785f5c
ERROR Traceback (most recent call last):
  File "/usr/lib64/python3.3/site-packages/btrfs_sxbackup/__main__.py", line 192, in <module>
    job.run()
  File "/usr/lib64/python3.3/site-packages/btrfs_sxbackup/core.py", line 904, in run
    raise e
  File "/usr/lib64/python3.3/site-packages/btrfs_sxbackup/core.py", line 900, in run
    compress=self.source.compress)
  File "/usr/lib64/python3.3/site-packages/btrfs_sxbackup/core.py", line 294, in transfer_btrfs_snapshot
    raise e
  File "/usr/lib64/python3.3/site-packages/btrfs_sxbackup/core.py", line 282, in transfer_btrfs_snapshot
    send_process.stderr.read())
subprocess.CalledProcessError: Command '['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '[email protected]', 'btrfs send -p "/.sxbackup/sx-20150323-142542-utc" "/.sxbackup/.temp.2df46184af934699946224a049785f5c"']' returned non-zero exit status -9

61.2KiB 0:00:02 [22.8KiB/s] [   <=>                                                                                       ]

The configuration reads as follows (same behaviour with a fresh configuration):

[Destination]
source = ssh://[email protected]:/
source-container = /.sxbackup
keep = 10
compress = True

Individual commands for local (`snapshot`) and remote transfer (`sync`)

I am running a MySQL subvolume backup, and must lock the database for run command duration.

Unfortunately, as database size increases, run is now taking 2 minutes and locked database is becoming increasingly uncomfortable to work with.

It'd be a lot smoother, if I could lock the database only for proposed new snapshot command (estimated to be significantly faster to unlock from), and transfer separately afterwards. Your thoughts?

source retention = 0 does not delete local snapshot after transfer

If you define source retention = 0, btrfs-sxbackup does not delete the source snapshot created after finishing the transfer.

It's useful to have source retention = 0 when you're taking backups of nocow stuff where you dont want to retain snapshots locally (since they make noCoW files effectively CoW again), and for me it's not that big of a deal that you'll have to transfer the whole thing every time you run a backup of noCoW stuff.

Snapshots without destination broken since 0.6.4

It seems release 0.6.4 made some changes that break the creation of snapshots without a destination:

$ btrfs-sxbackup -v run /mnt
INFO btrfs-sxbackup v0.6.6
DEBUG ['bash', '-c', 'if [ -f "/mnt/.btrfs-sxbackup" ]; then exit 10; fi']
DEBUG ['bash', '-c', 'cat "/mnt/.sxbackup/.btrfs-sxbackup"']
INFO source :: url [/mnt/] container [.sxbackup/] retention [1d:4/d, 1w:2/w, 3m:2/m, 12m:none] compress [False]
INFO preparing environment
DEBUG ['bash', '-c', 'if [ ! -d "/mnt/.sxbackup/" ]; then btrfs sub create "/mnt/.sxbackup/"; fi']
DEBUG ['bash', '-c', 'btrfs sub show "/mnt/.sxbackup/"']
DEBUG ['bash', '-c', 'if [ -d "/mnt/.sxbackup/.temp"* ]; then btrfs sub del "/mnt/.sxbackup/.temp"*; fi']
INFO source :: retrieving snapshots
DEBUG ['bash', '-c', 'btrfs sub list -o "/mnt/.sxbackup/"']
ERROR 'NoneType' object has no attribute 'snapshots'
ERROR Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/btrfs_sxbackup/__main__.py", line 216, in main
    job.run()
  File "/usr/local/lib/python3.4/dist-packages/btrfs_sxbackup/core.py", line 963, in run
    if self.source.snapshots[0].name.timestamp == self.destination.snapshots[0].name.timestamp:
AttributeError: 'NoneType' object has no attribute 'snapshots'

btrfs-sxbackup can cause btrfs deadlock

this problem is more related to btrfs than to btrfs-sxbackup. im still creating this issue to warn other users about it and because btrfs-sxbackup could avoid this problem.

when btrfs-sxbackup starts while another instance on the same target is running, btrfs-sxbackup does not recognize this and tries to delete the temp directories on both sides. this causes a deadlock in linux kernel versions < 4.0.3 (see https://www.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.0.3 ).

this deadlock brings the .sxbackup directory to a unusable state, only reboot will solve this.

btrfs-sxbackup could warn and refuse to run a backup job that appears already running because most users do not want to run two instances of the same job at the same time.

backup all subvolumes

Some way to backup all subvolumes (or maybe pattern-match?) for a particular remote filesystem would be great. In my use-case, remote subvolumes are created dynamically by other software (lxd), and it'd be useful to have all subvolumes backed up automatically.

btrfs receive returned non-zero exit status 1

I get the following for every run after the first:

# btrfs-sxbackup --source-container-subvolume .sxbackup ssh://[email protected]:/home /var/backup/myhost.home
INFO btrfs-sxbackup v0.4.6
INFO Source :: Url [ssh://[email protected]:/home] snapshot container subvolume [/home/.sxbackup] keep [10]
INFO Destination :: Url [/var/backup/myhost.home] snapshot container subvolume [/var/backup/myhost.home/] keep [10]
INFO Preparing environment
INFO Source :: Retrieving snapshot names
INFO Destination :: Retrieving snapshot names
INFO Source :: Creating snapshot
INFO Transferring snapshot
ERROR Command '['bash', '-c', 'btrfs receive /var/backup/myhost.home']' returned non-zero exit status 1
ERROR Traceback (most recent call last):
  File "/usr/lib64/python3.3/site-packages/btrfs_sxbackup/__main__.py", line 100, in <module>
    backup.run()
  File "/usr/lib64/python3.3/site-packages/btrfs_sxbackup/Backup.py", line 436, in run
    raise subprocess.CalledProcessError(receive_returncode, receive_command, None)
subprocess.CalledProcessError: Command '['bash', '-c', 'btrfs receive /var/backup/myhost.home']' returned non-zero exit status 1

This is with the following versions:

  • btrfs-sxbackup head 11996a1
  • btrfs-progs: 3.18, kernel: 3.18.2 (receiver)
  • btrfs-progs: 3.17.3, kernel: 3.17.3 (sender)
  • python: 3.4.1

the source is a simple subvolume (/home), the dest is a subvolume of a subvolume (/var/backup/myhost.home, /var/backup)

Snapshot directories

Hi,
I have seen and tried this program recently, after making plans for a backup with btrfs. The code seems to be very thoroughly written, congratulations. I have a few questions, though.
I'll backup the root fs of several machines to one place. For a test, I started with a destination on the local host with the following command:
# btrfs-sxbackup init / /.sxbackup
(I guess /.sxbackup can be any different subvolume, but I think it does not matter at this stage.)
The command above created 2 new subvolumes:

  • /sx-20180409-162320-utc and
  • /.sxbackup/sx-20180409-162320-utc
    for the source and destination, presumably. A new
    # btrfs-sxbackup run /
    command created other 2 files. My first question is: can I set a directory for the source snapshots? Otherwise they are going to pile up all on /, what doesn't look nice.
    The second question relates to the subvolume argument (of run and other subcommands). It is vaguely defined as "source or destination subvolume. local path or SSH url". Is that enough? What if the source of one job matches the destination of another? For example, I am backing up from machine2 to local/path1, then from local/path1 to machine3?
    I am missing a list of backup jobs. I've seen that they are listed in /.sxbackup/.btrfs-sxbackup, but it still doesn't answer the question and I don't know if it is generally true. This should be documented (as the backup process, what each command is doing, what subvolumes are created, deleted, a.s.o.)
    Thanks for helping,
    Zsolt

Document how to upgrade backup configuration from previous version

When trying to use the update command I get:

~ # btrfs-sxbackup update -c /var/backup/othermachine.home
INFO btrfs-sxbackup v0.5.5
ERROR update of existing locations requires uuids. this backup job was presumably created with an older version.

Please document how to upgrade to the new format.

Backup not working unexpected EOF in stream

Hello,

i'm using btrfs sxbackup over one year now to backup my nas to another server without any errors.
Since the last two weeks I'm getting the following error message:

INFO btrfs-sxbackup v0.6.11
INFO source :: url [/mnt/storage01/data/] container [.sxbackup/] retention [1] compress [False]
INFO destination :: url [ssh://nas02:/mnt/storage01/data/] retention [40] compress [False]
INFO preparing environment
INFO source :: retrieving snapshots
INFO destination :: retrieving snapshots
INFO source :: creating snapshot
INFO source :: transferring snapshot
ERROR source :: Command ['ssh', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', 'nas02', 'ionice -c3 btrfs receive "/mnt/storage01/data/"'] failed with error code 1 (At snapshot .temp.da96dc2032cb4320a1d88771f9b4ae05
ERROR: unexpected EOF in stream.)
ERROR source :: Command ['bash', '-c', 'ionice -c3 btrfs send -p "/mnt/storage01/data/.sxbackup/sx-20190825-230640-utc" "/mnt/storage01/data/.sxbackup/.temp.da96dc2032cb4320a1d88771f9b4ae05"'] failed with error code 1 (At subvol /mnt/storage01/data/.sxbackup/.temp.da96dc2032cb4320a1d88771f9b4ae05
ERROR: send ioctl failed with -5: Input/output error)
INFO destination :: removing subvolume [/mnt/storage01/data/.temp.da96dc2032cb4320a1d88771f9b4ae05]
INFO source :: removing subvolume [/mnt/storage01/data/.sxbackup/.temp.da96dc2032cb4320a1d88771f9b4ae05]
ERROR Transferring snapshot failed

I already installed the actually version of btrfs-sxbackup.

Does someone knows the solution?

email notification doesn't work

Tried to use -m flag with it, but didn't received anything. Created configuration file /etc/btrfs-sxbackup.conf:
[Default]
EMAIL_RECIPIENT = '[email protected]'
It didn't helped either. What would be a right way to debug it?

Allow to specify ssh cypher

Allowing a user to specify the cypher / encryption implementation in ssh can speed things up considerably in certain scenarios. The idea is that some implementations are accelerated by AES-NI.

In my case, the optimum is ssh -c [email protected]. It depends on what is available on both the local and remote machines, varying from Linux distro to Linux distro etc.

Yearly options for retention are missing

More prolonged backup schemes would profit from the possibility for yearly retention policies.

Example: -dr "[...] 1y:1/y, 5y:none"

Emulating this via 12m:1/12m, 60m:none does not work.

Having q as descriptor for quarterly could be useful for certain use cases.

Off by one at retention

It looks like the retention setting is off by one. I set 3 for source retention and 6 for destination retention. But it keeps 4 and 7 snapshots:

[root@thales ~]# btrfs-sxbackup info /mnt/data/processing/
INFO btrfs-sxbackup v0.6.11
INFO source :: retrieving snapshots
INFO destination :: retrieving snapshots
   UUID                   0844e6ba-3620-487a-a379-6ad20b58d797
   Compress               True
   Source URL             /mnt/data/processing
   Source info            Linux 4.12.13-300.fc26.x86_64 #1 SMP Thu Sep 14 16:00:38 UTC 2017 GNU/Linux, btrfs-progs v4.9.1
   Source container       .sxbackup
   Source retention       3
   Source snapshots       sx-20171005-094616-utc: Thu Oct  5 11:46:16 2017 (+0200)
                          sx-20171005-094526-utc: Thu Oct  5 11:45:26 2017 (+0200)
                          sx-20171005-094443-utc: Thu Oct  5 11:44:43 2017 (+0200)
                          sx-20171005-094313-utc: Thu Oct  5 11:43:13 2017 (+0200)
   Destination URL        ssh://[email protected]:/mnt/data
   Destination info       Linux 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 GNU/Linux, btrfs-progs v4.9.1
   Destination retention  6
   Destination snapshots  sx-20171005-094616-utc: Thu Oct  5 11:46:16 2017 (+0200)
                          sx-20171005-094526-utc: Thu Oct  5 11:45:26 2017 (+0200)
                          sx-20171005-094443-utc: Thu Oct  5 11:44:43 2017 (+0200)
                          sx-20171005-094313-utc: Thu Oct  5 11:43:13 2017 (+0200)
                          sx-20171005-094152-utc: Thu Oct  5 11:41:52 2017 (+0200)
                          sx-20171005-094048-utc: Thu Oct  5 11:40:48 2017 (+0200)
                          sx-20171005-093907-utc: Thu Oct  5 11:39:07 2017 (+0200)

Error at the end of incremental backup

Hi,

I'm getting an error at the end of an incremental backup initiated from the backup server.

Moreover it seems that the backup is working fine...

ERROR unsupported operand type(s) for /: 'datetime.timedelta' and 'NoneType'

If it mind, here is my retention settings : "retention = 1d:1, 1w:daily, 1m:1/w, 2m:none"

Best regards,

Error when destination has no snapshots

When running a backup with any amount of snapshots in the source but no snapshots in the destination, the program crashes with the following output:

INFO btrfs-sxbackup v0.6.7
DEBUG ['bash', '-c', 'if [ -f "/source/.btrfs-sxbackup" ]; then exit 10; fi']
DEBUG ['bash', '-c', 'cat "/source/.sxbackup/.btrfs-sxbackup"']
DEBUG ['bash', '-c', 'cat "/destination/.btrfs-sxbackup"']
INFO source :: url [/source/] container [.sxbackup/] retention [3] compress [False]
INFO destination :: url [/destination/] retention [1d:4/d, 1w:2/w, 3m:2/m, 12m:none] compress [False]
INFO preparing environment
DEBUG ['bash', '-c', 'if [ ! -d "/source/.sxbackup/" ]; then btrfs sub create "/source/.sxbackup/"; fi']
DEBUG ['bash', '-c', 'btrfs sub show "/source/.sxbackup/"']
DEBUG ['bash', '-c', 'if [ -d "/source/.sxbackup/.temp"* ]; then btrfs sub del "/source/.sxbackup/.temp"*; fi']
DEBUG ['bash', '-c', 'if [ ! -d "/destination/" ]; then btrfs sub create "/destination/"; fi']
DEBUG ['bash', '-c', 'btrfs sub show "/destination/"']
DEBUG ['bash', '-c', 'if [ -d "/destination/.temp"* ]; then btrfs sub del "/destination/.temp"*; fi']
INFO source :: retrieving snapshots
DEBUG ['bash', '-c', 'btrfs sub list -o "/source/.sxbackup/"']
INFO destination :: retrieving snapshots
DEBUG ['bash', '-c', 'btrfs sub list -o "/destination/"']
ERROR list index out of range
ERROR Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/btrfs_sxbackup/__main__.py", line 216, in main
    job.run()
  File "/usr/local/lib/python3.4/dist-packages/btrfs_sxbackup/core.py", line 971, in run
    and self.source.snapshots[0].name.timestamp != self.destination.snapshots[0].name.timestamp:
IndexError: list index out of range

In case of an empty destination a full snapshot should probably be transferred, in which case the check for a non-matching snapshot should be modified.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.