Currently qmpbackup uses the push based approach, which can only work by dumping the data to a certain directory,
this means the utility has to be executed on the same system running the qemu processes.

With the pull based approach (NBD based), remote backup would be possible.

Currently disk backup is implemented sequentially, this means that the bitmap for the second disk is created after the backup for the first disk has finished. This could lead to inconsistencies, because the bitmaps are not created within one transaction.

Behavior should be reworked so the bitmaps for all attached disks are created within one QMP transaction during full backup, not one after another.

qmpbackup uses qemu command "drive-backup" to initiate backup operation:

https://github.com/abbbi/qmpbackup/blob/master/libqmpbackup/qmpcommon.py#L153

this command will be deprecated as with qemu 6.2
There is a new blockdev-backup operation available:

https://qemu-project.gitlab.io/qemu/interop/live-block-operations.html#qmp-invocation-for-blockdev-backup

needs porting to support newer qemu versions.

Add signal handler which:

• correctly stops operation on sigint/
• stops the started block jobs on qemu side to cancel backup.

Hello.
Currently, when there is backing file + file, the backup for this device does not work.

json:{"backing": {"driver": "qcow2", "file": {"driver": "file", "filename": "/hdd/OC.qcow2"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "/var/tmp/vl.RUCII2"}} (qcow2)

So, for backup automation, would be very useful option like "--auto-commit" or something.

Currently, backup status should be monitored, and commits performed manualy (via qmp-shell), if scheduled backup failed because of such situation.

I am trying to test your program but I am experiencing a big problem:

• first "full" backup runs OK (very slowly, but the qcow file is created correctly)
• the "incremental" backup fails, both in "auto" and "inc" modes:

root@immortality-lvm:~# qmpbackup --socket /tmp/qmp-socket backup --level auto --monthly --target /aoe

[2022-01-24 17:02:05,267] INFO Version: 0.5
[2022-01-24 17:02:05,268] INFO Backup target directory: /aoe/2022-01
[2022-01-24 17:02:05,268] INFO FULL Backup operation: "/aoe/2022-01/drive-virtio-disk1/FULL-1643040125"
Traceback (most recent call last):
File "/usr/bin/qmpbackup", line 4, in
import('pkg_resources').run_script('qmpbackup==0.5', 'qmpbackup')
File "/usr/lib/python3.9/site-packages/pkg_resources/init.py", line 651, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3.9/site-packages/pkg_resources/init.py", line 1455, in run_script
exec(script_code, namespace, namespace)
File "/usr/lib/python3.9/site-packages/qmpbackup-0.5-py3.9.egg/EGG-INFO/scripts/qmpbackup", line 240, in
File "/usr/lib/python3.9/site-packages/qmpbackup-0.5-py3.9.egg/libqmpbackup/qmpcommon.py", line 181, in do_full_backup_with_bitmap
File "/usr/lib/python3.9/site-packages/qmpbackup-0.5-py3.9.egg/libqmpbackup/qmpcommon.py", line 92, in check_qmp_return
Exception: Bitmap already exists: qmpbackup-drive-virtio-disk1

root@immortality-lvm:~# qmpbackup --socket /tmp/qmp-socket backup --level inc --monthly --target /aoe
[2022-01-24 17:02:18,890] INFO Version: 0.5
[2022-01-24 17:02:18,891] INFO Backup target directory: /aoe/2022-01
[2022-01-24 17:02:18,891] ERROR Incremental backup requested but no active bitmap has been found.

"no active bitmap" is reported but the dirty bitmap is inside the QCOW file:

qmpbackup --socket /tmp/qmp-socket --debug info --show blockdev

2022-01-24 17:03:49,219] INFO VM Block Device Information:
[2022-01-24 17:03:49,219] INFO [
{
"device": "pflash0",
"inserted": {
"backing_file_depth": 0,
"bps": 0,
"bps_rd": 0,
"bps_wr": 0,
"cache": {
"direct": false,
"no-flush": false,
"writeback": true
},
"detect_zeroes": "off",
"drv": "raw",
"encrypted": false,
"file": "/usr/share/OVMF/OVMF_CODE.fd",
"image": {
"actual-size": 3657728,
"dirty-flag": false,
"filename": "/usr/share/OVMF/OVMF_CODE.fd",
"format": "raw",
"virtual-size": 3653632
},
"iops": 0,
"iops_rd": 0,
"iops_wr": 0,
"node-name": "#block157",
"ro": true,
"write_threshold": 0
},
"locked": false,
"qdev": "/machine/system.flash0",
"removable": false,
"type": "unknown"
},
{
"device": "pflash1",
"inserted": {
"backing_file_depth": 0,
"bps": 0,
"bps_rd": 0,
"bps_wr": 0,
"cache": {
"direct": false,
"no-flush": false,
"writeback": true
},
"detect_zeroes": "off",
"drv": "raw",
"encrypted": false,
"file": "/dev/shm/OVMF_VARS.fd",
"image": {
"actual-size": 540672,
"dirty-flag": false,
"filename": "/dev/shm/OVMF_VARS.fd",
"format": "raw",
"virtual-size": 540672
},
"iops": 0,
"iops_rd": 0,
"iops_wr": 0,
"node-name": "#block393",
"ro": false,
"write_threshold": 0
},
"locked": false,
"qdev": "/machine/system.flash1",
"removable": false,
"type": "unknown"
},
{
"device": "drive-virtio-disk1",
"inserted": {
"backing_file_depth": 0,
"bps": 0,
"bps_rd": 0,
"bps_wr": 0,
"cache": {
"direct": false,
"no-flush": false,
"writeback": true
},
"detect_zeroes": "unmap",
"dirty-bitmaps": [
{
"busy": false,
"count": 1093664768,
"granularity": 65536,
"name": "qmpbackup-drive-virtio-disk1",
"persistent": true,
"recording": true
}
],
"drv": "qcow2",
"encrypted": false,
"file": "/imm-100/wyk2022.qcow2",
"image": {
"actual-size": 53120364544,
"cluster-size": 262144,
"dirty-flag": true,
"filename": "/imm-100/wyk2022.qcow2",
"format": "qcow2",
"format-specific": {
"data": {
"compat": "1.1",
"compression-type": "zlib",
"corrupt": false,
"extended-l2": true,
"lazy-refcounts": true,
"refcount-bits": 16
},
"type": "qcow2"
},
"virtual-size": 107374182400
},
"iops": 0,
"iops_rd": 0,
"iops_wr": 0,
"node-name": "#block522",
"ro": false,
"write_threshold": 0
},
"io-status": "ok",
"locked": false,
"qdev": "/machine/peripheral/virtio-disk1/virtio-backend",
"removable": false,
"type": "unknown"
}
]

The durty bitmap is there,

BUT

qmpbackup --socket /tmp/qmp-socket --debug info --show bitmaps

[2022-01-24 17:04:46,972] INFO Version: 0.5
[2022-01-24 17:04:46,973] INFO Bitmaps:
[2022-01-24 17:04:46,973] INFO No bitmaps found for device: "drive-virtio-disk1"

Am I doing something wrong ?

Add exclude option to exclude certain disks from backup

Currently pflashX type devices (usually used for OVM BIOS) are not part of the backup, they are excluded.
qmpbackup could save these files too, to be able to restore UEFI related variable settings. It should be sufficient to simply copy the files to the backup folder.

Example:

[2024-02-14 08:42:16,900] WARNING  Excluding device with raw format from backup: [pflash0:OVMF_CODE_4M.fd]
[2024-02-14 08:42:16,900] WARNING  Excluding device with raw format from backup: [pflash1:OVMF_VARS.fd]

New library for async qmp is available:

https://jsnow.gitlab.io/qemu.qmp/

one may check wether if it is possible to port the qmp transactions to the library calls.

• dont create a bitmap for raw devices
• allways default to full backup without bitmap in case of raw device

The qemu agent client still uses the qmp class as shipped, it should be possible to use qemu.qmp library for the agent communication, too. negotiate option must be set to false, ..

Full backup writes to targetdir/disk/FULL-

If backup operation fails for some reason, future incremental backups may be based on an non successful full backup.
(im not sure if qemu resets the bitmap with sync=incremental if an block job fails for some rason)
Target file should be set to targetdir/disk/FULL-.partial and file should be renamed if block job was
successful.

Additionally introduce check that looks at the target backup directory if it contains files with .partial ending and stop
any incremental backup operation if so, to ensure consistency.