abbbi / qmpbackup Goto Github PK
View Code? Open in Web Editor NEWLive Qemu Incremental backup using dirty-bitmaps
License: GNU General Public License v3.0
Live Qemu Incremental backup using dirty-bitmaps
License: GNU General Public License v3.0
New library for async qmp is available:
https://jsnow.gitlab.io/qemu.qmp/
one may check wether if it is possible to port the qmp transactions to the library calls.
qmpbackup uses qemu command "drive-backup" to initiate backup operation:
https://github.com/abbbi/qmpbackup/blob/master/libqmpbackup/qmpcommon.py#L153
this command will be deprecated as with qemu 6.2
There is a new blockdev-backup operation available:
needs porting to support newer qemu versions.
I am trying to test your program but I am experiencing a big problem:
root@immortality-lvm:~# qmpbackup --socket /tmp/qmp-socket backup --level auto --monthly --target /aoe
[2022-01-24 17:02:05,267] INFO Version: 0.5
[2022-01-24 17:02:05,268] INFO Backup target directory: /aoe/2022-01
[2022-01-24 17:02:05,268] INFO FULL Backup operation: "/aoe/2022-01/drive-virtio-disk1/FULL-1643040125"
Traceback (most recent call last):
File "/usr/bin/qmpbackup", line 4, in
import('pkg_resources').run_script('qmpbackup==0.5', 'qmpbackup')
File "/usr/lib/python3.9/site-packages/pkg_resources/init.py", line 651, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3.9/site-packages/pkg_resources/init.py", line 1455, in run_script
exec(script_code, namespace, namespace)
File "/usr/lib/python3.9/site-packages/qmpbackup-0.5-py3.9.egg/EGG-INFO/scripts/qmpbackup", line 240, in
File "/usr/lib/python3.9/site-packages/qmpbackup-0.5-py3.9.egg/libqmpbackup/qmpcommon.py", line 181, in do_full_backup_with_bitmap
File "/usr/lib/python3.9/site-packages/qmpbackup-0.5-py3.9.egg/libqmpbackup/qmpcommon.py", line 92, in check_qmp_return
Exception: Bitmap already exists: qmpbackup-drive-virtio-disk1
root@immortality-lvm:~# qmpbackup --socket /tmp/qmp-socket backup --level inc --monthly --target /aoe
[2022-01-24 17:02:18,890] INFO Version: 0.5
[2022-01-24 17:02:18,891] INFO Backup target directory: /aoe/2022-01
[2022-01-24 17:02:18,891] ERROR Incremental backup requested but no active bitmap has been found.
"no active bitmap" is reported but the dirty bitmap is inside the QCOW file:
2022-01-24 17:03:49,219] INFO VM Block Device Information:
[2022-01-24 17:03:49,219] INFO [
{
"device": "pflash0",
"inserted": {
"backing_file_depth": 0,
"bps": 0,
"bps_rd": 0,
"bps_wr": 0,
"cache": {
"direct": false,
"no-flush": false,
"writeback": true
},
"detect_zeroes": "off",
"drv": "raw",
"encrypted": false,
"file": "/usr/share/OVMF/OVMF_CODE.fd",
"image": {
"actual-size": 3657728,
"dirty-flag": false,
"filename": "/usr/share/OVMF/OVMF_CODE.fd",
"format": "raw",
"virtual-size": 3653632
},
"iops": 0,
"iops_rd": 0,
"iops_wr": 0,
"node-name": "#block157",
"ro": true,
"write_threshold": 0
},
"locked": false,
"qdev": "/machine/system.flash0",
"removable": false,
"type": "unknown"
},
{
"device": "pflash1",
"inserted": {
"backing_file_depth": 0,
"bps": 0,
"bps_rd": 0,
"bps_wr": 0,
"cache": {
"direct": false,
"no-flush": false,
"writeback": true
},
"detect_zeroes": "off",
"drv": "raw",
"encrypted": false,
"file": "/dev/shm/OVMF_VARS.fd",
"image": {
"actual-size": 540672,
"dirty-flag": false,
"filename": "/dev/shm/OVMF_VARS.fd",
"format": "raw",
"virtual-size": 540672
},
"iops": 0,
"iops_rd": 0,
"iops_wr": 0,
"node-name": "#block393",
"ro": false,
"write_threshold": 0
},
"locked": false,
"qdev": "/machine/system.flash1",
"removable": false,
"type": "unknown"
},
{
"device": "drive-virtio-disk1",
"inserted": {
"backing_file_depth": 0,
"bps": 0,
"bps_rd": 0,
"bps_wr": 0,
"cache": {
"direct": false,
"no-flush": false,
"writeback": true
},
"detect_zeroes": "unmap",
"dirty-bitmaps": [
{
"busy": false,
"count": 1093664768,
"granularity": 65536,
"name": "qmpbackup-drive-virtio-disk1",
"persistent": true,
"recording": true
}
],
"drv": "qcow2",
"encrypted": false,
"file": "/imm-100/wyk2022.qcow2",
"image": {
"actual-size": 53120364544,
"cluster-size": 262144,
"dirty-flag": true,
"filename": "/imm-100/wyk2022.qcow2",
"format": "qcow2",
"format-specific": {
"data": {
"compat": "1.1",
"compression-type": "zlib",
"corrupt": false,
"extended-l2": true,
"lazy-refcounts": true,
"refcount-bits": 16
},
"type": "qcow2"
},
"virtual-size": 107374182400
},
"iops": 0,
"iops_rd": 0,
"iops_wr": 0,
"node-name": "#block522",
"ro": false,
"write_threshold": 0
},
"io-status": "ok",
"locked": false,
"qdev": "/machine/peripheral/virtio-disk1/virtio-backend",
"removable": false,
"type": "unknown"
}
]
The durty bitmap is there,
BUT
[2022-01-24 17:04:46,972] INFO Version: 0.5
[2022-01-24 17:04:46,973] INFO Bitmaps:
[2022-01-24 17:04:46,973] INFO No bitmaps found for device: "drive-virtio-disk1"
Am I doing something wrong ?
Currently pflashX type devices (usually used for OVM BIOS) are not part of the backup, they are excluded.
qmpbackup could save these files too, to be able to restore UEFI related variable settings. It should be sufficient to simply copy the files to the backup folder.
Example:
[2024-02-14 08:42:16,900] WARNING Excluding device with raw format from backup: [pflash0:OVMF_CODE_4M.fd]
[2024-02-14 08:42:16,900] WARNING Excluding device with raw format from backup: [pflash1:OVMF_VARS.fd]
Full backup writes to targetdir/disk/FULL-
If backup operation fails for some reason, future incremental backups may be based on an non successful full backup.
(im not sure if qemu resets the bitmap with sync=incremental if an block job fails for some rason)
Target file should be set to targetdir/disk/FULL-.partial and file should be renamed if block job was
successful.
Additionally introduce check that looks at the target backup directory if it contains files with .partial ending and stop
any incremental backup operation if so, to ensure consistency.
Add exclude option to exclude certain disks from backup
Currently qmpbackup uses the push based approach, which can only work by dumping the data to a certain directory,
this means the utility has to be executed on the same system running the qemu processes.
With the pull based approach (NBD based), remote backup would be possible.
Hello.
Currently, when there is backing file + file, the backup for this device does not work.
json:{"backing": {"driver": "qcow2", "file": {"driver": "file", "filename": "/hdd/OC.qcow2"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "/var/tmp/vl.RUCII2"}} (qcow2)
So, for backup automation, would be very useful option like "--auto-commit" or something.
Currently, backup status should be monitored, and commits performed manualy (via qmp-shell), if scheduled backup failed because of such situation.
Currently disk backup is implemented sequentially, this means that the bitmap for the second disk is created after the backup for the first disk has finished. This could lead to inconsistencies, because the bitmaps are not created within one transaction.
Behavior should be reworked so the bitmaps for all attached disks are created within one QMP transaction during full backup, not one after another.
Add signal handler which:
The qemu agent client still uses the qmp class as shipped, it should be possible to use qemu.qmp library for the agent communication, too. negotiate option must be set to false, ..
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.