It would be nice to be able to compress (or at least package) backups about xbmcbackup HOT 8 CLOSED

I agree this would be something kind of nice to have. Every time I think about a good way to implement this the main trouble piece is limitations to the xbmcvfs module. This module is excellent in that it is the "backbone" of the backup addon. It handles all the underlying os file code and allows transparent use of the various protocols for reading/writing files. The downside is that it would be extremely difficult to implement any sort of compression since the python module can do very limited operations on individual folders/files. Once the files are on the remote system I can't really do anything with them.

One thing I have been wondering about though is only transferring modified files. With Gotham there are some methods to get modified and creation times for files now. Only transferring what needs to be transferred would be kind of nice for limiting bandwidth, especially if you are moving those off-site via another syncing tool or using the Dropbox method directly. This would almost be getting in the realm of "incremental" backups so there would need to be a common starting point (full backup) and rotation scheme but could be doable. Would something like this help you out?

from xbmcbackup.

You're right, the xbmcvfs doesn't really allow a clean solution for writing a compressed file. Although it should be possible to let Python zipfile the data using the ZipFile module combined with a xbmcvfs.File object. Since it behaves as a normal file including seek methods.

Effectively it should be possible to do this (theoretically of course):

import zipfile
import xbmcvfs

def compress_files(filename, *files)
    file_ = xbmcvfs.File(filename, 'w')
    with file_.ZipFile(file_, 'w') as zip:
        for file_ in files:
            zip.writestr(filename, xbmcvfs.File(file_).read())

But a differential copy would help as well, is it possible to set the mtime using the xbmcvfs api actually?

from xbmcbackup.

That is very interesting, I hadn't thought of using the File class and doing it that way. I'd have to do some testing but I wonder if this would "play nice" if trying to zip directories across network shares? I'll have to take your code for a spin and see what happens in different situations.

As you said earlier, zipping the file will only gain you so much if your archives are very large anyway. Maybe a combination of compression and incremental copies would help reduce the total size? More investigation is needed for sure. Definitely given me a lot to think about here.

from xbmcbackup.

move the files locally then zip the backup dir then transfer the files? this isn't possible?

from xbmcbackup.

This is possible, since the compression would then happen locally and is totally doable. I guess my reservation with this is you have to make sure the local drive has enough space for this. This can be done fairly trivally but I guess I have a personal objection to it. Just seems like you shouldn't require backup software to duplicate files on your local filesystem before sending it off for archiving. For small form factor systems (thinking RPI type stuff here) the SD card probably won't have the space for this. Not really the addon's problem I guess but seems kind of sloppy.

On the flip side - is having the compression worth the extra file space on the frontend? If you're backing up to the cloud lots of people would probably say yes.

from xbmcbackup.

I back up to a store over cifs, this means that it's very slow (since it's one file at a time). Doing it locally will make it much faster on top of reducing storage space.

from xbmcbackup.

Take a look at the request I just merged in. Hopefully this will help with the compression issue. I posted some comprehensive info about how it all works here: http://forum.xbmc.org/showthread.php?tid=129499&pid=1763652#pid1763652

from xbmcbackup.

yep, very nice, can't test this week though, will let you know if I find any issues though

from xbmcbackup.