rugk / borg-cron-helper Goto Github PK

Helper shell scripts for BorgBackup to automate backups and make your life easier… 😉

License: MIT License

Shell 100.00%

borg backup backups borgbackup shell shell-script posix posix-compatible automation cron

borg-cron-helper's Introduction

Borg cron helper scripts

Automate backups with borg in a more convenient and reliable way! These scripts are some small and handy shell scripts to automate the backup process with BorgBackup. They are POSIX-compatible, so they should run with all shells. You're free to modify them for your needs!

They add some convenient features around borg, regarding environments with only one client.

Features

Local lock system: Circumvent the issue of stale lock files.
Automated retries: When backups stop mid-way they are automatically restarted.
Simple configuration: Using shell-scripts you can configure each backup and then execute it in the order/way you want.
Status information: You can use a login script to get a notice when backups failed.
For servers & desktops: The script is also usable on desktops. But rest assured: We'll never forget servers.
Optional & adjustable: You do not have to use all features and you can adjust them in a simply way.

Local lock (borg < v1.1.0)

When the backup process is interrupted, sometimes the remote borg repository stays locked. That's why further backups will fail.

The issue has been addressed in borg v1.1.0, but I have not tested it and until it works there, here is my workaround.

Borg's current PID is written into a file. As long as the file exists the backup is considered to be "locked". At the end of the backup process (no matter whether it was successful or not), the local lock is being removed, permitting further backups to start. The "local lock" is more reliable than the "remote lock" system, currently implemented in stable versions of borg.

Less maintenance, more safety!

Sometimes backups stop mid-way. This can have different reasons (e.g. an unreliable network). However we still want our data to be backed up. That's why the script integrates a retry mechanism to retry the backup in case of a failure (for up to three times by default). Between the retry attempts the script pauses some minutes by default, to wait for your server to restart or the connection to reestablish. This is also a workaround for the "connection closed by remote" issue, which seems to affect some users.

Additionally the script has the ability to write stats of the last executed backup. cronsizecache.sh outputs the size of backups stored on the backup server. Both can be used (in conjunction with a script to check the last backup time) to monitor your backups and to automatically notify you, in case a backup failed/didn't work.

Rest assured your backups are safe and recover themselve if possible. (If they don't, you'll get to know that.)

Desktop integration

The whole script can, of course, run on servers, but some features making it also suitable for desktops. It can display notifications about started and finished backups, including its result and it can even ask the user, whether to retry the backup in case it failed. Or you ask the user about the password for your backup interactively, in a GUI window.

It has been tested with GNOME, but should work on any system, where zenity is installed. And: You can always easily adjust it to work with a different program.

It is even possible to configure it to show the notification as a "standard" user while the backup is actually running as root.

For examples, see the wiki page about more ways for GUI integration.

Modular approach

The main "work" is done by borgcron.sh. This script can be used to execute a single backup. The configuration files per repository/backup are saved in the config directory. The file borgcron_starter.sh is the file you should call from cron or when debugging manually. Using it, you can execute multiple backups by passing the config names to it.

Also, you can of course not use some features outlined here. That's why the whole functionality is broken into multiple scripts and configurable in a single config file per backup.

More features...

easy to understand, easy to modify
POSIX-compatible
pretty logs
passphrase can be protected better, because it's saved in an external file
pruning after the backup
script to dump databases before backing up
privilege-separation (login scripts can have higher privilege than backup process)
tested and used in production (but no guarantees, use at your own risk! 😉)
logging if backup is interruped by signals (e.g. at shutdown)
tested with more than 30 automated (unit) tests (see Travis-CI badge above)
more to come...

What's in here?

borgcron_starter.sh – Cycles through backup files and interprets passed parameters.
borgcron.sh Main script. Actually executes the backup and runs borg.
config – Directory for config files
- example-backup.sh – Example configuration file for a backup. Please use this template to add your backup(s).
tools – additional scripts
- checklastbackup.sh – Script, which you can execute at login (add it to your .bashrc file or so). It notifies you when a backup has failed. Otherwise it remains silent.
- cronsizecache.sh – Small one-liner to cache the size of the dir where backups are stored. (useful for remote backup servers) You can then include the result with cat in your login script.
- databasedump.sh – Dumps one or several databases into a dir/file. Make sure that this script and the dump dir are only readable by your backup user. Script might have to be executed with higher privileges (i.e. root) for creating the backup.
system – Various system scripts, you may need for your setup.
tests – Scripts for unit testing, etc. See Readme inside of it, not needed in production.

How to setup?

Setup instructions can be found in the wiki.

borg-cron-helper's People

Contributors

Stargazers

Watchers

Forkers

m3adow csskysea dcherian patlegu geotexan nicholas-gh cod3fr3ak shankari kinafu wikinaut heartshare cheoppy w3bservice

borg-cron-helper's Issues

Restructure/Refactor files (translations, common functions)

For translations extra files would be useful. (maybe a dir like translations/de-DE.sh)

And when we are at it, we could also put functions into a file, which can be load by many other scripts. This would possibly also be useful for #32 and we could reduce code duplication (in borgcron.sh and borgcron_starter.sh).
A file like commons.sh or functions.sh could then also be loaded by unit tests, which makes testing easier…

Retry pruning

Hello,
I have an unreliable connection, and decided to use borg-cron-helper to get a reliable backup procedure. Thanks for that! 👍
The problem is that my connection got disconnected during the pruning phase, and that phase isn't protected. Could it be protected with the same principles?

Rewrite messages of unit test

Currently all unit test messages (the ones in "assert…") describe the case when the test/assert passes/is satisfied. However, as they are only printed when the assert actually fails, they should better describe the opposite case: What actually fails.

This is an easy wording task. Just turn around the meaning of all these messages. (Remember to use present tense.)

Example: "Should print 1234."
Change to: "Does not print 1234."

Restructure Readme with mostly bullet points?

Currently we have a big wall of text. (okay structured though, but still continuous text) May it be more suitable for the internet to just use shorter bullet points and less sentences?

Support excluding directories from backup

I looked at the example parameters, and there is support for adding directories (ADD_BACKUP_PARAMS). However, there is no support for excluding directories from the backup.

The default automating example on the borg backup page (https://borgbackup.readthedocs.io/en/1.1-maint/quickstart.html#automating-backups) adds /var and excludes '/var/cache/*' and '/var/tmp/*', and I think I'd like to follow a similar pattern.

Error on borgcron_starter.sh execution

I cloned the git repo on my local machine.
I edited the example backup script to fit my needs and renamed it to home-backup.sh.
Then, I followed your guide on how to set up, finally, I executed borgcron_starter.sh manually to check if everything works.
This is the output:

[2019-01-15 00:40:23] Backup home started with borg 1.1.8, helper PID: 5490.
usage: borg create [-h] [--critical] [--error] [--warning] [--info] [--debug]
                   [--debug-topic TOPIC] [-p] [--log-json]
                   [--lock-wait SECONDS] [--show-version] [--show-rc]
                   [--umask M] [--remote-path PATH] [--remote-ratelimit RATE]
                   [--consider-part-files] [--debug-profile FILE] [-n | -s]
                   [--list] [--filter STATUSCHARS] [--json] [--no-cache-sync]
                   [--no-files-cache] [--stdin-name NAME] [-e PATTERN]
                   [--exclude-from EXCLUDEFILE] [--pattern PATTERN]
                   [--patterns-from PATTERNFILE] [--exclude-caches]
                   [--exclude-if-present NAME] [--keep-exclude-tags]
                   [--exclude-nodump] [-x] [--numeric-owner] [--noatime]
                   [--noctime] [--nobirthtime] [--nobsdflags] [--ignore-inode]
                   [--files-cache MODE] [--read-special] [--comment COMMENT]
                   [--timestamp TIMESTAMP] [-c SECONDS]
                   [--chunker-params PARAMS] [-C COMPRESSION]
                   ARCHIVE [PATH [PATH ...]]
borg create: error: argument ARCHIVE: "/home/philipp/..": No archive specified
[2019-01-15 00:40:24] Borg exited with fatal error.
[2019-01-15 00:40:24] Wait 5m…

# I pressed Ctrl+C here, for obvious reasons
^C[2019-01-15 00:40:26] Backup home (PID: 5490) interrupted by INT.
[2019-01-15 00:40:26] Showing zenity (notification) to philipp (id=1000).
/usr/bin/sudo

** (zenity:5536): WARNING **: 00:40:26.581: Error showing notification: Verbindung ist gescheitert: Datei oder Verzeichnis nicht gefunden

How is this borg create error possible? I didn't touch the borg command at all - the only place where I changed borgcron.sh is here.
See the personal-changes branch in my fork for all changes I made.

Can you spot my mistake?

Provide init.d and systemd scripts for creation of PID dir

To make the setup process easier.

Last fail when retrying should not trigger waiting

What happens

[start]

Execute borg. -> fails
Wait
Retry: Execute borg. -> fails
Wait
Retry: Execute borg. -> fails
[…]
(last retry) Retry: Execute borg. -> fails
Wait

[prune/exit]

What should happen

The last sleep time is useless. We do not need to wait if we will not retry afterwards, anyway.

Implementation

Note that this requires unit test changes. Some TODOs already added.

Add donation button

https://liberapay.com/rugk 😊

If borg throws an exception, borg-cron-helper reports a "warning"

…"it took 0 seconds to complete".

This is a little misleading, as borg actually throws many, many exceptions as the log shows:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
[2019-05-01 00:43:15] Backup AcerTravel started with , helper PID: 7108.
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
[2019-05-01 00:43:16] Borg had some WARNINGS, but everything else was okay.
Running prune for "AcerTravel"…
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
[2019-05-01 00:43:17] Backup "AcerTravel" ended, but something seems to went wrong.
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
[2019-05-02 08:01:46] Backup AcerTravel started with , helper PID: 3934.
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
[2019-05-02 08:01:47] Borg had some WARNINGS, but everything else was okay.
Running prune for "AcerTravel"…
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
[2019-05-02 08:01:48] Backup "AcerTravel" ended, but something seems to went wrong.
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirement.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6' distribution was not found and is required by borgbackup

The exceptions itself are a known issue (see the Fedora bug report). What matters for us is that borg seems to end with error code 1 when it encounters an exception. However, borg defines this as a "warning" only, i.e. no serious error…

Allow customisation of retry count

Possibly using the seq command and allowing to customize it per backup.

Use trap

The official example uses a trap to catch some signals sent to the script. We may do this too… 😄

Generate parameter handling

We could use https://argbash.readthedocs.io/en/latest/ to generate the parameter handling programmatically, making it easy and "get it right". Also we have plenty of space in borgcron_starter.sh as this script mainly just contains the param parsing part, so we could use it and keep the main script clear.
And if the code is really too much argbash accepts a --mode minimal to make it less, if we want…

Append-only mode helper handling

Nice new feature upcoming… (borgbackup/borg#36, borgbackup/borg#1772)

Verify shunit2 in unit tests

with public key, see kward/shunit2#76

Example for backing up snaps

Feel free to add an example somewhere around here in the example script for how to backup snaps.

Similar to #68

Test coverage for databasedump.sh

We need to create a unit test :)

autoupdate borg-cron-helper scripts

I'd like to automate my setup so that I don't have to fiddle around with manual updates. With your current release strategy, it seems like the easiest option to update is to pull from master periodically (maybe weekly?).

So my questions are:

is this a reasonable option? can I assume that most development work happens on a branch and master is typically stable?
does it make sense to add the git pull to the configuration scripts? I personally don't like this option because it is insufficiently modular and adds a network dependency for the backup script.
would you accept a PR for new system scripts for the git pull? I would only contribute scripts for systemctl since that is what I'm using.

`borg.timer` needs to be both enabled and started with systemctl

I followed the instructions to enable automatic backups, however, the timer was listed as inactive.

# systemctl list-timers --all
NEXT                         LEFT          LAST                         PASSED     UNIT
Sun 2018-12-30 13:03:36 PST  5min left     Sun 2018-12-30 12:04:22 PST  53min ago  anacr
Sun 2018-12-30 14:13:44 PST  1h 16min left Sun 2018-12-30 05:43:41 PST  7h ago     certb
Sun 2018-12-30 15:15:13 PST  2h 17min left Sun 2018-12-30 02:21:41 PST  10h ago    motd-
Sun 2018-12-30 16:05:45 PST  3h 8min left  Sun 2018-12-30 05:22:09 PST  7h ago     apt-d
Mon 2018-12-31 00:00:00 PST  11h left      Mon 2018-12-24 00:00:40 PST  6 days ago fstri
Mon 2018-12-31 06:08:26 PST  17h left      Sun 2018-12-30 06:29:34 PST  6h ago     apt-d
Mon 2018-12-31 12:12:08 PST  23h left      Sun 2018-12-30 12:12:08 PST  45min ago  syste
n/a                          n/a           n/a                          n/a        borg.
n/a                          n/a           Thu 2018-12-27 11:58:12 PST  3 days ago uread

I poked around on the internet and found https://www.certdepot.net/rhel7-use-systemd-timers/
So I started the timer as well, and then it showed as active.

# systemctl start borg.timer
# systemctl list-timers --all
NEXT                         LEFT          LAST                         PASSED     UNIT
Sun 2018-12-30 13:03:36 PST  4s left       Sun 2018-12-30 12:04:22 PST  59min ago  anacr
Sun 2018-12-30 14:13:44 PST  1h 10min left Sun 2018-12-30 05:43:41 PST  7h ago     certb
Sun 2018-12-30 15:15:13 PST  2h 11min left Sun 2018-12-30 02:21:41 PST  10h ago    motd-
Sun 2018-12-30 16:05:45 PST  3h 2min left  Sun 2018-12-30 05:22:09 PST  7h ago     apt-d
Mon 2018-12-31 00:00:00 PST  10h left      Mon 2018-12-24 00:00:40 PST  6 days ago fstri
Mon 2018-12-31 00:55:39 PST  11h left      n/a                          n/a        borg.
Mon 2018-12-31 06:08:26 PST  17h left      Sun 2018-12-30 06:29:34 PST  6h ago     apt-d
Mon 2018-12-31 12:12:08 PST  23h left      Sun 2018-12-30 12:12:08 PST  51min ago  syste
n/a                          n/a           Thu 2018-12-27 11:58:12 PST  3 days ago uread

The wiki needs to be updated to include $ systemctl start borg.timer in addition to systemctl enable borg.timer

zenity : Error showing notification

Backups seem to be working properly but I'm getting following error when using zenity :

Mar 20 00:15:06 myhost borgcron_starter.sh[9251]: [2018-03-20 00:15:06] Backup "test-backup" ended successfully.
Mar 20 00:15:06 myhost dbus-daemon[9298]: [session uid=1000 pid=9296] Activating service name='org.freedesktop.Notifications' requested by ':1.0' (uid=1000 pid=9291 comm="zenity --notification --window-icon /bin/borg-cron")
Mar 20 00:15:31 myhost zenity[9291]: Error showing notification: Error calling StartServiceByName for org.freedesktop.Notifications: Timeout was reached
Mar 20 00:15:34 myhost systemd[1]: Started BorgBackup.

I have replaced this line in borgcron.sh:
zenity --notification --window-icon "$icon" --text "BorgBackup: $BACKUP_NAME
with this one:
kdialog --title "BorgBackup" --icon "$icon" --passivepopup "BorgBackup: $BACKUP_NAME

just to see if it works with something else and now a notification appears.

OS: Arch with KDE Plasma

Move setup into wiki?

The Readme seems to get quite large. Maybe better move the setup instructions into the wiki…

Switch to BORG_PASSCOMMAND

v1.1.0 of borg has a new envoriment variable called BORG_PASSCOMMAND. It is somewhat more secure to use that if you have the password saved in a file and just "cat it" there.

No releases - how to properly set up?

Hi all,
first of all I want to say thanks for theses scripts. I've just recently found borg-cron-helper and am planning to build it into my borg container Therefore I want to automate the installation of borg-cron-helper. The Readme states to use releases and compare them against a PGP key. But there are no releases (yet?).

Are future releases planned or shall I just use git to get the repo?

Either way this is a bug report either for the missing of releases described in the README or for a wrong set up description in the README.

Pause backup while being on metered connection

With Gnome NetworkManager one can set, wheter a connection is metered (other word for metered is "fee-based").
On a metered connection, smart programs will stop unimportant background activities such as system updates, backup jobs and so on.

It comes handy to set the property to yes in situations like a capped data plan or on a generally slow WiFi connection.

Borg-cron-helper should also respect the metered property and take a break.

Add possibility to pass mysqldump parameters

For my nextcloud (backup) I need to pass an option (--default-character-set=utf8mb4) to the mysqldump command (in order to properly backup strings containing unicode smileys).

Using our tool databasedump.sh you should be able to pass arguments or environment variables to mysqldump.

If you're curious the nextcloud docs gave me the hint to check, wether one's backup strategy will still work with its new way of storing data in mysql.

Fix Travis-CI build

Travis-CI broke their apt addon for my things… or whatever…

Maybe need to use full VMs… 😠

Laptop features (wifi, low battery)

Maybe it is possible to integrate it a bit more here, too…

Provide systemd timer scripts

https://wiki.archlinux.org/index.php/Systemd/Timers

Disable desktop notifications in terminal-only mode

Right now, borgcronhelper attempts to show notifications or at least tries to do sth. like that by default.
I use borgcronhelper on a headless server and can get notified only by cron/chronic.

With the default setting, the backup report always contains the error:
(zenity:29707): Gtk-WARNING **: cannot open display:

My proposal:
In each backup config we need a variable to drop desktop notifications.
Right now, you can change borgcron.sh to deactivate the feature. But changes to borgcron.sh would become overwritten by updates to borg-cron-helper.

Ability to invoke databasedump.sh with parameters

Right now it's possible, to write username and password of the db-user into the mysql backup script directly.
The ability to have this kind of interface
$ ./databasedump.sh backup password /root/backup/db
additionally to

USER="backup"
PASSWORD="password"
DIR="/root/backup/db"

would help in avoiding redundancy and facilitate maintainability.

Right now the easiest way is to create copies of databasedump.sh with "hard-coded" credentials.
If one needs to make a change in the code executing the backup, he would need to update all of his copies.

Docker image

I'm investigating whether an official Docker image would be useful for this.

There is unfortunately no official Docker image (borgbackup/borg#4364), so we cannot base it on there and there are a lot of more or less popular images on Docker Hub.

Redirect error messages to stderr in borgcron_starter.sh

In borgcron_starter.sh there are some error messages, that are sent to stdout.
They can be written to stderr, to get unix-consistent behaviour.

Consider notify-send instead of zenity

There is also a command called notify-send.

I don't know whether it is better or worse than zenity, but we should evaluate it.

AFAIK it does (and can do) exactly the same… 🤔

License change

I am going to change the license from CC0 to MIT license for borgcron.sh. Naturally old code stays licensed under the old license, but any new addition afterwards is licensed under the new license.
All other scripts will stay CC0-licensed as they are really minimal.

Collecting contributor agreements here:

@kinafu

Update shunit2

We can now use the stable new release: https://github.com/kward/shunit2/releases/tag/v2.1.7

Backup details in notification

Show this in the notification, when a backup succeeded:

backup time (rather use the one calculated by borg itself, not calculate it by yourself)
backup size (of course, the deduplicated/compressed version of only this backup)
whole backup size (deduplicated/compressed) of all backups – Does this make sense to see or is it too much information?

Implementation

The big problem is how to get the output of borg, while also letting it flow into STDOUT. I think, the best way may be to use tee to additionally write the output into a file and then read (cat) cat it.

Improve formatting of error message in unit tests

Currently when one assert fails in a unit test, we get a huge bunch of one-liner output.

It would be good to split that into multiple lines, so it is more readable.

Improve vulnerability reporting process

add PGP key for mail
add info to Readme

Rework unit tests to be less silent

See https://github.com/kward/shunit2/wiki/Cookbook

Basically:

do not make the tests as silent as they currently are
run code inside the tests, catch output (including STDERR!) and exit code
include output/exit code in your assert message, so the output is shown, when the assert fails

Just to make debugging easier.

Depends on #23 as I'd first have to have the small changes in the code.

Run prune even if backup fails

The official example executes prune even when the backup fails. This may be an idea, but I am not yet sure whether it is really useful. In most cases when the backup fails you cannot prune anyway afterwards…

Remove local lock system/Require borg v1.1.0

Depreciating old software is never easy, but in v2.0 we may:

still have a 1.0.0 branch, which receives security fixes (and maybe critical bugfixes) if needed
continue development for borg >= v1.1.0 only, i.e. we can remove the whole local lock system as borg 1.1.0 has fixed the originating issue

Restructure Readme

Structure:

short intro (can keep current one)
short list of features (move from bottom)
complete feature list (the big part currently), but remove setup instructions (they come below), only shows aims/what is solved/idea behind that system
file list (as currently)
"How to setup?" (step by step instructions what to do, also explains what file does what), maybe with sub-headlines for each feature; warnings/restrictions of scripts also go here

Export variables directly in config file

First setting $REPOSITORY and then exporting it as BORG_REPO later, is unnecessary. Adjust the template…

Also add other useful environment variables of borg.

Of course this is a breaking change.

Proper exit code

@kinafu I see you already did some stuff in kinafu@835af05.

Do you want to do it (in a new branch) here as well?

Features should be:

exit with highest exit code, when some borg commands fail
the thing is called "exit code" on Linux, so please use this term (in your variables, etc.)
skipping prune when borg exited with exit code "2" is a good idea, you can include it here, too (for future changes maybe use a variable skipprune or similar, which is set independently from the exit code, as we may introduce further conditions, where prune should be skipped)
do not forget to also handle the exit code in borgcron_starter.sh

Number is rounded down

Here (?) seems to be a bug, so that the time 4 minutes 38 seconds is shown as 4 minutes.

Mac/OS X compatibility & unit tests

I tried to run the tests in OS X via Travis, but this failed horribly. See this build for when this is happening.

I also see that the main borgcron.sh script fails (it e.g. cannot do locking) and thus it is very likely incompatible.
If anyone wants to fix that, make the script compatible with OS X and fix the unit tests, feel free to do so. PRs are appreciated.

You would also have to fix things like these strange distorted character display, which comes from here and is correctly displayed only when the variable is accessed another time. (?! 😯)

Notification support

Using zenity we may display notifications. Of course, this only makes sense on non-server systems and only when installed, but that can easily be checked (just check whether the program is there).

rugk / borg-cron-helper Goto Github PK

borg-cron-helper's Introduction

Borg cron helper scripts

Features

Local lock (borg < v1.1.0)

Less maintenance, more safety!

Desktop integration

Modular approach

More features...

What's in here?

How to setup?

borg-cron-helper's People

Contributors

Stargazers

Watchers

Forkers

borg-cron-helper's Issues

What happens

What should happen

Implementation

Implementation

Recommend Projects

Recommend Topics

Recommend Org