restic_backup.sh backs up supplied paths, but also the folder it's being executed from about restic-automatic-backup-scheduler HOT 13 CLOSED

I'm in the middle of a long upload (made an empty folder and ran the script from there) that I don't want to mess with, but I'll try it once that's done. What I can note from the currently running command is that it's not listed in the "start scan on [" and "start backup on [" lines but it is in the completed snapshot.

from restic-automatic-backup-scheduler.

Awesome. Thanks again for all the help and quick responses! Everything's working now and my god is it better than SpiderOak. Uploaded overnight what it took literally weeks to upload to them, and I finally have fine grained control of retention policies that actually work.

from restic-automatic-backup-scheduler.

The best way to figure out what’s is happening to look at the actual restic commandant that are being executed. Run the backup script with the bash -x flag and identify the restic backup line printed, this is the answer to what is being backed up

git-bash$ source /etc/restic/default.env.sh
git-bash$ restic init
git-bash$ bash -x restic_backup.sh

from restic-automatic-backup-scheduler.

I added a debugging section https://github.com/erikw/restic-automatic-backup-scheduler#debugging

from restic-automatic-backup-scheduler.

Here's the start of the command it's sending:

restic backup --verbose=1 --tag systemd.timer --option b2.connections=10 --exclude-file /etc/restic/backup_exclude.txt '' /C/backup /C/Code...

Is the '' causing the problem? It looks like that might be the empty backup_extra_args

from restic-automatic-backup-scheduler.

Oh this might be related
#122

Are you in the latest commit on the main branch of the repo?

from restic-automatic-backup-scheduler.

I have everything but today's commits, so I'm at the October 3rd "Add stale action" commit. Potentially the "Include minimal one argument in extra args array" commit from that pull request might fix it? But I don't understand bash script well enough to say for sure.

from restic-automatic-backup-scheduler.

Here's the start of the command it's sending:

restic backup --verbose=1 --tag systemd.timer --option b2.connections=10 --exclude-file /etc/restic/backup_exclude.txt '' /C/backup /C/Code...

Is the '' causing the problem? It looks like that might be the empty backup_extra_args

I'm trying to reproduce this, to get '' from $backup_extra_args, but haven't been able so far. Could you share how your /usr/local/etc/restic/_global.env.sh and /usr/local/etc/restic/default.env.sh looks with erasing personal / sensitive data of course!

from restic-automatic-backup-scheduler.

_global.env.sh

# shellcheck shell=sh

# Global envionment variables
# These variables are sourced FIRST, and any values inside of *.env.sh files for
# specific configurations will override if also defined there.


# Official instructions on how to setup the restic variables for Backblaze B2 can be found at
# https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2


# The restic repository encryption key
export RESTIC_PASSWORD_FILE="/etc/restic/pw.txt"
# The global restic exclude file
export RESTIC_BACKUP_EXCLUDE_FILE="/etc/restic/backup_exclude.txt"

# Backblaze B2 credentials keyID & applicationKey pair.
# Restic environment variables are documented at https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables
export B2_ACCOUNT_ID="<redacted>"   # *EDIT* fill with your keyID
export B2_ACCOUNT_KEY="<redacted>" # *EDIT* fill with your applicationKey

# How many network connections to set up to B2. Default is 5.
export B2_CONNECTIONS=10

# Optional extra space-separated args to restic-backup.
# This is empty here and profiles can override this after sourcing this file.
export RESTIC_BACKUP_EXTRA_ARGS=

# Verbosity level from 0-3. 0 means no --verbose.
# Override this value in a profile if needed.
export RESTIC_VERBOSITY_LEVEL=1

# (optional) Desktop notifications. See restic_backup.sh for details on how to set this up.
export RESTIC_NOTIFY_BACKUP_STATS=false
export RESTIC_BACKUP_NOTIFICATION_FILE=

default.env.sh

# shellcheck shell=sh

# This is the default profile. Fill it with your desired configuration.
# Additionally, you can create and use more profiles by copying this file.

# This file (and other .env.sh files) has two purposes:
# - being sourced by systemd timers to setup the backup before running restic_backup.sh
# - being sourced in a user's shell to work directly with restic commands e.g.
#  $ source /etc/restic/default.env.sh
#  $ restic snapshots
#  Thus you don't have to provide all the arguments like
#  $ restic --repo ... --password-file ...

# shellcheck source=etc/restic/_global.env.sh
. "/etc/restic/_global.env.sh"

# Envvars below will override those in _global.env.sh if present.

export RESTIC_REPOSITORY="b2:<redacted>"   # *EDIT* fill with your repo name

# What to backup. Colon-separated paths e.g. to different mountpoints "/home:/mnt/usb_disk".
# To backup only your home directory, set "/home/your-user"
export RESTIC_BACKUP_PATHS="/C/backup:/C/Code:/C/Git:<... redacted ...>:/S/Shows"  # *EDIT* fill conveniently with one or multiple paths

# Example below of how to dynamically add a path that is mounted e.g. external USB disk.
# restic does not fail if a specified path is not mounted, but it's nicer to only add if they are available.
#test -d /mnt/media && RESTIC_BACKUP_PATHS+=" /mnt/media"

# A tag to identify backup snapshots.
export RESTIC_BACKUP_TAG=systemd.timer

# Retention policy - How many backups to keep.
# See https://restic.readthedocs.io/en/stable/060_forget.html?highlight=month#removing-snapshots-according-to-a-policy
export RESTIC_RETENTION_HOURS=1
export RESTIC_RETENTION_DAYS=14
export RESTIC_RETENTION_WEEKS=16
export RESTIC_RETENTION_MONTHS=18
export RESTIC_RETENTION_YEARS=3

# Optional extra space-separated arguments to restic-backup.
# Example: Add two additional exclude files to the global one in RESTIC_PASSWORD_FILE.
#RESTIC_BACKUP_EXTRA_ARGS="--exclude-file /path/to/extra/exclude/file/a --exclude-file /path/to/extra/exclude/file/b"
# Example: exclude all directories that have a .git/ directory inside it.
#RESTIC_BACKUP_EXTRA_ARGS="--exclude-if-present .git"

from restic-automatic-backup-scheduler.

Currently I've just removed the line that adds the extra args from the backup script and that works for now. Potentially just combining the different arrays before calling the command would fix it? I can try that if you want.

from restic-automatic-backup-scheduler.

Thank you for sharing the config. When I set up and use your _global.env.sh and default.env.sh and run the backup script with tracing, this commands get run:

restic backup --verbose=1 --one-file-system --tag systemd.timer --option b2.connections=10 --exclude-file /etc/restic/backup_exclude.txt /C/backup /C/Code /C/Git <... redacted ...> /S/Shows

and I don't see that '' that you reported to see:

restic backup --verbose=1 --tag systemd.timer --option b2.connections=10 --exclude-file /etc/restic/backup_exclude.txt '' /C/backup /C/Code...

I ran this on a macOS system with bash version 5.2.15 from HomeBrew.

What bash version are you on?It might be an issue on your bash version or/and in combination with the Windows environment. I'm unable to support Windows at the moment as I have no access to such system anymore.

Really not a solution... but I wonder if it happens to work around the issue if you would swap the order of these 2 lines?

from restic-automatic-backup-scheduler.

I tried a few different things since I'm using the extra parameters for one backup now and not another, so I couldn't just remove them anymore. Combining didn't work and even only combining if there was something in the array didn't work, because I realized that the extra args array was reporting a size of 1 even when the original variable was empty. Here's what did work:

# Convert to array, and preserve spaces. See #111
backup_extra_args=( )
if [ ! -z "$RESTIC_BACKUP_EXTRA_ARGS" ]; then
  while IFS= read -r -d ''; do
    backup_extra_args+=( "$REPLY" )
  done < <(xargs printf '%s\0' <<<"$RESTIC_BACKUP_EXTRA_ARGS")
fi

So it seems that potentially just in windows, that method of building the array from the args variable doesn't work quite right.

from restic-automatic-backup-scheduler.

That fix looks good to me! Applied.

from restic-automatic-backup-scheduler.