Hello there, this is a nice script.
would like your advise to understand where to check or debug this issue...

Thank you

Instead of specifying include and exclude lists in the config file, allow use of include/exclude filelists.

(use of duplicity's option --include-globbing-filelist instead of --include/--exclude)

(ref: http://duplicity.nongnu.org/duplicity.1.html#sect10)

Is there a way to accomplish this using the script as it stands today?

Thanks for putting this together!

Is it possible to backup in chunks larger than 25mb? I am backing up to S3 a 10GB file. I would like to use a chunk larger than 25MB. Where do I set this option?

Thanks

Import of duplicity.backends.sshbackend Failed: No module named paramiko
Import of duplicity.backends.giobackend Failed: No module named gio

(Sorry this should be a question not an issue, but I dont see where I set the label)

Hello,

Could someone explain to me why I get the Permission denied message when I try to run the script? I am running it as root and as far as I can see, root has all the required access to run the script.

Thanks and regards,

Pradeep

Hi!
I'm getting the following error (?) in my log, using scp://

---------[ Destination Disk Use Information ]--------
unsupported on  s: type backend

Also changing to sftp (for example) leads to this error.

Implement possibility to restore a directory to a specific directory.
For example, restore /var/www to /tmp

Hello

I don't know if it is a problem in my config or a script problem.
When I'm listing the file :

~/duplicity-backup# ./duplicity-backup.sh -c duplicity-backup.conf --list-current-files | grep default.cfg
Tue may 20 14:59:05 2014 home/martin/.willie/default.cfg

The path of the file start without the slash so when I do a restore I have to follow this rule.

./duplicity-backup.sh -c duplicity-backup.conf --restore-file home/martin/.willie/default.cfg /home/martin/.willierestore/

Is it normal?

Thanks in advance.

Hi there, it seems that duplicity-backup can't find sendmail on the server it's set up on. At the end of each duplicity-backup log I see:

---------    END DUPLICITY-BACKUP SCRIPT    ---------

Email couldn't be sent. 'sendmail' not available.
Removing lock. Exit code: 0

The backup script runs via a Cron job under user root, similar to the example in the docs:

30 23 * * * cd /path/to/duplicity-backup; ./duplicity-backup.sh -c /path/to/duplicity-backup.conf -b

Checking through the duplicity-backup.sh script I see the following:

MAILCMD=$(which $MAIL)
      if [ ! -x "$MAILCMD" ]; then
          echo -e "Email couldn't be sent. '${MAIL}' not available." >> ${LOGFILE}
      else

$MAIL is set to sendmail in the conf file:

MAIL="sendmail"

If I run which sendmail on a shelll as any user, including root, I see:

/usr/sbin/sendmail

I cobbled together a script test.sh like so:

#!/bin/bash
MAIL="sendmail"
MAILCMD=$(which $MAIL)

echo "MAIL is '${MAIL}'"
echo "MAILCMD is '${MAILCMD}'"

if [ ! -x "$MAILCMD" ]; then
        echo "Email can't be sent. '${MAIL}' not available."
else
        echo "Email can be sent. '${MAIL}' available."
fi

which, when run as a user and root both output:

$ ./test.sh
MAIL is 'sendmail'
MAILCMD is '/usr/sbin/sendmail'
Email can be sent. 'sendmail' available.

The only difference I see between my script and duplicity-backup.sh is the shebang line #!/usr/bin/env bash
I switched my script to that and the output still shows that sendmail is available.

Fairly puzzling as to why this would happen. If I try mailx then an error appears:

/usr/bin/mailx: invalid option -- 'r'

Which shows to me that a 'which mailx' is returning the path to mailx (i.e it's executing the 'else' part of the MAILCMD test).

I'll keep digging but if you have any ideas on this it would help greatly.
This is running on a Ubuntu 14.04 server.

Thanks
James

I'm working on automating our duplicity solution using duplicity-backup, however the script is not non-interactive compliant at the moment. It would be great if there was an option to automatically answer yes to any questions and stop the script asking for user input.

Obviously needs some thought around how to do that for the encryption password - see my other issue!

Hi, is it possible to make signing optional? Also, what are the benefits of signing. Thanks!

Relavant line

could you please add support for amazon glacier? http://aws.amazon.com/glacier/

thank you, jose

Hi zertrin, I have been doing a fair bit of testing with duplicity-backup.

I've noticed that when a backup fails (eg. because of dropped connection - I'm backing up to s3), on the next run it will try to resume. On resume, it downloads the first volume of the backupset and tries to decrypt it.

At this point, I was getting the following error:

GPG error detail: Traceback (most recent call last):
File "/usr/bin/duplicity", line 1411, in
with_tempdir(main)
File "/usr/bin/duplicity", line 1404, in with_tempdir
fn()
File "/usr/bin/duplicity", line 1374, in main
full_backup(col_stats)
File "/usr/bin/duplicity", line 509, in full_backup
globals.backend)
File "/usr/bin/duplicity", line 346, in write_multivol
validate_encryption_settings(globals.restart.last_backup, mf)
File "/usr/bin/duplicity", line 334, in validate_encryption_settings
fileobj.close()
File "/usr/lib64/python2.6/site-packages/duplicity/dup_temp.py", line 225, in close
assert not self.fileobj.close()
File "/usr/lib64/python2.6/site-packages/duplicity/gpg.py", line 251, in close
self.gpg_failed()
File "/usr/lib64/python2.6/site-packages/duplicity/gpg.py", line 218, in gpg_failed
raise GPGError, msg
GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: encrypted with 2048-bit RSA key, ID 3C6E2351, created 2013-05-19 "Duplicity Backup Encryption Key for profiaa22 (Encryption key for Duplicity Backup for profiaa22) <securityteam@***>"
gpg: public key decryption failed: Bad passphrase
gpg: decryption failed: No secret key
can't connect to `/root/.gnupg/S.gpg-agent': No such file or directory ===== End GnuPG log =====

The key bit of this was "gpg: public key decryption failed: Bad passphrase". Since it is using the Encryption key to decrypt, and this key is not stored in the duplicity-backup config file, it doesn't seem surprising that it can't decrypt.

I've worked around this by 1). creating an ENC_PASSPHRASE variable in my duplicity-backup.conf file, eg:

PASSPHRASE="12345"
ENC_PASSPHRASE="1234"

and 2). modifying duplicity-backp.sh to export this as follows:

SIGN_PASSPHRASE=$PASSPHRASE
PASSPHRASE=$ENC_PASSPHRASE
export AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY
export PASSPHRASE
export SIGN_PASSPHRASE

Clearly before the script was exporting the Signature passphrase as both PASSPHRASE & SIGN_PASSPHRASE - I'm not sure why this would be? Can you see any problems with what I've done? If not, then it would seem sensible to roll this into the production version...

Regards, Ben

Hi,

Great script, thanks!

It would be nice to have the ability to have backups going to two different locations (probably different backends) in only one .conf file. I mean, backup to a local NAS and to Amazon S3 for local and offsite backup copies of same data. This is very easy to accomplish with two (or more) .conf files for the same data set, however it is more difficult to maintain and still need to call script twice for the backup. Would be interesting to manage from just one place (.conf) even do those two backup could be executed one after another sequentially by the script.

Please forgive if this is already possible but my lack of sleep avoid me from figuring out how :)

Thanks,
Benito

When using "mail" to send it sends to 4 recipients, $FROM $TO and -f@<my.domain.name> and --@<my.domain.name>

Somehow it's misunderstanding the -f and -- flags. I'm using mailx for now.

Hi @zertrin,

I don't know if it is a bug or an OS depending thing, but when running the script without s3cmd installed I get:

which: no s3cmd in (/usr/lib64/qt-3.3/bin:/usr/NX/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/local/sbin:/usr/sbin:/home/jan/.local/bin:/home/jan/bin)

in the very first line of the output (the script is running anyway).
Even when the variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are commented out.

I didn't find a quick solution to suppress the warning :(

I used to have a problem with backing up /home because root didn't have permission to read ~/.gvfs, even though */.gvfs was in the excluded list. While this appears fixed in the (most recent) version I now use, I am facing the same problem again with ~/xrdp_client, again in the excluded list. As far as I can see, the backup completes regardless, but does still generate the log entry "Error accessing possibly locked file /home/user/xrdp_client".

If this isn't a simple fix, it would be useful to know which commit fixed the earlier problem with ignoring excluded files, so that I can try to understand what is happening.

When using MAIL="mail" on a Linux system (tested with Debian wheezy) the command parameters should be the same as for FreeBSD|Darwin|DragonFly|OpenBSD.
For instance, -f ${EMAIL_FROM} is not an option in Debian wheezy and results in an additional email send to -f@domain

Hello,

First, thanks for this excelent script. This works fine in my local network, but I can't send my backups to amazon s3.

I have installed and configured s3cm and my credentials are correct.

The error messages, after command, are:

Traceback (most recent call last):
File "/usr/bin/duplicity", line 1434, in
with_tempdir(main)
File "/usr/bin/duplicity", line 1428, in with_tempdir
fn()
File "/usr/bin/duplicity", line 1277, in main
do_backup(action)
File "/usr/bin/duplicity", line 1310, in do_backup
globals.archive_dir).set_values()
File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 696, in set_values
self.get_backup_chains(partials + backend_filename_list)
File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 819, in get_backup_chains
map(add_to_sets, filename_list)
File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 813, in add_to_sets
log.Debug(_("File %s is not part of a known set; creating new set") % (filename,))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4: ordinal not in range(128)

Could you help me?

Thanks
Marcelo

bae default it is /var/log/duplicity-backup/backup.lock. Is it possible to change that? I'd like to run duplicity simultaneously.

Using the lasted revision on Debian wheezy I get the following error message:

## ./duplicity-backup.sh -c config
USAGE:
    duplicity-backup.sh [options]

  Options:
    -c, --config CONFIG_FILE   specify the config file to use

    -b, --backup               runs an incremental backup
    -f, --full                 forces a full backup
    -v, --verify               verifies the backup
    -l, --list-current-files   lists the files currently backed up in the archive
    -s, --collection-status    show all the backup sets in the archive

        --restore [PATH]       restores the entire backup to [path]
        --restore-file [FILE_TO_RESTORE] [DESTINATION]
                               restore a specific file
        --restore-dir [DIR_TO_RESTORE] [DESTINATION]
                               restore a specific directory

    -t, --time TIME            specify the time from which to restore or list files
                               (see duplicity man page for the format)

    --backup-script            automatically backup the script and secret key(s) to
                               the current working directory

    -n, --dry-run              perform a trial run with no changes made
    -d, --debug                echo duplicity commands to logfile

  CURRENT SCRIPT VARIABLES:
  ========================
    DEST (backup destination)       = XXXXX
    INCLIST (directories included)  = XXXXX
    EXCLIST (directories excluded)  = 
    ROOT (root directory of backup) = /
    LOGFILE (log file path)         = /root/new_backup/duplicity-2015-01-20_13-26.txt


/usr/bin/mailx: invalid option -- 'r'
usage: mailx [-dEIinv] [-a header] [-b bcc-addr] [-c cc-addr] [-s subject] to-addr ...
       mailx [-dEIiNnv] -f [name]
       mailx [-dEIiNnv] [-u user]

When I change MAIL to "mail" everything works fine.

This would drop the "cost of ownership" significantly.

Glacier = $0.010 per GB/month
$0.055 per 1,000 requests

S3 = $0.140 per GB/month
$0.011 per 1,000 requests

http://aws.amazon.com/glacier/

Hello

I know it's not related to duplicity-backup but more to duplicity itself but when I try to do a backup I always get stuck
Restarting after volume 10, file opt/vc/src/hello_pi/hello_video/test.h264, block 13

If someone have an idea?

Thanks in advance

Im running this script on freebsd, however, in freebsd du doesn't support --exclude-from.
The script gives an error:
du: illegal option -- -
usage: du [-Aclnx] [-H | -L | -P] [-h | -k | -m ] [-a | -s | -d depth] [-B blocksize] [-I mask] [-t threshold] [file ...]

hoping for a fix.

I have two .conf files set up to run via cron writing to two log files. The first one is a daily backup which write all output to the log and emails me via ssmtp. I know email is working.

When I run the second job which uses a copy of the daily backup conf file just modified to run -s instead of -b and writing to a different log file I get no proper output int the log:

-------- START DUPLICITY-BACKUP SCRIPT --------

Attempting to acquire lock /var/log/backup.lock
successfully acquired lock.
-------- END --------

-------- END DUPLICITY-BACKUP SCRIPT --------

I'll fork the code and see if I can fix.

/usr/local/bin/duplicity-backup.sh -c /etc/duplicity-backup.conf -f
/usr/local/bin/duplicity-backup.sh: eval: line 490: unexpected EOF while looking for matching `''
/usr/local/bin/duplicity-backup.sh: eval: line 491: syntax error: unexpected end of file

From line 578 - just needs to be piped to tee again:

echo "Restoring now ..."
#use INCLUDE variable without create another one
INCLUDE="--file-to-restore ${FILE_TO_RESTORE}"
duplicity_backup  | tee -a ${LOGFILE}
;;

Also - is there an option to choose from what set within a backup to restore from ? What if I wanted to restore a file from last month ? Cheers

--s3-european-buckets should be listed in the static options, in the sample config file, otherwise backup will fail

My backups have been failing silently (returning a code 0, which should mean success). My monitoring scripts have failed to alert me (because of the code 0), and I didn't get any email (probably because I don't have mailx, mail, or ssmtp). In the logs it says "Current Remote Backup File Size: s3cmd not installed.". I have s3cmd installed, so that's not the issue.

I see three issues here:

1. Backups should never fail with a code 0.
2. The "s3cmd not installed" message is wrong, or at least not very helpful in debugging the real issue.
3. The default mail command setting should be something that is typically installed by default (e.g. sendmail).

I get this error when running the script

/usr/local/bin/duplicity-backup.sh: line 200: unexpected token `newline', conditional binary operator expected
/usr/local/bin/duplicity-backup.sh: line 200: syntax error near `\'
/usr/local/bin/duplicity-backup.sh: line 200: `         ${GPG_SIGN_KEY} = "foobar_gpg_key" || \         '

There's extra whitespace after the line.

On Centos 5.5 i686, even with the mailx package installed from yum, the script still won't send via mailx (which isn't in the PATH, or even an installed file (?)) I'm not a big Centos user, so I fixed the script so that it will also send with the stock mail command, this requires two patches:

diff --git a/duplicity-backup.conf.example b/duplicity-backup.conf.example
index 89861be..5037292 100644
--- a/duplicity-backup.conf.example
+++ b/duplicity-backup.conf.example
@@ -160,6 +160,7 @@ EMAIL_SUBJECT=
 
 # command to use to send mail
 MAIL="mailx"
+#MAIL="mail"
 #MAIL="ssmtp"
 
 # TROUBLESHOOTING: If you are having any problems running this script it is



diff --git a/duplicity-backup.sh b/duplicity-backup.sh
index c3fc583..68c5f35 100755
--- a/duplicity-backup.sh
+++ b/duplicity-backup.sh
@@ -241,6 +241,8 @@ email_logfile()
           elif [ "$MAIL" = "mailx" ]; then
             EMAIL_FROM=${EMAIL_FROM:+"-r ${EMAIL_FROM}"}
             cat ${LOGFILE} | ${MAILCMD} -s """${EMAIL_SUBJECT}""" $EMAIL_FROM ${EMAIL_TO}
+          elif [ "$MAIL" = "mail" ]; then
+            cat ${LOGFILE} | ${MAILCMD} -s """${EMAIL_SUBJECT}""" $EMAIL_FROM ${EMAIL_TO} -- -f ${EMAIL_FROM}
           fi
           echo -e "Email alert sent to ${EMAIL_TO} using ${MAIL}" >> ${LOGFILE}
       fi

I need to use the --allow-source-mismatch option of duplicity, how can I do that with your script?

I'll get this error when running the script.

Import of duplicity.backends.giobackend Failed: No module named gio

I'll use

• duplicity 0.6.18
• s3cmd version 1.0.0
• gpg (GnuPG) 1.4.11
• Ubuntu 12.04.4 LTS (GNU/Linux 3.2.0-60-virtual x86_64)

Running the "naked" duplicity command throws

Import of duplicity.backends.giobackend Failed: No module named gio
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1414, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1407, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1258, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1007, in ProcessCommandLine
    backup, local_pathname = set_backend(args[0], args[1])
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 900, in set_backend
    globals.backend = backend.get_backend(bend)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 158, in get_backend
    return _backends[pu.scheme](pu)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 77, in __init__
    self.resetConnection()
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 144, in resetConnection
    self.conn = S3Connection(is_secure=(not globals.s3_unencrypted_connection))
  File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 154, in __init__
    suppress_consec_slashes=suppress_consec_slashes)
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 486, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "/usr/lib/python2.7/dist-packages/boto/auth.py", line 406, in get_auth_handler
    'Check your credentials' % (len(names), str(names)))
NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials

What am I missing?

Here my configuration file:

#!/bin/bash

# AMAZON S3 INFORMATION
# Comment out these lines if you're not using Amazon S3
AWS_ACCESS_KEY_ID="asdf"
AWS_SECRET_ACCESS_KEY="asdf"

# S3CMD INFORMATION
# Most people don't need this, but in some cases
# you may want to specify a custom configuration file
# to pass to s3cmd. If so, set the S3CMD_CONF_FILE variable
# to the full path of this custom config file.
# Per default s3cmd uses ${HOME}/.s3cfg
# S3CMD_CONF_FILE='/path/to/your/s3cmd/conf/file'

ENCRYPTION='yes'

PASSPHRASE="asdf"

GPG_ENC_KEY="asdf"
GPG_SIGN_KEY="asdf"

ROOT="/var"

# BACKUP DESTINATION INFORMATION
# The s3+http scheme uses the default aws s3 hostname.
# Use s3://host/bucket/[backup-folder/] if you want to specify the host name.
# If using the s3://... scheme and you have s3cmd installed, be sure to change
# 's3.amazonaws.com' to the appropriate host in your .s3cfg file so that the
# remote file size check will work.
#DEST="s3://host/backup-bucket/backup-folder/"
DEST="s3+http://asdf/asdf/"

INCLIST=(  "/var/backup/db" \
           "/var/www/wordpress/wp-content" \
        )

STATIC_OPTIONS="--full-if-older-than 14D --s3-use-new-style  --s3-european-buckets"

CLEAN_UP_TYPE="remove-all-but-n-full"
CLEAN_UP_VARIABLE="4"

LOGDIR="/var/log/duplicity-backup/"
LOG_FILE="duplicity-`date +%Y-%m-%d_%H-%M`.txt"
LOG_FILE_OWNER="root:ubuntu"
VERBOSITY="-v3"

EMAIL_TO=
EMAIL_FROM=
EMAIL_SUBJECT=

MAIL="mailx"     # default command for Linux mail

Slight change needed here on testing $FRIEND_NAME for the size check;

if $FRIENDLY_TYPE_NAME ; then

.. to ..

if [[ $FRIENDLY_TYPE_NAME ]] ; then

Hi there, I updated my copy of duplicity-backup.sh to the current version last week. Since then, duplicity has not been running - I went to check it today manually, and when I call duplicity-backup.sh (with or without parameters), I receive the error "No such file or directory". I do not know which file or directory it is referring to which is missing. I always supply the full path to the config file when I run duplicity via cron using the -c switch, I have checked the cron line and it is supplying the correct path and the file exists. However, the same error is returned whether or not -c is specified.

Could you update the script to inform the user which file or directory it is looking for which is missing?

AHA, Ben

Hello,

in case of a full backup for a virtual machine, how can I force a full restoration in / path ?

Thanks for your scripts they are very good for simply use duplicity.

Sincerely
Guidtz

This would allow to backup to different S3 buckets (with different credentials) with the same system user, which would be very useful to e.g. cron jobs and the like.

First, great job on the script it's wonderful and saved me heaps of time.

Not really an issue but didnt warrant a merge request. I noticed if you try to include multiple email recipients in the EMAIL_TO="" as so:

EMAIL_TO="[email protected] [email protected]"

The script returns the error "unary operator expected"

If you make a quick change to your bash syntax in duplicity-backup.sh at line 249 this is no longer a problem.

[$EMAIL_TO] becomes ["$EMAIL_TO"]

Hopefully that helps out someone else, Thanks for sharing the great work!

Hello there,

This is not an issue. Just a question but I'm not sure how to mark this as a question.

I have installed your script and everything is running so smoothly now so the next step would be automating it. I entered the following two lines to the crontab file inside /etc folder but nothing seems to be happening.

0 4 * * * 1-6 root /root/duplicity-backup.sh -c /etc/duplicity-backup.conf --backup
0 4 * * * 7 root /root/duplicity-backup.sh -c /etc/duplicity-backup.conf --full

Could you explain what I'm doing wrong in here? Is it because the crontab file inside the /etc folder can't run this as root? If it is the case, how can I overcome that issue?

Thanks and regards,

Pradeep

When running duplicity.sh -b, I know get a 404 error saying:

ClientLoginFailed: Server responded to ClientLogin request: 404, https://developers.google.com/accounts/docs/AuthForInstalledApps

I believe it may be because Google OAuth1 is now officially deprecated? I'm not quite sure. Some new form of authentication may be required, or it's totally on my end. Here's the whole output I get when I Run duplicity:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1523, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1517, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1355, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python2.7/site-packages/duplicity/commandline.py", line 1101, in ProcessCommandLine
    backup, local_pathname = set_backend(args[0], args[1])
  File "/usr/lib/python2.7/site-packages/duplicity/commandline.py", line 992, in set_backend
    globals.backend = backend.get_backend(bend)
  File "/usr/lib/python2.7/site-packages/duplicity/backend.py", line 223, in get_backend
    obj = get_backend_object(url_string)
  File "/usr/lib/python2.7/site-packages/duplicity/backend.py", line 209, in get_backend_object
    return factory(pu)
  File "/usr/lib/python2.7/site-packages/duplicity/backends/gdocsbackend.py", line 54, in __init__
    self._authorize(parsed_url.username + '@' + parsed_url.hostname, self.get_password())
  File "/usr/lib/python2.7/site-packages/duplicity/backends/gdocsbackend.py", line 134, in _authorize
    captcha_response=captcha_response)
  File "/usr/lib/python2.7/site-packages/gdata/client.py", line 441, in client_login
    captcha_token=captcha_token, captcha_response=captcha_response)  
  File "/usr/lib/python2.7/site-packages/gdata/client.py", line 373, in request_client_login_token
    response, ClientLoginFailed, response_body)
ClientLoginFailed: Server responded to ClientLogin request: 404,        https://developers.google.com/accounts/docs/AuthForInstalledApps


Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1523, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1517, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1355, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python2.7/site-packages/duplicity/commandline.py", line 1093, in  ProcessCommandLine
    globals.backend = backend.get_backend(args[0])
  File "/usr/lib/python2.7/site-packages/duplicity/backend.py", line 223, in get_backend
    obj = get_backend_object(url_string)
  File "/usr/lib/python2.7/site-packages/duplicity/backend.py", line 209, in get_backend_object
    return factory(pu)
  File "/usr/lib/python2.7/site-packages/duplicity/backends/gdocsbackend.py", line 54, in __init__
    self._authorize(parsed_url.username + '@' + parsed_url.hostname, self.get_password())
  File "/usr/lib/python2.7/site-packages/duplicity/backends/gdocsbackend.py", line 134, in _authorize
    captcha_response=captcha_response)
  File "/usr/lib/python2.7/site-packages/gdata/client.py", line 441, in client_login
    captcha_token=captcha_token, captcha_response=captcha_response)
  File "/usr/lib/python2.7/site-packages/gdata/client.py", line 373, in request_client_login_token
    response, ClientLoginFailed, response_body)
ClientLoginFailed: Server responded to ClientLogin request: 404,     https://developers.google.com/accounts/docs/AuthForInstalledApps

Thank you!

Thanks for this and to Damon. I have it working nicely on my VPS at Linode.

I have a quick question.

What happens if I add more to INCLIST now that I have it running? I'm using the default --full-if-older-than 14D option. The new directories won't be included in the last full backup. Do I need to force a full backup after adding to INCLIST or does it somehow take care of this?

I have the ROOT set up as "/"

and the INCLIST set up as:

INCLIST=( "/etc/nginx"
"/etc/php5"
"/home/ilium007"
"/apps"
)

When the backup runs I see these errors:

Last full backup date: Sun Aug 18 15:26:54 2013
Error accessing possibly locked file /etc/.pwd.lock
Error accessing possibly locked file /etc/at.deny
Error accessing possibly locked file /etc/fuse.conf
Error accessing possibly locked file /etc/group-
Error accessing possibly locked file /etc/gshadow
Error accessing possibly locked file /etc/gshadow-
Error accessing possibly locked file /etc/passwd-
Error accessing possibly locked file /etc/shadow
Error accessing possibly locked file /etc/shadow-
Error accessing possibly locked file /etc/sudoers
Error accessing possibly locked file /lost+found
Error accessing possibly locked file /root
Error accessing possibly locked file /swapfile

I have asked for none of these files to be backed up yet there are 13 errors.

Hi!

Nice script but it would be cool if you could support COMPRESSION_LEVEL.
like in this here: https://github.com/vitobotta/admin-scripts/wiki/duplicity.sh

Thanks!

Hi @zertrin,

just a little idea:

How about an option to create separate backup sets for each folder defined in "INCLIST"?

Before I used this script, I created a little script by myself which created seperate backups sets for each folder I wanted to backup at a "global" destination.

For example:

Backup folders:
/home/foo
/home/bar
/home/foobar

Backup destination:
scp://user@host/duplicity_backup_folder/home_foo/
scp://user@host/duplicity_backup_folder/home_bar/
scp://user@host/duplicity_backup_folder/home_foobar/

So the backups are independent.
IMHO the advantages of using separate backup sets are:

• You don't have to download the complete duplicity archives for folders you don't want to restore (just think of Amazons traffic fees for outgoing traffic if you are using S3 as destination)
• If a backup set, for whatever reasons, gets corrupted then just one folder is affected - not the complete backup

What do you think?

Greetz
puredoze

I'm baffled by what I'm doing wrong, as when I run for example

./duplicity-backup.sh --config duplicity-backup.conf --backup

I see the log output like:

-------- START DUPLICITY-BACKUP SCRIPT --------

Attempting to acquire lock /home/keithzg/workspace/test_offsite/logs/backup.lock
successfully acquired lock.
/usr/bin/duplicity -v3 --full-if-older-than 14D --s3-use-new-style --no-encryption --exclude /home/keithzg/workspace/test_offsite/logs --include=/home/keithzg/workspace/test_offsite --exclude=** /home/keithzg/workspace/ s3+http://keithzg_test1/
-----------[ Duplicity Cleanup ]-----------
/usr/bin/duplicity remove-all-but-n-full 2 --full-if-older-than 14D --s3-use-new-style --force --no-encryption s3+http://keithzg_test1/

---------[ Source File Size Information ]---------
110M /home/keithzg/workspace/test_offsite

------[ Destination File Size Information ]------
Current Remote Backup File Size: 388M

-------- END DUPLICITY-BACKUP SCRIPT --------

Removing lock. Exit code: 0

If I try and run the commands listed in the log file manually, it works fine. So why is it assuming I want to just do a test run when I'm running the script itself?

Mail notification works great but it could be useful to receive such alerts only when a backup process fails.

First I must say, great script. I had to look through the entire internet and was almost ready to pay someone to create sync to S3 for me.

I have gotten everything to backup to S3 which is nice.

But I am having trouble restoring. I have tried multiple commands. First I want to test to restore one file.

Here is an example:

./duplicity-backup.sh --restore-file /home/httpd/backups/dbdump_2012-04-03_13-00-52.tar.bz2

I get this:
Restoring now ...
../duplicity-backup.sh: eval: line 366: unexpected EOF while looking for matching `''
../duplicity-backup.sh: eval: line 367: syntax error: unexpected end of file

I try this:

./duplicity-backup.sh --restore-file dbdump_2012-04-03_13-00-52.tar.bz2

I get this:

Restoring now ...
Traceback (most recent call last):
File "/usr/bin/duplicity", line 1251, in
with_tempdir(main)
File "/usr/bin/duplicity", line 1244, in with_tempdir
fn()
File "/usr/bin/duplicity", line 1198, in main
restore(col_stats)
File "/usr/bin/duplicity", line 538, in restore
restore_get_patched_rop_iter(col_stats)):
File "/usr/bin/duplicity", line 560, in restore_get_patched_rop_iter
backup_chain = col_stats.get_backup_chain_at_time(time)
File "/usr/lib/python2.6/dist-packages/duplicity/collections.py", line 934, in get_backup_chain_at_time
raise CollectionsError("No backup chains found")
CollectionsError: No backup chains found

I have not tested the full restore yet, since the backup is 5GB.

I don't run as root, if that is required?

Thanks.

If I run ./duplicity-backup.sh with no options set at CLI, only STATIC_OPTIONS="--full-if-older-than 7D" in .conf, script exits. If I set --full-if-older-than 7D at CLI, script exits.

It looks like the script isn't passing $STATIC_OPTIONS to duplicity, that the variable isn't actually set in the script.

It is more elegant to be able to set static options in .conf, allowing use of only one cron entry to handle incremental and full backups, so this would be an excellent feature to complete.

Great work on the script. Thanks.

Ehancement idea from a comment on Damon Timm's blog:

June 10, 2012 at 11:38 am
Lou says:
Is there anyway specify the duplicity time variable in a restore a specific backup?
Something like the man page give here:

duplicity [restore] [options] [--file-to-restore ] [--time time] source_url target_directory