mkdir -m 0700 folder is atomic (with -p it is not).

Thus one can use it to make singleton instances of processes, eg:-

singleton_instance_lock()
{
local temporary_folder_path=...
local program_name=...
	single_instance_lock_folder_path=''

	local potential_single_instance_lock_folder_path="$temporary_folder_path"/"$program_name"
	local lock_holder_pid_file_path="$potential_single_instance_lock_folder_path"/lock_holder.pid
	local loop_count=0
	
	local exit_code
	while true
	do
		set +e
			mkdir -m 0700 "$potential_single_instance_lock_folder_path" 1>/dev/null 2>/dev/null
			exit_code=$?
		set -e
		
		if [ $exit_code -eq 0 ]; then
			printf '%s\n' $$ >"$lock_holder_pid_file_path"
			single_instance_lock_folder_path="$potential_single_instance_lock_folder_path"
			break
		fi
		
		local loop_division=$((loop_count % 5))
		if [ $loop_division -eq 0 ]; then
			local lock_holder_pid=''
			set +e
# Uses cat rather than $(<X) as the later echos failures to stderr irrespective.
				lock_holder_pid="$(cat "$lock_holder_pid_file_path" 2>/dev/null)"
			set -e
			
			if [ -z "$lock_holder_pid" ]; then
				lock_holder_pid='(unknown)'
			fi
			
			printf '%s:%s\n' "$_program_name" "Still waiting for single instance lock $potential_single_instance_lock_folder_path held by process $lock_holder_pid"
		fi
		loop_count=$((loop_count + 1))
		
		# Non-portable fractional sleep
		set +e
			sleep 0.1 1>/dev/null 2>/dev/null
			exit_code=$?
		set -e
		
		if [ $exit_code -ne 0 ]; then
			sleep 1 1>/dev/null 2>/dev/null
		fi
		
	done
	
	remove_single_instance_lock_folder_path()
	{
		if [ -n "$single_instance_lock_folder_path" ]; then
			rm -rf "$single_instance_lock_folder_path"
			single_instance_lock_folder_path=''
		fi
	}
	
	trap remove_single_instance_lock_folder_path EXIT
}

Dear Developer,
I have a question. Does this script copy every time all the data to backup or only change files that have been changed during a previous backup and a current backup?

Cheers
Luigi

I've followed your example in the thread at https://forum.rclone.org/t/ive-made-a-nice-backup-script-tutorial-for-rclone/4999/9

Here is the job under discussion:

source="/home/joti/test_rclone_data/music"
dest="${test_remote}:"
new="music"
move_old_files_to="dated_directory"
#options="--dry-run"
monitoring_URL=""

This creates the directory music under test_remote i.e. test_remote:/music/files...

However, any deletions/modifications of these files don't end up under test_remote:/archive/music/VERSION

instead they end up under test_remote:/archive/_VERSION_

This is problematic because then when I backup with the following script:

source="/home/joti/test_rclone_data/docs"
dest="${test_remote}:"
new="docs"
move_old_files_to="dated_directory"
#options="--dry-run"
monitoring_URL=""

All my versions of docs are also under test_remote:/archive/_VERSION_ and I have no idea to tell whether a file came from /home/joti/test_rclone_data/music or /home/joti/test_rclone_data/docs

How do I use this script to push to a remote that already contains the first version of the files but keep archived versions in their own directories?

The script checks $source with ls and therefore cannot perform any backup from a cloud provider.

Hi, thanks for this. :)

I'm just testing out rclone and this script helps to make it more of a backup tool rather than just a sync tool.

A couple of suggestions that popped up just now:

1. Add an option to write the warnings to the journalctl. At least I have this on a VM, and not on my desktop. So the notify pop-up part does not work.

This command should work:

printf "$message" | systemd-cat -t RCLONE_JOBBER

2. Log rotation in the script? Not sure if that is a good idea, or just to set up a log rotation manually. At least the initial backup will be huge and fill up the log file.

I also set up this to use the conf and filter file from the same directory as the script. That makes the most sense to me. All in one place if I i.e are going to put it on another VM, etc.

To make it a bit easier for myself I made a symlink of the config file to the default rclone directory. An alternative method could be to set up an alias in .bashrc/.zshrc/etc.

rclone_jobber=<path-to-backup-script-directory> #path to rclone_jobber directory
remote=<name of remote> #added this

source="<path-to-source>"
dest="${remote}:"
move_old_files_to="dated_directory"
options="--config ${rclone_jobber}/rclone.conf --filter-from=${rclone_jobber}/filter_rules"
monitoring_URL=""

${rclone_jobber}/rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL"

Thanks!

Hi, in lights of the recent Google Issue (forum) (github issue), I'm trying to add --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36", but I can't get it to work because of the spaces and ( ).

I've temporarily solved it by just using a random string, but i would prefer using an actual user agent string from Chrome.

Any suggestion on how to add that?

Please consider implementing Healthchecks' script run time measurement. More info here.

Usually, its curl -fsS --retry 3 https://hc-ping.com/xxxxx/start.

Healthchecks also supports logs via curl POST (more info here).

Would love it if you could implement these.

No matter whether you pass double quoted or single quoted paths to rclone_jobber, you get an error of the following type:

Command lsf needs 1 arguments maximum: you provided 2 non flag arguments: ["'/mnt/user/Google" "Drive/'"]
ERROR: backup_Google aborted because source is empty.

May I possibly get a nudge in the right direction? I'm having problems getting the script to run using a remote rclone connection.

I have the following elements in the script:

source="/home/x/storage"            #the directory to back up (without a trailing slash)
dest="${dropbox_crypt_WS1bkp}:"              #the directory to back up to (without a trailing slash or "last_snapshot") destination=$dest/last_snapshot

With the objective to backup /home/x/storage/* (all from this point and downwards) to Dropbox location (config made in rclone) dropbox_crypt_WS1bkp. It passes the following command

Back up in progress 2021-12-08_11:52:12 WS1bkp_storage
rclone sync /home/x/storage :/last_snapshot --backup-dir=:/archive/2021/2021-12-08_11:52:12 --log-file=/home/x/scripts/rclone_jobber/WS1backup.log --log-level=INFO
ERROR: WS1bkp_storage failed.  rclone exit_code=1

but I am not getting sense into my head. I've tried adding the source without the dollar sign, without the curly brackets as well (different parts of the script/docs give the impression that they are left out, adding to my confusion).

Any guidance and pointers are welcome.

Hello,

I noticed that the script fails if a monitoring_URL is provided empty or missing.
I don't use that kind of feature, and I don't think I'm alone

I just added an if test to check if $monitoring_URL is not empty before the wget.
Should work better that way

Commit with fix available here : https://github.com/lordslair/rclone_jobber/commit/7ec95f84095dff08015bac29192a1c874655145c

All the best,

I've realised that I sometimes have multiple instances of rclone running parallel, the check if the script is already running does not work on my Open Media Vault server (Linux nas 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux).

Does anyone have a modified version of the script using flock or similar to prevent running more than once at a time?

Or, as the past update to this repo is 4 years ago, what alternative script have you moved on to?

Thanks!

If I run the script and say Y or y nothing happens:

root@fs  🖿  ~ ▓▒░ rclone_jobber/examples/job_restore_last_snapshot.sh                                             ░▒▓ ✘  11.81  3%  20:30 🕣  
rclone copy remote:del/last_snapshot ~/test/last_snapshot
>>>>>>>>>>>>>>> Run the above rclone command? (y) <<<<<<<<<<<<<<<<< 
y

 root@fs  🖿  ~ ▓▒░ ll                                                                                         ░▒▓ ✔  1.5s  11.82  3%  20:30 🕣  
total 0
lrwxrwxrwx 1 root root  21 Apr 29 15:57 mdcmd -> /usr/local/sbin/mdcmd
drwxrwxrwx 5 root root 180 Apr 29 19:48 rclone_jobber
drwxrwxrwx 5 root root 120 Apr 29 19:58 rclone_test_data

Conversely, if I just run the rclone command it suggests, it works fine:

 root@fs  🖿  ~ ▓▒░ rclone copy remote:del/last_snapshot ~/test/last_snapshot                               ░▒▓ ✔  11.82  3%  20:30 🕣  

 root@fs  🖿  ~ ▓▒░ ll                                                                                         ░▒▓ ✔  8.9s  11.71  3%  20:31 🕣  
total 0
lrwxrwxrwx 1 root root  21 Apr 29 15:57 mdcmd -> /usr/local/sbin/mdcmd
drwxrwxrwx 5 root root 180 Apr 29 19:48 rclone_jobber
drwxrwxrwx 5 root root 120 Apr 29 19:58 rclone_test_data
drwxrwxrwx 3 root root  60 Apr 29 20:30 test

It would be nice to make this an option so the user does not need to edit the master script with every update:

timestamp="$(date +%F_%H%M%S)" #time w/o colons if thumb drive is FAT format, which does not allow colons in file name

Hi, is it possible to filter out files from the "move/archive" function?
Or any ideas on how to do that?

I have a few files that changes almost every day, that I want the current version backed up, but I don't want it to copy the old to the "archive", since most days that is the only few files that has changed.

I'm having a hard time having the job_backup_to_remote script to recognize a space in the path of my directory for my source. Here's the modified script:

rclone_jobber="/Users/Olive/Documents/GitHub/rclone_jobber" #path to rclone_jobber directory

source="/Volumes/Oliver/User Library" <----- issue here
dest="pcloud:Production/Ableton"
move_old_files_to="dated_directory"
options="--stats=10s"
monitoring_URL=""

/Users/Olive/Documents/GitHub/rclone_jobber/rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL"

Here's the error message I get:

~ '/Users/Olive/Documents/GitHub/rclone_jobber/olive_backup/ableton_remote.sh' 07:58:52
ls: /Volumes/Oliver/User: No such file or directory
/Users/Olive/Documents/GitHub/rclone_jobber/rclone_jobber.sh: line 86: pidof: command not found
Back up in progress 2018-08-02_08:02:31 ableton_remote.sh

(I renamed job_backup_to_remote to ableton_remote)

Thanks!

Running something like:
“pidof -o $PPID -x “$job_name””

causes:
pidof: illegal option — o

rclone_jobber.sh version 1.5.1

pidof -v
pidof version 0.1.4
Copyright 2003 - 2004 Night Productions

brew -v
Homebrew 2.0.6
Homebrew/homebrew-core (git revision f321f; last commit 2019-03-26)

uname -a
Darwin localhost 18.2.0 Darwin Kernel Version 18.2.0: Thu Dec 20 20:46:53 PST 2018; root:xnu-4903.241.1~1/RELEASE_X86_64 x86_64

Are there any alternatives?