Git Product home page Git Product logo

home-server-backup's Introduction

Multi-purpose Backup script dedicated for Home Servers

โ— Work-in-progress ๐Ÿ”จ
Script under developement, designed for homelab application.
Feel free to test, modify, report bugs & propose features.
Have fun ๐Ÿš€

Table of content

Features

  • Separate parameters.sh file for settings & sensitive data
  • Core functionalities:
    • Multiple directory local backup:
      • Docker Volumes | With stopping containers
      • Docker Bind Mounts | With /or without stopping containers
      • Server Home Directory | Separated by sub-directiories
    • Cloud backup using rclone
    • Delete old backup files (both locally & in the Cloud)
    • Archive locally backup every 10 days
  • Additional functionalities:
    • Gotify notifications
    • Netdata notifications silencer during backup
    • Separate 'instant' type to test choosen features
    • 'cleaner' mode to quickly purge needless 'instant' backups
    • Built-in conditional messages based on exit codes

Prequisitions and Dependencies

  • Linux environment (to execute bash script)
  • For required dependencies for each feature see How to use it

Files descriptions

File Description
homeServerBackup.sh core of the application
parameters-sample.sh external file with settings & sensitive data
.gitignore git configuration file
.shellcheckrc [ShellCheck][shellCheck] configuration file
README.md documentation file
LICENSE.md open source license details
CODE_OF_CONDUCT.md community code of conduct
CONTRIBUTING.md contribution guidelines
SECURITY.md security guidelines
.github/ISSUE_TEMPLATE templates for bug reports, feature requests & other issues

How to use it

  1. Clone GitHub repository to you server
  2. Make core script executable
    $ chmod u+x homeServerBackup.sh
  3. Configurate parameters
    (!) Before changing any settings / parameters copy parameters-sample.sh file and remove -sample part of the name.
    $ cp parameters-sample.sh parameters.sh
    Complete configuration of script acc. Configuration
  4. Execute script
    $ sudo ./homeServerBackup.sh -t <TYPE (instant/daily/cleaner)>
     `instant` => used for executing script from terminal
     `daily` => used in 'sudo crontab' for scheduled backups
     `cleaner` => remove all instant backups from local & cloud directories
    
  5. For scheduled backups add above line as cronjob (preferebly with 'daily' type)
    1. Open crontab
      $ sudo crontab -e
    2. Add scheduled script execution
      For example below code with execute script located in everyday at 2:00 am.
      0 2 * * * cd /<example directory> && ./homeServerBackup.sh -t daily
      To set up cron schedule expressions see crontab gutu

Configuration

00. Functionality

Comment out (with <#>) features to exclude them from executingin 'instant' backups

declare -x -a functionalityInstant=(
    'Local Backup | Home directory'
    'Local Backup | Docker Volumes'
    'Local Backup | Docker Bind Mounts'
    'Cloud Backup'
    # (!) It is advised to use below features only for 'daily' backups
    #'Daily-backup cleaner'
    #'Daily-backup cloud cleaner'
    #'Daily-backup archiver'
)
#
declare -x -a functionalityDaily=(
    'Local Backup | Home directory'
    'Local Backup | Docker Volumes'
    'Local Backup | Docker Bind Mounts'
    'Cloud Backup'
    'Daily-backup cleaner'
    'Daily-backup cloud cleaner'
    'Daily-backup archiver'
)

To run Docker Volumes & Bind Mounts backups you need Docker environment.
To run Cloud Backup you have to have installed rclone and remote configured (named as homeServerBackup) For Daily-backup local and cloud cleaners you can specify how many backups you want to preserve

#   Choose how many daily backups you want to keep (daily-cleaner settings)
declare -x -i dailyLocal=5
declare -x -i dailyCloud=5
#

01. Directories & folders

Set up your:

  • Backup Directory - local destination of backups
  • Home Directory - server Home directory + name of it (of the backup)
#   Local Backup Directory
declare -x backupDir="path/to/location/where/backup/will/be/stored"
#
#   Server Home Directory
declare -x homeDir="/your/home/directory"
#
#   Name for backup of Server Home Directory
declare -x homeName="homeYourNameForExample"

To exclude certain sub-directories of Home directory just add it to the array:

#   Exclude those homeDir sub-directories from backup
declare -x -a excludeDir=(
    'photos'                #exclude homeDir/photos
    'folder/sub-folder'     #exclude homeDir/folder/sub-folder
)

02. Arrays of Docker Containers

Set up details of your Docker Volumes you would like to backup

#   declare -A volumeDocker00=(
#       [container]='Container name'
#       [volumePath]='Path to volume'
#       [name]='Backup .tar file name'
#       [stop]=true #or 'false' if you want to backup without stopping the container
#    )
#
declare -x -A volumeDocker00=(
    [container]='1st-container'
    [volumePath]='/path/to/volume'
    [name]='1st-container-backup'
    [stop]=true
)
#
declare -x -A volumeDocker01=(
    [container]='2nd-container'
    [volumePath]='/path/to/volume'
    [name]='2nd-container-backup'
    [stop]=false
)

03. List od Docker Containers with Bind Mounts

Set up you Docker Containers of which you would like to backup Bind Mounts. Code assume that you have Bind Mounts located in directory homeDir/Docker/<container name>. For each of the Docker Container you can choose whether to stop during backup or not.

declare -x bindDocker=(
    '3rd-container'
    '4th-container'
)
#
#   List of Docker Container with Bind Mounts to stop during backup
declare -x bindDockerStop=(
    #'3rd-container'
    #'4th-container'
)

04. Gotify Configurations

Set up Gotify server to notify you after each backup

#   Gotify website + token, for example: https://push.example.de/message?token=<apptoken>
declare -x GotifyHost="https://push.example.de/message?token=<apptoken>"
#
#   Title of Notification
declare -x GotifyTitle="yourServerName"

05. Netdata silencer

Backup of the server could couse high disk backlog and therefore spam notifications from Netdata about it (if you have that tool installed on the server). You can disable those notifications for the time of the backup script.

#   Replace <apptoken> with Netdata 'api authorization token'
#   that is stored in the file you will see in the following entry of http://NODE:19999/netdata.conf:
#       [registry]
#           # netdata management api key file = /var/lib/netdata/netdata.api.key
declare -x NetdataAuthToken="<apptoken>"
#
#   Change to 'true' to enable Netdata silencer
declare -x NetdataSilencer=false

If you would like to silende Netdata backlog notification not only for the time of the backup but also other scheduled activities (like automatic updates) it is better to disable those directly in the crontab:

# Disable Netdata disk.backlog notifications during scheduled activities
0 0 * * * docker exec netdata curl -s "http://localhost:19999/api/v1/manage/health?cmd=SILENCE&context=disk.backlog" -H "X-Auth-Token: <apptoken>"

# Daily Backups
0 1 * * * cd /<example directory> && ./homeServerBackup.sh -t daily

# Re-enable Netdata silenced notifications
0 3 * * * docker exec netdata curl -s "http://localhost:19999/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: <apptoken>"

License

home-server-backup is distributed under the GNU General Public License version 3.0 license. See LICENSE for more details.

home-server-backup's People

Contributors

gromoslaw-kroczka avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

home-server-backup's Issues

[Feature request] Release version to download (in which file format?)

Is your feature request related to a problem? Please describe.
Currently to execute script in other computer there is need to download whole repository, enable execution of main script and then execute it.

Describe the solution you'd like
Simpler download & run option should be available.

[Feature request] Documentation on restoring backup

I am fairly new to the whole homelab thing and this looks like a project which could help me solve backup situation coz my life has started to rely on services I am hosting like paperless-ngx, actual budget, etc.

What would be super helpful is to know how to restore the backups in case such a mishap do happen.

[Bug report] Unnecessary associative array of Docker Volumes in Configuration file

Is your feature request related to a problem? Please describe.
Below part of parameters.sh brings unnecessary configuration step (which could be avoided):

(...)

03. Associative Array

Expend list of docker volumes to correspond with number of your Docker Volumes.

declare -x -a volumeDockers=(
    'volumeDocker00'
    'volumeDocker01'
)

(...)

Describe the solution you'd like
Associative Array in Configuration file should be implemented directly in homeServerBackup.sh to avoid unnecessary configuration action.

[Feature request] Support for docker compose

I tried a proof of concept for my setup and realised that the configuration expects individual docker containers name in order to stop them and perform the backups.
I see a few limitations with this approach

  • How do you maintain the order of containers because most of the times they are dependent on other containers
  • Have to explicitly mention all containers in the file; even as a poc, I reached 10-12 containers with inter linking. My production load is roughly 20 containers.

So, I am not sure if this doable or not but a possibility to utilize docker compose in order to stop containers would be great. I imagine most of the production loads might be running via docker-compose.yml.

PS: apologies if this was already covered in your documentation; if you can point me to the right direction, that would be great.

[Bug report] False (?) error during daily-backup cleaner

Describe the bug
Daily-backup cleaner returns error message:
(!) Error during daily-backup cleaner (!)

To Reproduce
Steps to reproduce the behavior:

  1. Execute script in daily type (with daily-backup cleaner function enable)
  2. At which stage of the process => during / right after daily-backup cleaner

Expected behavior
Daily-backup cleaner performed message

Screenshots / Logs

find: โ€˜/media/external-HDD-500GB/backup/daily/2023-06-20_01-00-01โ€™: No such file or directory
#########################
(!) Error during daily-backup cleaner (!)
#########################

Additional context
Bug should be removed taking guidelines from for example -> How to find and delete directory recursively on Linux or Unix-like system

[Bug report] Empty folders in remote directory after Cleaning

Describe the bug
Cloud daily-cleaner doesn't remove directories, just deletes it's content => old empty folders are still present.

To Reproduce
Use cloud daily-cleaner functionality of the script

Expected behavior
Empty folders should be removed right after removing content from them.

[Feature request] Add "exclusions" for backing up local folders

This is a nice looking backup script! It would be awesome if you were able to add a "local exclusions" array to your parameters script.

For example, I have a photo cache in my home directory that I don't want to back up and it takes up a lot of space. It would be nice if I could specify a list of folders I don't want to be backed up.

[Feature request] Custom App Backup solutions implementation

Is your feature request related to a problem? Please describe.
Currently script backups only docker volumes and bind mounts. Many Apps include own backup systems, for example:

  • Paperless-ngx
  • Nextcloud
  • Home Assistant
    etc.

Those backups should be lighter in size and easier to use for example to restore in different environment.

Describe the solution you'd like
Add option for custom backups of popular applications (as described above)

Describe alternatives you've considered
This is just additional option, to be implemented

Additional context
I've recently moved to Proxmox and start using whole VM snapshots as main backup. In this scenario per-app-backups will be supplementary ones.

Additional challange: implement that using Python scripting?

[Feature request] Backup size comparator

Is your feature request related to a problem? Please describe.
Currently script preserve X previous daily backups locally and Y in the cloud (as configured in parameters.sh).
With growing data it will result in fast growth in size of the backup files.

Describe the solution you'd like
Keeping previous backups help to avoid overwriting valuable data with corrupted / missing one (before noticed that something broken). The same (or at least similar) protection we could assure by simply measuring sizes of backups old vs new and preserving previous one (not deleting it) only when new backup is smaller in size thank the old one.
We could measure backup as a whole, or separate parts of it, as required.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.