bernieo / calcardbackup Goto Github PK

Infos:
Ubuntu 18.04 Server
User are synched from LDAP Server
Nextcloud 15.0.5 (just updated from 15.0.2 didn't work there neither)
MariaDB 10.1.38

Problem:
Doesn't backup calendars of existing users/calendars. Backup of admin calendar works without a problem, it also sees the users (I assume) because of the empty adressbook it finds.

Tests:
If I create a new calendar within a user it sees it "...test-cal-shared.ics"
If I create a new user in LDAP and then create a calendar it also works "...Jodl.ics"
script is executed with
"sudo -u www-data ./calcardbackup "/var/www/nextcloud"
also tried with
"sudo -u www-data ./calcardbackup "/var/www/nextcloud -u users.txt -z"
"sudo -u www-data ./calcardbackup "/var/www/nextcloud -u users.txt -i"

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+

• Wed Mar 27 12:55:51 UTC 2019 --> START calcardbackup ver. 0.8.7 (20.03.2019), AGPL-3.0
• Checking dependencies and preparing...
• no usersfile given: will backup all available items
• Using URL: https://example.org
• no valid status.php found at https://example.org.
• Nextcloud 15.0.5 detected.
• Database of chosen Nextcloud installation is MySQL/MariaDB.
• Looking for calendars in your Nextcloud:
• saving calendar admin-Personal.ics (from database)......success!
• saving calendar 5d766aba-a5bb-1037-8940-e3222c59b1f9-test-cal-shared.ics (from database)......success!
• saving calendar 5186a584-55b1-1038-9c51-b125d912bbcd-Jodl.ics (from database)......success!
• Looking for calendarsubscriptions in your Nextcloud:
• No calendarsubscriptions found.
• Looking for addressbooks in your Nextcloud:
• saving addressbook admin-Contacts.vcf (from database)......empty file. Addressbook 'Contacts' of user 'admin' is empty.
• saving addressbook cb3e0866-d5b6-1037-8478-fb9428484cf6-Contacts.vcf (from database)......empty file. Addressbook 'Contacts' of user 'cb3e0866-d5b6-1037-8478-fb9428484cf6' is empty.
• saving addressbook b0bc3152-d5b6-1037-8477-fb9428484cf6-Contacts.vcf (from database)......empty file. Addressbook 'Contacts' of user 'b0bc3152-d5b6-1037-8477-fb9428484cf6' is empty.
  +++
  more adressbooks of all users - same message
  +++
• saving addressbook 5d766aba-a5bb-1037-8940-e3222c59b1f9-redacted.vcf (from database)......success!
• Compressing backup as *.tar.gz file. Be patient - this may take a while.
• Backup successfully compressed!
• Find your backup here: /backup/calcardbackup/backups/calcardbackup-2019-03-27.tar.gz
• Wed Mar 27 12:55:52 UTC 2019 --> END calcardbackup

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I've installed calcardbackup to my Synology NAS with installed Nextcloud in web-folder.
After starting script I get this Error:

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+  Sun Nov 24 18:51:27 CET 2019 --> START calcardbackup ver. 0.9.5-1 (25.09.2019), AGPL-3.0
+  Checking dependencies and preparing...
+  no usersfile given:
+     - will fetch all available items from database
+  Using URL: https://192.178.168.99/web
+  can't read status.php (cURL code '60').
+  Nextcloud 17.0.1 detected.
-- calcardbackup: ERROR --
-- ERROR: Nextcloud database is MySQL/MariaDB. Therefore this script requires package 'mysql-client' or 'mariadb-client'.
-- Install according package and run script again
-- calcardbackup: Exiting.

Ip Adresse is correct, backup folder has been created by script, but I have no idea, what I have to do now.

An empty dbtableprefix does not always mean that it is the default oc_. It can mean it is no prefix.

  # set default dbtableprefix if empty or not found in config.php:
  [[ -z "${dbtableprefix:-}" ]] && dbtableprefix="oc_"

My nextcloud configuration for dbtableprefix is 'dbtableprefix' => '' and it mean that the tables are not prefixed. The calcardbackup does not backup my calendars and addressbooks.

+  Looking for calendars in your Nextcloud:
-- NOTICE: table 'oc_calendars' containing calendars does not exist in database.
-- NOTICE: Looks like your Nextcloud did not create any calendars yet.
-- INFO: Couldn't find a single calendar in your Nextcloud!
+  Looking for addressbooks in your Nextcloud:
-- NOTICE: table 'oc_addressbooks' containing addressbooks does not exist in database.
-- NOTICE: Looks like your Nextcloud did not create any addressbooks yet.
-- INFO: Couldn't find a single addressbook in your Nextcloud!

Script Version: 0.8.10
Nextcloud Version: 16.0

Usage: sudo -u www ./calcardbackup /usr/local/www/apache24/data/nextcloud/ -o /mnt/backup/ -r 30

When passing "-o 30" as parameter (currently there are no backups older than 30 days, not sure if this is relevant), I get the following error:

find: -daystart: unknown primary or operator
-- calcardbackup: ERROR --
-- ERROR: deleting backups older than 30 days produced above error.
-- calcardbackup: Exiting.

I've followed the instructions, but running sudo -u www-data ./calcardbackup /var/www/nextcloud/ -i -x neither produces a directory with content nor any (error) message.

• Prerequisites are fullfilled
• Nextcloud 13.0.4 is up and running
• Server certificate is valid (not self-signed)
• users.txt file contains valid users and passwords
• users have personal and shared calendar data

Any suggestions what's wrong? Is there a way to make the script more verbose?

thank you for your tool... was working great....

but: as i want to export and reimport into a fresh new instance, some old contacts were exported as Version2.1, others as V3.0.... maybe 4.0 but i have not checked that.

those v2.1 vCards cannot be imported...

is it possible to add a feature to check those files before saving them?
a tool to convert is e.g. https://github.com/jowave/vcard2to3

Not directly compatible with snap installation as snap installation comes with a mysql bundle and does not require a standalone mysql server to be installed.

Notes:
URL to find configs is:
/var/snap/nextcloud/current/nextcloud/
MySql files are at:
/var/snap/nextcloud/current/mysql/nextcloud

Relevant outputs:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+

• Mon Apr 9 08:01:12 UTC 2018 --> START calcardbackup ver. 0.4.1 (06.03.2018), AGPL-3.0
• Checking dependencies and preparing...
• Using URL
• Nextcloud 12.0.5 detected.
• Database of chosen Nextcloud installation is MySQL/MariaDB.
• Looking for calendars in your Nextcloud:
  ./calcardbackup: line 535: mysql: command not found
  -- NOTICE: table 'oc_calendars' containing calendars does not exist in database.
  -- NOTICE: Looks like Nextcloud did not create any calendars yet.
  -- INFO: Couldn't find a single calendar in your Nextcloud!
• Not backing up addressbooks as configured.
  -- WARNING: No files in backup directory - meaning no backup created !!
• Mon Apr 9 08:01:13 UTC 2018 --> END calcardbackup

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Output after a standalone mysql server is installed:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+

• Mon Apr 9 08:31:20 UTC 2018 --> START calcardbackup ver. 0.4.1 (06.03.2018), AGPL-3.0
• Checking dependencies and preparing...
• Using URL
• Nextcloud 12.0.5 detected.
• Database of chosen Nextcloud installation is MySQL/MariaDB.
• Looking for calendars in your Nextcloud:
  ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/sockets/mysql.sock' (2)
  -- NOTICE: table 'oc_calendars' containing calendars does not exist in database.
  -- NOTICE: Looks like Nextcloud did not create any calendars yet.
  -- INFO: Couldn't find a single calendar in your Nextcloud!
• Not backing up addressbooks as configured.
  -- WARNING: No files in backup directory - meaning no backup created !!
• Mon Apr 9 08:31:21 UTC 2018 --> END calcardbackup

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Occurances of the psql command with dbhost and/or dbprotocol arguments require proper option assignments :

psql "${dbhost}" "${dbprotocol}"
becomes
psql -h "${dbhost}" -p "${dbprotocol}"

Using the script on OpenBSD I noticed the following interop issue:

• readlink -vf

Flag support overview:
OpenBSD, FreeBSD have: -f
NetBSD has: both
Dragonfly has: neither
Busybox has: both
GNU has: both

Proposed solution:
I think, you could get rid of "-v". For error reporting you already use $? anyway. You could enrich the error message with the used argument, if you want this information to be displayed.

Edit: Former, similar issue '-m' in #4

Hi BernieO,
i have a feature request for you very nice backup tool.
Is it possible to integrate a backup of the Bookmarks app?
https://apps.nextcloud.com/apps/bookmarks
Because the bookmarks are also saved in database tables.

Thanks and regards
Rene

I tried to backup calendar and addressbook of a Nextcloud installation:

• Debian 9
• MariaDB 10.1.37
• NC 13.0.8
• 3000+ users

Unfortunately the script cannot run because of an faulty address book (backup of calendars went successfully). It gets the following error message:

ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '' at line 1

After line 1283 ("db_query='"SELECT ${convert:-carddata} FROM ${table_cards}...") I added the following output-line:
_output printf "Principal-Name='${principal[$i]}' SELECT '${convert:-carddata}' FROM '${table_cards}' WHERE addressbookid='${id[$i]}'"

and get the following output:

+ saving addressbook 37506df8-49ba-4b35-a08a-603a896b61ec-37506df8-49ba-4b35-a08a-603a896b61ec.vcf (from database)...Principal-Name='37506df8-49ba-4b35-a08a-603a896b61ec' SELECT 'carddata' FROM 'oc_cards' WHERE addressbookid=''ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '' at line 1

But I could not find the string "37506df8-49ba-4b35-a08a-603a896b61ec" in database (where does it come from?). Therefore I could not find out why it has no addressbook id..

Any help?

A user reported that the backups were not encrypted due to an error.
The relevant part of the error is:

+  Compressing backup as *.tar.gz file. Be patient - this may take a while.
+  Backup successfully compressed!
gpg: gpg-agent is not available in this session
gpg: can’t query passphrase in batch mode
gpg: error creating passphrase: invalid passphrase
gpg: symmetric encryption of `/opt/calcardbackup/backups/calcardbackup-2018-11-29.tar.gz’ failed: invalid passphrase
-- calcardbackup: ERROR –
-- ERROR: Encrypting the compressed backup did not work!
-- Check messages above.
-- calcardbackup: Exiting.

Searching the internet and with a little help of that user, I found out that the problem is caused when the versions of gpg and gpg-agent are different. The gpg-agent in version 2.x doesn't export the environment variables that gpg version 1.x expects:
https://askubuntu.com/questions/860370/gpg-agent-cant-be-reached/875179#875179
https://unix.stackexchange.com/questions/231386/how-to-make-gpg-find-gpg-agent

I can't export the environment variable in the script, because the paths are different on the different systems out there.

The solution to the problem is to install gpg2 and let the script prefer gpg2 over gpg (if installed).

I was able to export nextcloud calendar entries using this script, but it appears that it dumps everything into a single file. I am looking to import these entries to Radicale which needs separate entries. Is there a way this could be incorporated into the script?

Thanks in advance.

HI,

not sure but the script does not connect my mysql Database:

# sudo -u www-data ./calcardbackup
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+  Sa 26. Mai 11:16:42 CEST 2018 --> START calcardbackup ver. 0.4.1 (06.03.2018), AGPL-3.0
+  Checking dependencies and preparing...
+  Using configuration file /SCRIPTS/calcardbackup/calcardbackup.conf, ignoring all other command line options.
+  Using URL https://cloud.myhost.org
+  Nextcloud 13.0.2 detected.
+  Database of chosen Nextcloud installation is MySQL/MariaDB.
+  Looking for calendars in your Nextcloud:
ERROR 1045 (28000): Access denied for user 'owncloud'@'localhost' (using password: YES)
-- NOTICE: table 'oc_calendars' containing calendars does not exist in database.
-- NOTICE: Looks like Nextcloud did not create any calendars yet.
-- INFO: Couldn't find a single calendar in your Nextcloud!
+  Looking for addressbooks in your Nextcloud:
ERROR 1045 (28000): Access denied for user 'owncloud'@'localhost' (using password: YES)
-- NOTICE: table 'oc_addressbooks' containing addressbooks does not exist in database.
-- NOTICE: Looks like Nextcloud did not create any addressbooks yet.
-- INFO: Couldn't find a single addressbook in your Nextcloud!
-- WARNING: No files in backup directory - meaning no backup created !!
+  Sa 26. Mai 11:16:42 CEST 2018 --> END calcardbackup
+
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I added echo's to your script and checked the variables, they are correct, and executing with the echoed vars

sudo -u www-data mysql -u owncloud -p -h 127.0.0.1

also works without problems

ANALYSIS:
Ive created a mysql.cnf file like the printf does and tested:

 mysql --defaults-extra-file=mysql.cnf

ERROR 1045 (28000): Access denied for user

Then I escaped my complex password in mysql.cnf
from

[client]
user = owncloud
password = mypass&%_.,7c

to

[client]
user = owncloud
password = "mypass&%_.,7c"

and the file based connection works.. so it seems complex passwords have to be quoted.

SOLUTION:

mysql --defaults-extra-file=<(printf "[client]\nuser = %s\npassword = %s\nhost = %s\n%s"
has to be
mysql --defaults-extra-file=<(printf "[client]\nuser = %s\npassword = \"%s\"\nhost = %s\n%s"

Using the script on OpenBSD I noticed the following interop issue:

• The shebang is "#!/bin/bash"

Background:
On some operating sytems you do not find bash in "/bin", but in "/usr/bin" or "/usr/local/bin". This is at least true on OpenBSD and FreeBSD, I expect this on Mac OS and Solaris and probably some non-mainstream Linux Distros (esp. lightweight ones).

Possible solution:
"#!/usr/bin/env bash" is POSIX compatible and should work almost everywhere, where bash is installed into the $PATH.

We run multiple instances with the same codebase, by using the NEXTCLOUD_CONFIG_DIR variable (nextcloud/server#300).

In order for calcardbackup to support multiple instances, I propose the following patch :

# location of ownClouds/Nextclouds config.php:
configphp="${nextcloud_path}/config/config.php"

to

# location of ownClouds/Nextclouds config.php:
[[ -z "${NEXTCLOUD_CONFIG_DIR:-}" ]] && configphp="${nextcloud_path}/config/config.php" || configphp="${NEXTCLOUD_CONFIG_DIR}/config.php"

With fe7d9d8 you added a (better) detection of the vendor and version. Unfortunately this requires filesystem access to the Nextcloud/Owncloud documentation.

I'm running calcardbackup in a docker container where the filesystem of Nextcloud/Owncloud is not accessable. Anyway the resource is available also via http://host/base-uri/core/doc/user/index.html and it would be very great to fetch that resource by http (or make it optional) to work around this issue.

Many thanks, Jan.

Alpine (and others) ship with busybox.

$ /opt/calcardbackup/calcardbackup -h 
readlink: unrecognized option: m
BusyBox v1.27.2 (2017-12-12 10:41:50 GMT) multi-call binary.

Usage: readlink [-fnv] FILE

Display the value of a symlink

	-f	Canonicalize by following all symlinks
	-n	Don't add newline
	-v	Verbose
calcardbackup version 0.3.1 (06.02.2018), AGPL-3.0
Usage: ./calcardbackup [DIRECTORY] [option [argument]] [option [argument]] [option [argument]] ...
Find more details in attached file 'README.md' or visit 'https://github.com/BernieO/calcardbackup'

# readlink -h
readlink: unrecognized option: h
BusyBox v1.25.1 (2018-01-25 21:40:40 CET) multi-call binary.

Usage: readlink [-fnv] FILE

Display the value of a symlink

	-f	Canonicalize by following all symlinks
	-n	Don't add newline
	-v	Verbose

Hello Bernhard,
Thank you very much for your excellent bash script.
I missed an example for running calcardbackup in a daily CRON job.
Here it is:
/!\ In case you want to start a CRON job with a different user, you need to set it up in the file /etc/crontab.

# add calcardbackup to a daily CRON job
$ sudo nano /etc/crontab
# add to the end of the file:
# calcardbackup at every day at 2 o'clock with user "www-data"
0 2 * * * www-data /home/<user>/calcardbackup/calcardbackup "/var/www/nextcloud" > /dev/nul
<empty line>

# check in the log for a run of the script
$ grep www-data /var/log/syslog
Jul 17 02:00:01 rudiswiki74 CRON[2215]: (www-data) CMD (/home/rudi/calcardbackup/calcardbackup "/var/www/nextcloud" > /dev/null)

Regards, Rudi