See: https://community.openenergymonitor.org/t/backup-module-usb-import-sh-script-beta/12234/17

The location of the files etc are hardcoded to assume /home/pi/backup.

This location is not feasible for me. I have changed what I can find but the backup module still does not work for me.

These 2 chown commands have hardcoded ownership.

The backup module uses settings.php

It will also need to backup the ini file.

The following errors are seen in the Apache logs when the backup page is selected.

cat: /var/log/emoncms/exportbackup.log: No such file or directory
cat: /var/log/emoncms/exportbackup.log: No such file or directory
cat: /var/log/emoncms/usbimport.log: No such file or directory

Could direct compression work better than creating the backup file first?

https://github.com/maennchen/ZipStream-PHP

Testing on home system consistently fails when appending 111mb of phpfina files. Partition becomes unresponsive and requires a reboot to get the pi working again.

I'm working on some pull requests for minor tweaks on the import and export shell scripts and noticed the following code which I cannot find being used anywhere:

It used to work here and was commented out some time ago:

And similar for export it has this code which isn't referenced anywhere:

I propose that these code segments are no longer of use and should be removed to tidy up the scripts a little.

I'm happy to work on pull requests and some testing and wanted to get your thoughts before spending time on it. What do you think?

@borpin I've started the process of implementing this.

First of all, the locations are changed in the master branch of the EmonScripts install scripts
openenergymonitor/EmonScripts@dc44ca2

Then the backup module install.sh script creates the folders in the right locations:
https://github.com/emoncms/backup/blob/master/install.sh

I've tested both import and export here and it works great.

So far this will only work for people building a new system using EmonScripts master branch and master branch of backup module, next step is to work out the update process...

I can't run backup from user interface

backup button does nothing
also the black screen shows no text
I can run backup from the script emoncms-export and the resulting achive can be seen and downloaded in backup module

The service-runner instructions confilct with the core instructions.

So just upgraded from v9.8 to v10.3.2 via a more or less manual reinstall of emoncms on a debian system which was reasonably painless considering emoncms is built for PIs

I did this to be able to backup my system from the webgui. However it didn't work. Pressing the backup button, absolutely nothing happened.

The problem. The backup_controller.php uses this line to push to redis:
$redis->rpush("service-runner","$export_script $export_flag>$export_logfile")

But the service-runner.py was doing nothing. A little bit of python monitoring thus:

import redis
server = redis.Redis()
with server.monitor() as m:
   for command in m.listen():
      print (command)

...revealed that when pressing the Create backup button backup_controller.php was sending this to redis: (shortened a bit..)

u'command': u'RPUSH emoncmsservice-runner /opt/emoncms/modules/backup/emoncms-export.sh

But this code in service-runner.py:

      if server.exists('service-runner'):
        flag = server.lpop('service-runner')
      elif server.exists('emoncms:service-runner'):
        flag = server.lpop('emoncms:service-runner')

checks for 'service-runner' or 'emoncms:service-runner and not the 'emoncmsservice-runner' which is being posted by the .php

Note the discrepancy over the colon after 'emoncms' and before 'service-runner'
I've not discovered yet where this is set in emoncms so for now I modifed the service-runner.py file and removed the colon from there. Once this was done the service-runner picked up the script and the backups sprang into life.

For the record my settings.ini file has this relating to redis:

; Redis Database (used as a cache for improved performance)
[redis]
enabled = true
host = 'localhost'
port = 6379
auth = ''
dbnum = ''

So it looks like a bug and not a configuration problem. Or at least a very nasty gotcha!
I thought this would be useful to know.

Posted in backup as that's what was being affected for me although it could be a service-runner issue or and emoncms (config but where?) issue.

Thanks.
Andy.

Edit: code formatting barf...

Status, flashed Oct18 SD, let update run.

On importing data I get this error

And it seems to have logged me out as I get an error if I try to access the administration screen

As per discussion (and solution) here the import fails if there are only 2 partitions i.e. the data is on the root partition.

The script recovers from this later when looking for the mysql database, but not for the data files in /var/lib.

Suggest an extra check is made to find the data folders (including phpfiwa).

https://community.openenergymonitor.org/t/backup-v2-1-7-error-restoring-an-local-emonbase-backed-up-today-image-emonsd-17oct19-stable/12522

A large backup file restore failed with an error that did not make sense to the user. It seems it failed because the backup file was too large.

Should the size of file that can be backed up match the size of a file that can be restored?

If a data partition of 10Gb is being created, it should be possible to backup and restore larger data sets.

https://github.com/emoncms/backup/blob/8a0f0afaf911a80fce5e67ea65dc0dcff9002a4f/install.sh#L22-l26

Ref https://community.openenergymonitor.org/t/permanent-raspberry-pi-booting-after-failed-emonpi-update/10963/27

This has some discussion about modifying php.ini bu drop-in rather than changing the master file. As I think the Vhost mechanism has been enabled, this could probably be done an just the vhost.

=== Emoncms export start ===
Tue Jan 2 14:47:27 UTC 2018
EUID: 1000
Reading /home/pi/backup/config.cfg....
Location of mysql database: /home/pi/data
Location of emonhub.conf: /home/pi/data
Location of emoncms.conf: /home/pi/data
Location of Emoncms: /var/www/emoncms
Backup destination: /home/pi/data
old image
Log is turned off
Stopping feedwriter: stopped
Emoncms MYSQL database dump complete, adding files to archive ..
tar: Removing leading '/' from member names
tar: /home/pi/data/node-red/flows_emonpi.json: Cannot stat: No such file or directory
tar: /home/pi/data/node-red/flows_emonpi_cred.json: Cannot stat: No such file or directory
tar: /home/pi/data/node-red/settings.js: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
Error: failed to tar config data
emoncms export failed

the problem is that the ~/data directory is getting full of daily backup files.

It'd be tidy if the default backup location for the backup was another directory within ~/backup

for example I have ~/backup/backups made

but when I set it in config.cfg I get this error:

When import script successful imports an old backup for best user experience it should auto log out the current user session landing the user back at the login page where they can log in with the newly import account credentials.

May be worth exploring this as an alternative backup method

A check needs to be done to not overwrite an existing config file.

ref https://community.openenergymonitor.org/t/permanent-raspberry-pi-booting-after-failed-emonpi-update/10963/22?u=pb66 and https://www.php.net/manual/en/function.parse-ini-file.php

Originally I submitted PR #37 to swap out the hashes for semi-colons but that caused and issue for the bash script so I closed it again.

Note that reading the parse_ini_file() changelog the use of hashes should effect php7.0 users, but as it turns out, the linked case is a php5.6 setup ????

If someone is trying to do a manual backup on migration this will not work.

Check for presence of a ini file in the root folder I suggest.

From a number of queries in the community, I suspect that emonhub.conf i not always imported correctly.

Because there are 2 ifs it is possible that the one in use, could be overwritten by a very old version.

Compress to a single if and save the default version. Bias towards the new structure configuration

PHP Warning:  Invalid "upload_max_filesize" setting. Invalid quantity "3G": value is out of range, using overflow result for backwards compatibility in Unknown on line 0
PHP Warning:  Invalid "post_max_size" setting. Invalid quantity "3G": value is out of range, using overflow result for backwards compatibility in Unknown on line 0

same pb with 2G, only 1G seems to be OK

This variable is checked but has never been set.

When I try to run the backup it says it fails to connect to mysql on localhost.
For emoncms I use a mysql-database on another server.
Somehow it doesn't use the settings in settings.php og emoncms...

The install instructions (and possibly the module) have not been updated to Stretch.

See: https://community.openenergymonitor.org/t/usb-import-script-failed/13425

These scripts rely too heavily on hardcoded paths and assume the 'pi' user.

When installed on a different system (such as ubuntu) with a different username, the various scripts fail (import for instance) because of this.

This could be a daily process that does an incremental backup to improve chances of system recovery. Needs good external drive discovery (re #52).

Implementation date: for review

This module only works with emonpi, is there a working version for Ubuntu?

Hi,

As part of my import/ export script tidy up I saw this shell script, but cannot find anywhere within this module that its called and it looks to be an early script as locations are hard coded under /home/pi etc.

https://github.com/emoncms/backup/blob/master/emoncms-copy.sh

I propose to remove the script, is this is use or can I submit a pull request to remove it?

Thanks,
Clive.

I think this should be changed to emoncms_mqtt

The Import Export tab colour should be darker, currently it's very light and even disappears on mouseover

For people that have multiple emonPi / emonCMS installations it would be better for the backup filename to include the machine hostname.

e.g.
myemonpi-emoncms-backup-2020-04-28.tar

Doing a restore on a non-pi emoncms Ubuntu install. Import database seems fine, but can't login or register.

The Backup module needs sidebar compatible menu added. The backup module has it's own sidebar menu to switch between Export / Import pages.

With the module install location moving. This is no longer guaranteed to be correct. However, during transition it cannot simply be changed, two checks will be required one for each possible location.

The export script, looks for the config in the new location.

When I try to create a backup via GUI, it shows me the following error: Segmentation fault.
Is there a way to fix this?