sebastianfeldmann / phpbu Goto Github PK

It seems that phpbu uses Mcrypt CLI which is not available for CentOS/AMI servers

phpbu 2.1.5

Configuration read from /var/www/phpbu/phpbu.xml

which: no mcrypt in (/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin)
Time: 3.74 seconds, Memory: 2.50Mb

Exception 'RuntimeException' with message ''mcrypt' was nowhere to be found please specify the correct path'
in phar:///var/www/phpbu/phpbu.phar/Util/Cli.php:141

FAILURE!
Backups: 1, failed Checks: 0, failed Crypts: 0, failed Syncs: 0, failed Cleanups: 0.

# php -m | grep mcrypt
mcrypt

# yum list installed | grep mcrypt
libmcrypt.x86_64                     2.5.8-9.1.2.amzn1             @amzn-main
libmcrypt-devel.x86_64               2.5.8-9.1.2.amzn1             @amzn-main
php55-mcrypt.x86_64                  5.5.28-1.106.amzn1            @amzn-updates

Are you going to add password-protected archives or OpenSSL file encryption? Current version can't do either. Absence of those features stops usage of phpbu in production.

hi guys,

how do you do a multiple source backup? for instance i want to backup mongodb data and at the same time i would want to backup source files from a folder

thanks

Haven't found any option to backup only changed files or mysql dumps

Hello @sebastianfeldmann
Congratulations for the great job they did here.

I'm testing running using the bootstrap but never with success.
I noticed that the file sent as a parameter is never executed.

Searching by code I saw that the method Runner::handleBootstrap() is never called. It is correct?

Cheers!

Hi,
I just check my backup, and for the sync options, seems the sync ( remote ) backup is not clean up. However the local backup is perfectly clean up. The phpbu email said the clean up process is okay.

Here is my config :

        "cleanup": {
            "type": "Quantity",
            "options": {
                "amount": "3"
            }
        },
        "syncs": [{
            "type": "sftp",
            "options": {
                "host": "xxx",
                "user": "xxx",
                "password": "xxx",
                "path": "xxx"
            }
        }]

Kind Regards,
Todi

Add a --simulate or --check option like the one available in ansible to execute a dry run without actually executing anything.

Best case this should output the things that would have happened on a normal phpbu execution.

Hi,
Is it possible to add mysqldump specific option , for example --lock-tables=false ? Do you interesting to add such feature ? If i want to fork it, what file i should look at ?

Kind Regards

<?xml version="1.0" encoding="UTF-8"?>
<phpbu xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="http://schema.phpbu.de/2.1/phpbu.xsd">
  <backups>
    <backup>
      <!-- backup source -->
      <source type="mysqldump">
        <option name="user" value="root"/>
        <option name="password" value="xxx"/>
      </source>
      <!-- where should the backup be stored -->
      <target dirname="backup/mysql"
              filename="mysqldump-%Y%m%d-%H%i.sql"
              compress="bzip2"/>
      <!-- cleanup the backup location
           as soon as all created backups exceed 500MB
           remove oldest backups -->
      <cleanup type="Capacity">
        <option name="size" value="500M"/>
      </cleanup>
    </backup>
  </backups>
</phpbu>

➤ phpbu --debug -v
phpbu 2.1.15

Configuration read from /root/phpbu.xml

create backup (mysqldump): /usr/bin/mysqldump --user='root' --password='xxx' --all-databases 2> /dev/null > /root/backup/mysql/mysqldump-20160119-1901.sql
exception: mysqldump failed
failed
Time: 71 ms, Memory: 1.50Mb

Exception 'phpbu\App\Exception' with message 'mysqldump failed'
in phar:///usr/local/bin/phpbu/Backup/Source/Mysqldump.php:203

backup mysqldump: FAILED

          | executed | skipped | failed |
----------+----------+---------+--------+
 checks   |        0 |         |      0 |
 crypts   |        0 |       0 |      0 |
 syncs    |        0 |       0 |      0 |
 cleanups |        0 |       0 |      0 |
----------+----------+---------+--------+

FAILURE!
Backups: 1, failed Checks: 0, failed Crypts: 0, failed Syncs: 0, failed Cleanups: 0.

But if I execute the debugged command

/usr/bin/mysqldump --user='root' --password='xxx' --all-databases 2> /dev/null > /root/backup/mysql/mysqldump-20160119-1901.sql

The the .sql file is created successfully.

• /root/backup/mysql/ exists and has write permissions
• logged as root
• root (mysql) user has plenty privileged on MySQL
• Using MariaDB 10.1 Ubuntu 14.04 LTS x86

... 😕

Add pg_dump support.

Copy.com has been discontinued: https://www.copy.com/page/

The sync driver should probably be removed to prevent people from using it.

Create rsync source.

The idea it to collect files (backups) from a remote server.
This is great if the production server can't access the backup server in a local network.
So instead of pushing backups to the backup server use phpbu to pull all created backups.

Hi Sbas,

alles fit?
will continue in english now for the github users.

Nice tool you are doing there. Tried it on one of my servers.
Found some bugs.
First is a timezone error which php is complaining about.
And on the last line i get an error message about dropbox.

Here is the output:

#php phpbu.phar --configuration=myconf.xml

phpbu 1.1.2

Configuration read from myconf.xml

PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in phar:///root/git/phpbu.phar/Util/String.php on line 32
PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in phar:///root/git/phpbu.phar/Util/String.php on line 32
PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in phar:///root/git/phpbu.phar/Util/String.php on line 32
PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in phar:///root/git/phpbu.phar/Util/String.php on line 32
PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in phar:///root/git/phpbu.phar/Util/String.php on line 32
PHP Fatal error:  Class 'phpbu\Backup\Sync\dbx\WriteMode' not found in phar:///root/git/phpbu.phar/Backup/Sync/Dropbox.php on line 85

Here is my php version:

#php -version

PHP 5.6.5-1~dotdeb.1 (cli) (built: Jan 28 2015 05:40:56) 
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies

Running on:

lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 7.8 (wheezy)
Release:    7.8
Codename:   wheezy

I'm using non default datadir, i'm use /backup/mysql and then when configuration file in source section use xtrabackup

Using server version 10.1.9-MariaDB
Error: option 'datadir' has different values:
'.' in defaults file
'/backup/mysql/' in SHOW VARIABLES

ps : need option datadir

Remove all deprecated 3.x code for version 4.0

Add the settings necessary for incremental backups f.e., --incremental-basedir

https://www.percona.com/doc/percona-xtrabackup/2.1/xtrabackup_bin/incremental_backups.html

After the backup is done (bz2 file is created, dump folder deleted) , i get this error :

Exception 'phpbu\App\Exception' with message 'unknown check: minsize'
in phar:///usr/local/bin/phpbu/Factory.php:96

phpbu.xml

<?xml version="1.0" encoding="UTF-8"?>
<phpbu xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="http://schema.phpbu.de/2.1/phpbu.xsd">
  <backups>
    <backup>
      <!-- backup source -->
      <source type="mongodump">
      <option name="host" value="x.x.x.x"/>
          <option name="databases" value="zzzz"/>
        </source>
      <!-- where should the backup be stored -->
      <target dirname="backup/mongo"
              filename="mongodump-zzzz-%Y%m%d-%H%i.json"
              compress="bzip2"/>
      <!-- check the created backup -->
      <check type="MinSize" value="10M"/>
      <!-- cleanup the backup location
           as soon as all created backups exceed 500MB
           remove oldest backups -->
      <cleanup type="Capacity">
        <option name="size" value="3G"/>
      </cleanup>
    </backup>
  </backups>
</phpbu>

I am currently implementing an XtraBackup source which uses Percona's XtraBackup tool to perform backups for mysql.

The problem is that XtraBackup will dump a bunch of files into a folder rather than producing 1 file such as backup.sql. I have already implemented the source and it works properly.

The only problem is if compress and encrypt is enabled on the target, it will break because those methods expect a file and not a folder of files.

1 solution is to have the XtraBackup source tar the files, but I feel the source is probably doing a bit too much.

Let me know what you think.

I also plan to implement a few more sources which will produce dumps that are a bunch of files instead of 1 file.

Since copy.com is now discontinued, I think google cloud drive is a good viable alternative (same amount of storage is offered in the free plan).

The point here is that using $result = $this->runner->run($configuration); the following should return false when a check failed: $result->wasSuccessful(). But in my test case, it returns true as in success. While it should return that the entire backup failed because the check failed, right?

Debugged it and boils down to this (I could be totally wrong though):

1. Added a check to a backup
2. Check fails and https://github.com/sebastianfeldmann/phpbu/blob/master/src/Result.php#L285 is called. This adds a failed check and also fails the backup.
3. This only updates the backup, not the result which is used to determine a successful backup run! This is done here: https://github.com/sebastianfeldmann/phpbu/blob/master/src/Result.php#L133 where the value being used (backupsFailed) is only updated when a backup itself failed (https://github.com/sebastianfeldmann/phpbu/blob/master/src/Result.php#L237). The checksFailed value is not being used anywhere except in the verbose CLI logging, I think?

In short, shouldn't wasSuccessful in the result class also check for failed checks, syncs, … instead of just backups?

I started to use phpbu for backup databases from my projects some days ago.

Locally all work well and old backups have been deleting normally.

But in remote storage I have all backups from the beginning.
I use copy.com as remote.

Is it normal?
Are you going to make deleting old backups from remotes?

Add a Flysystem Sync that supports all file systems supported by Flysystem

• https://github.com/thephpleague/flysystem
• http://flysystem.thephpleague.com

The docs on extending phpbu seems to be missing some sections: http://phpbu.de/documentation.html#extend-source

http://phpbu.de/manual/current/en/configuration.html#configuration.json

Line 26: sould be 10M not 10MB

It seems i am having a issue with mongodump, i think the problem is at the

<?xml version="1.0" encoding="UTF-8"?>
<phpbu xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="http://schema.phpbu.de/3.0/phpbu.xsd"
       verbose="true">

  <logging>
    <log type="json" target="logfile.json"/>
  </logging>

  <backups>
    <backup>
      <!-- data to backup -->
      <!-- source mongodump -->
      <source type="mongodump">
        <option name="host" value="localhost"/>
        <option name="databases" value="all"/>

      </source>

      <!-- where should the backup be stored -->
        <target dirname="backup/" filename="mongodata-%Y%m%d-%H%i.bz2" compress="bzip2"/>


      <!-- do some sanity checks to make sure everything worked as planned -->
      <!-- <check type="sizemin" value="2M"/> -->

      <!-- sync backup to some location or service -->
      <sync type="amazons3">
        <option name="key" value="x"/>
        <option name="secret" value="x"/>
        <option name="bucket" value="x"/>
        <option name="region" value="us-east-1"/>
        <option name="path" value="back/%Y"/>
      </sync>

      <!-- deletes old backups -->
      <cleanup type="capacity">
        <option name="size" value="100M"/>
      </cleanup>
    </backup>
  </backups>
</phpbu>

Exception 'phpbu\App\Exception' with message 'Mongodump failed'
in phar:///usr/local/bin/phpbu/Backup/Source/Mongodump.php:163

backup mongodump: FAILED

Edit:
After some digging i foud this command, witch makes the error, generated by phpbu```
root@ip-172-31-63-218:/root# /usr/bin/mongodump --out '/root/backup/' --host 'localhost' --database 'status'
2016-02-11T03:44:41.287+0000 error parsing command line options: unknown option "database"
2016-02-11T03:44:41.287+0000 try 'mongodump --help' for more information

The Runner class is currently a test nightmare.
Do something to fix this ;)

After #15 and #16 are merged, can you release 2.0.1 and build a phar? I think the changes we have made warrant a release 😄

For completition and beginners, haven't found anything about scheduling, i.e. with crontab. Examples?

Use openSSL to encrypt backups

There is no documentation for array of cleanups on json. Not like tag of checks:

"checks": [
        {
          "type": "sizemin",
          "value": "10MB"
        }
      ]

Here is my config of cleanup, but not work and error. If i change "cleanups" to "cleanup" my backup is not running.

"cleanups": [
        {
          "type": "capacity",
          "options": {"size": "200M"}
        },
        { 
         "type": "outdated",
          "options": {"older": "2W"}
        }
      ]

is possible make cleanup array json like check?

Add new Backup Source for redis

Use redis-bgsave to backup the redis persistence.
http://redis.io/commands/bgsave
Loop over redis-lastsave to check for succeeded backup.
http://redis.io/commands/lastsave

The Amazon AWS SDK should be upgraded to version 3.x

It would be awesome if the configuration files could be written in JSON and YAML, which are easier to read than XML.

I am currently working on the XtraBackup source. One annoying problem is that innobackupex will refuse to backup if the folder to store the backups in already exist. This is a safety feature to prevent accidental overwrites of backups. However, it also exits when the target folder is empty (which is really annoying!). This issue is documented here, but it is uncertain when the problem will be fixed.

Would it be possible if phpbu is updated so that sources can signal whether it wants the target directory created? This would provide a nice work around to fix the problem.

When trying to install using composer, I get error messages because apparently phpseclib/phpseclib 2.0.*@dev doesn't exist?

$ php composer.phar require "phpbu/phpbu"
Using version ~2.0 for phpbu/phpbu
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for phpbu/phpbu ~2.0 -> satisfiable by phpbu/phpbu[2.0.0].
    - phpbu/phpbu 2.0.0 requires phpseclib/phpseclib 2.0.*@dev -> no matching package found.

Potential causes:
 - A typo in the package name
 - The package is not available in a stable-enough version according to your minimum-stability setting
   see <https://groups.google.com/d/topic/composer-dev/_g3ASeIFlrc/discussion> for more details.

Read <http://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.

Installation failed, reverting ./composer.json to its original content.

We should create a vagrantfile so that an local environment can be easily booted for testing.

We also need a setup script (for travis and vagrant) to install the servers (Elasticsearch, etc), so that we can actually test the back up process by backing up real sources.

\phpbu\App\Backup\Source\Tar has a removeDir setting in its setup method to specify if the source directory should be removed after archiving, but it is not being used anywhere.

Shouldn't this be passed to the executable's (\phpbu\App\Cli\Executable\Tar) removeSourceDirectory method? See https://github.com/sebastianfeldmann/phpbu/blob/master/src/Backup/Source/Tar.php#L131.

After painfully configuring the correct values for S3 demo below, maybe will make a pull request for the docs

      "syncs": [
        {
            "type": "amazons3",
            "options": {
            "key": "KEY",
            "secret": "SECRET",
            "bucket": "https://s3.amazonaws.com/bucket-name",
            "region": "us-east-1",
            "path": "/back"
          }
        }
      ],

I get this exception and it's annoying.

Exception 'phpbu\App\Backup\Sync\Exception' with message 'The request signature we calculated does not match the signature you provided. Check your key and signing method.'
in phar:///app2/phpbu.phar/Backup/Sync/AmazonS3.php:130

Using the latest phpbu 2.1.15

And no, the keys and secret are correct, checked for whitespaces, and all of the above.

Any ideas ?

It fails because of the -f flag, which doesn't means freshen for zip, not force (like for gzip and bzip2).

Create a ducumentation repository so the documentation could be updated via pull requests.

It isn't recommended to put database credentials the git repository, so perhaps we should keep them out of the backup config also? And also, a lot of projects already have defined them anyways somewhere.

An option would be to provide some default project parsers, to load the configuration from the actual project.

Laravel -> Load database.php config file, or from .env directly.
Magento -> Parse local.xml file
Drupal -> Parse sites config file

Etc

Hi, my mysql database password contain &

and get the error

Error loading file "/home/wwwroot/project/backup/phpbu.xml.dist".

EntityRef: expecting ';'

Hi ,
I try to configure sync (configuration posted below) . However, when i run phpbu the sync command is not executed, is there anything wrong with my config ?

{
  "verbose": true,
  "logging": [
    {
      "type": "json",
      "target": "/home/kasatka/backup/one/one.json.log"
    }
  ],
  "backups": [
    {   
      "source": {
        "type": "mysqldump",
        "options": {
          "databases": "wordpress",
          "user": "root",
          "password": ""
        }
      },
      "target": {
        "dirname": "/home/kasatka/backup/one",
        "filename": "mysql-%Y%m%d-%H%i.sql",
        "compress": "bzip2"
      },
      "sync":{
        "type": "dropbox",
        "options": {
          "token": "myCrazyLongApiTokenThatIGotFromDropbox",
          "path": "/some/dir"
        }
      },
      "cleanup": {
        "type": "Quantity",
        "options": {
          "amount": "3"
        }
      }


    }

  ]
}

For security reasons no passwords or api keys should be logged to potentialy "public" logfiles, mails or the terminal output.

For example mongodump doesn't create a single file backup.
Mongodump like many other tools create multiple files in a directory.

Allow to define a target without filename and without compression to keep the unkompressed directory as backup.

Todo:

• Checks have to work with directories as well
• Collector / Cleaner have to work with directories
• Syncs...

Hi, Sebastian!

Are you going to make your project working in Windows?

If no, I'd like to do it, because my project works both Linux and Windows and I'd like one 'backup script' in both OS.

In the sample config file here: http://phpbu.de/getting-started.html

The check should be SizeMin and not MinSize.

Hi!
I'm trying to run my phpbu setup from crontab.

The command line runner phpbu --configuration=/home/d.xml works fine.
Also the cronjob (CentOS 7) is working: 0 * * * * /usr/local/bin/phpbu --configuration=/home/d.xml, except the syncing:

Exception 'phpbu\App\Backup\Sync\Exception' with message 'Error executing HTTP request: '
in phar:///usr/local/bin/phpbu/Backup/Sync/Dropbox.php:101

backup mysqldump: OK, but skipped or failed Crypts, Syncs or Cleanups!

What am I missing? Why crontab can't execute HTTP request, while I have a working setup of phpbu?

Thank you for your help!

(I'm sorry for the spam in the issue tracker, I couldn't find any other maillist/forum etc...)

There is an error in the docs or in the code with MinSize...

In docs we can see:
<check type="MinSize" value="30M"/>

In code works only this variant:
<check type="SizeMin" value="30M"/>

There is problem with formatting of email content.

This is what I see in command line:

and this is screenshot of email content: