Git Product home page Git Product logo

danpu's Introduction

Danpu - MySQL dump tool for PHP

Build Status Coverage Status Packagist

Danpu is a dependency-free, cross-platform, portable PHP library for backing up MySQL databases. It has no hard dependencies, and is fit for restricted environments where security is key and access is limited. Danpu requires nothing more than access to your database, PDO and a directory it can write the backup to. The script is optimized and has low memory-footprint, allowing it to handle even larger databases.

Danpu supports backing up table structures, the data itself, views and triggers. Created dump files can optionally be compressed to save space, and generated SQL output is optimized for compatibility.

Requirements

Minimum:

  • PHP 5.3.0 or newer
  • MySQL 4.1.0 or newer
  • PDO

Recommended, but optional:

  • PHP 5.4.0 or newer
  • MySQL 5.0.11 or newer
  • zlib

Backing up views and triggers requires MySQL 5.0.11 or newer.

Install

Using Composer:

$ composer require rah/danpu

Usage

To create a new backup or import one, configure a new Dump instance and pass it to one of the worker classes. To begin, first make sure you have included Composer's autoloader file in your project:

require './vendor/autoload.php';

If you are already using other Composer packages, or a modern Composer-managed framework, this should be taken care of already. If not, merely add the autoloader to your base bootstrap includes. See Composer's documentation for more information.

Take a backup

Backups can be created with the Export class. The class exports the database to a SQL file, or throws exceptions on errors. The file is compressed if the target filename ends to a .gz extension.

use Rah\Danpu\Dump;
use Rah\Danpu\Export;

try {
    $dump = new Dump;
    $dump
        ->file('/path/to/target/dump/file.sql')
        ->dsn('mysql:dbname=database;host=localhost')
        ->user('username')
        ->pass('password')
        ->tmp('/tmp');

    new Export($dump);
} catch (\Exception $e) {
    echo 'Export failed with message: ' . $e->getMessage();
}

The database is dumped in chunks, one row at the time without buffering huge amount of data to the memory. This makes the script very memory efficient, and can allow Danpu to handle databases of any size, given the system limitations of course. You physically won't be able backup rows that take more memory than can be allocated to PHP, nor write backups if there isn't enough space for the files.

Import a backup

Danpu can also import its own backups using the Import class. While the importer works with backups it has made, it doesn't accept freely formatted SQL. The importer is pretty strict about formatting, and expects exactly the same format as generated by Danpu. It expects that values in statements are escaped properly, including newlines, queries have to end to a semicolon and statements preferably should not wrap to multiple lines.

To import a backup, create a new instance of the Import class. It uncompresses any .gz files before importing.

use Rah\Danpu\Dump;
use Rah\Danpu\Import;

try {
    $dump = new Dump;
    $dump
        ->file('/path/to/imported/file.sql')
        ->dsn('mysql:dbname=database;host=localhost')
        ->user('username')
        ->pass('password')
        ->tmp('/tmp');

    new Import($dump);
} catch (\Exception $e) {
    echo 'Import failed with message: ' . $e->getMessage();
}

Options

In addition to the mandatory connection and file location, Danpu accepts various optional configuration options. These include filtering tables and views by prefix, ignoring tables and creating dumps without row data. See the src/Rah/Danpu/Config.php for full list of options. The source file contains detailed documentation blocks, outlining each option.

Troubleshooting

Running out of memory, backup taking too long

As with any PHP script, Danpu is restricted by the safe limits you have set for PHP. When dealing with larger databases, taking a backup will take longer and require more memory. If you find yourself hitting the maximum execution time or running out of memory, its time to increase the limits enough to let the script to work.

PHP lets you to change these limits with its configuration options, which can be modified temporarily during script execution, or in global configuration files. The configuration options you will be the most interested in, are memory_limit and max_execution_time, and possibly ignore_user_abort. You can change these values in your script before running a backup with Danpu:

ini_set('memory_limit', '256M');
set_time_limit(0);
ignore_user_abort(true);

This of course requires that you have access to these values and you actually have more memory to give. Keep in mind that PHP may be affected by other limitations, such as your web server.

danpu's People

Contributors

gocom avatar hmoragrega avatar tvb avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

danpu's Issues

How to correctly check if dump/restore went successful?

Is this the correct way to check if the dump/restore went ok?

            // Lets try to import the dump or catch the error if any.
            try {
                new Import($dump);

                // Send a successfully restored backup message.
                echo "Good job!";
            } catch (Exception $e) {

                // Oops, something went wrong! Let's show the error..
                echo "some error";
            }

Remove cache file if backup returns an error

I noticed danpu will place files of 0 bytes (i.e.: Rah_Danpu_CpCOO0) in the temporary folder if the backup fails for any reason (like unable to connect). Those should be removed to keep the working directory clean.

Implement better temporary file management

Temporary file management would be implemented using rah/eien. This would include:

  • An option, and defaults to, searching automatically for the system's temporary directory. This means the end user doesn't (necessarily) have to care about configuring it.
  • Using temporary files when doing the compressed version of the file. Current the temporary file is streamed to the final location without locking, and due to on-the-fly stream compression, the moving takes longer than simply renaming or copying the file. We should first compress, then move.

Cannot delete or update a parent row: a foreign key constraint fails

I am getting this error while trying to import my dump created by danpu:

SQLSTATE[23000]: Integrity constraint violation: 1217 Cannot delete or update a parent row: a foreign key constraint fails

SHOW ENGINE INNODB STATUS; shows me:
Error: Cannot drop table db.tableX because it is referenced by db.tableY

Now, I know I can work around it by setting foreign_key_checks to false.
https://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html#sysvar_foreign_key_checks

But imo it's a dirty solution and I would like to find a proper solution instead. Ideas?

add include-tables config option

It would be handy to send an array with included tables instead of only excluding the rest.
I.e.: If you want to include only 2 of the 50 tables you wouldn't have to exclude the other 48.

exporting/importing takes longer the mysqldump

exporting and importing a database takes longer than CLI mysqldump. Is there a way to speed this up?

Nothing immediate jumps out in Config.php

specifics: mysqldump created a file of 840M and took approximately 1 minute

this library took just over 15 minutes and created a file size of 930 M

Cannot add foreign key constraing

Importing exported database on the exact same mysql database via danpu Export,

following exception thrown

Caught exception: SQLSTATE[HY000]: General error: 1215 Cannot add foreign key constraint

other adapters?

Will this project support other database adapters such as Postresql and Sql Server?

create database if not exists?

is there a way to enable creating the database name ? I looked at config, import, export libraries, I don't see anything there, this would be a really helpful feature.

Add transactions to import?

Currently the function executes query by query and fails on one with incorrect syntax. This means that prior queries were executed. So we are left with a half-baked import. Either all or none of the queries should be executed. So maybe it would be a good idea to implement transactions into the import function?

Sorry if this is not how one should suggest this.

protected function import()
	{
		$query = '';
		$this->pdo->beginTransaction();
		try {
			while (!feof($this->file)) {
				$line = fgets($this->file);
				$trim = trim($line);

				if ($trim === '' || strpos($trim, '--') === 0 || strpos($trim, '/*') === 0) {
					continue;
				}

				if (strpos($trim, 'DELIMITER ') === 0) {
					$this->delimiter = substr($trim, 10);
					continue;
				}

				$query .= $line;

				if (substr($trim, strlen($this->delimiter) * -1) === $this->delimiter) {
					$this->pdo->exec(substr(trim($query), 0, strlen($this->delimiter) * -1));
					$query = '';
				}
			}
			$this->pdo->commit();
		} catch (\PDOException $e) {
			$this->pdo->rollBack();
                        // throws \PDOException as Exception to make things easier
			throw new Exception($e->getMessage(), $e->getCode(), $e);
		}
	}

Import problems

In the case that a php variable is serialized in a field sometimes the import fails.

Currently I don't see why, I try to detect the problem, in the next days.

Avoid heavy work in constructor

Creating an object does not carry the expectation that something is done behind the scenes. That's not an absolute, though side effects should be kept minimal. Here however, creating the object carries huge side effects, creating a potentially big file or overwriting a database.

Also, you don't necessarily want to do the task at hand right at construction time. Consider dependency injection: component A would get a danpu dumper injected via As constructor. Right now, that isn't possible (or rather it's possible but undesirable, because of the side effect).

Since new Export($foo) is the only public interface of Export it is equivalent to a function call, e.g. danpu_export($foo). Another way that stays in object oriented confines is moving the side effects to a dedicated public method.

Are you open to moving the side effects? It would require new major version, because it would break compatibility.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.