Git Product home page Git Product logo

mysqlsuperdump's Introduction

MySQL Super Dump

Build Status Go Report Card

MySQL Super Dump is a tool to efficiently create filtered and manipulated database dumps. It relies in the power of the SQL native language to do this, using WHERE clauses and complete SELECT statements with aliases to do this.

Currently it does not support every kind of MySQL structure (views, triggers, etc), but it supports the most basic stuff: schemas, tables and rows.

History

Once uppon a time there was a development team that liked to use dumps from the production database in their development environments to have the same content and behavior of the production system in their machines.

To avoid security problems, the system administrator created a script to dump the production database, import in a temporary database, then replace all sensitive data, like salts, passwords, customer names, emails, etc, for fake data, then export a dump of this temporary database to a file that is the dump developers would use.

However this script was taking more time to run, day by day, and each day it was using more resources from the server to run, until it exploded.

Features

  • Filter dumped rows by a native WHERE clause ([where] config's section)
  • Replace dumped data with native SELECT functions ([select] config's section)
  • Disable data output of specific tables ([filter] config's section: nodata)
  • Ignore entire tables ([filter] config's section: ignore)

Usage

  • Install the latest Go compiler installed (check instructions at: http://golang.org)
  • Check you environment with go env:
  • The repository will be clones at $GOPATH/src/github.com/hgfischer/mysqlsuperdump
  • The binary will be installed in $GOBIN
  • Then run go get to download, build and install mysqlsuperdump: go get github.com/hgfischer/mysqlsuperdump
  • Create a config file based on example.cfg and place where you like it.
  • Run mysqlsuperdump -h to see command line options and voilá.

Configuration Example


[mysql]
# See https://github.com/Go-SQL-Driver/MySQL for details on this
dsn = username:password@protocol(address)/dbname?charset=utf8
extended_insert_rows = 1000
#use_table_lock = true
max_open_conns = 50

# Use this to restrict exported data. These are optional
[where]
sales_order           = created_at >= DATE_SUB(NOW(), INTERVAL 7 DAY)
customer_upload       = created_at >= DATE_SUB(NOW(), INTERVAL 7 DAY)
newsletter_subscriber = created_at >= DATE_SUB(NOW(), INTERVAL 7 DAY)

# Use this to override value returned from tables. These are optional
[select]
system_user.salt = 'reset salt of all system users'
system_user.password = 'reset password of all system users'

customer.first_name = CONCAT('Charlie ', id)
customer.last_name = 'Last'
customer.salt = 'reset salt of all customers'
customer.password = 'reset password of all customers'
customer.username = CONCAT(id, '@fiction.tld')
customer.username_canonical = CONCAT(id, '@fiction.tld')
customer.email = CONCAT(id, '@fiction.tld')
customer.email_canonical = CONCAT(id, '@fiction.tld')

newsletter_subscriber.email = CONCAT(id, '@fiction.tld')

customer_address.recipient_name = CONCAT('Recipient Name ', id)
customer_address.company = CONCAT('Company Name ', id)
customer_address.phone = CONCAT('(', id, ') 1234-1234')

sales_order_address.recipient_name = CONCAT('Recipient Name ', id)
sales_order_address.company = CONCAT('Company Name ', id)
sales_order_address.phone = CONCAT('(', id, ') 1234-1234')

system_dump_version.created_at = NOW()

# Use this to filter entire table (ignore) or data only (nodata)
[filter]
customer_stats = nodata
customer_private = ignore

TO DO

  • Extend MySQL support, with other objects like views, triggers, etc
  • Refactor dumper interface to support another SQL databases
  • Add support for PostgreSQL

License

Please, check the LICENSE file.

mysqlsuperdump's People

Contributors

hgfischer avatar kslimani avatar nathanwar avatar andreas-bergstrom avatar googilyboogily avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.