Git Product home page Git Product logo

dbsc's Introduction

Complete documentation is available at https://github.com/LHCgreg/dbsc/wiki

dbsc (DataBase Source Control) is a command line tool for helping you create, recreate, and update your database. It works by running a series of numbered SQL scripts that you write, giving you access to the full power of SQL. It keeps track of what revision the database is on, allowing you to easily get your changes to the rest of your team and deploy updates to QA and production.

When creating or updating a database, dbsc allows you to import data from another database. When the database being created or updated reaches the revision that the other database is on, dbsc will clear the tables in the database being created or updated and import data for all tables from the other database. This makes it a snap to bring in test data.

Quick Introduction

Say a new developer joins your team. Your project uses a PostgreSQL database. Developers use a local database for testing and debugging. He checks out your project's code from source control. He opens a command prompt in a directory with 100 scripts named project_x.0000.sql, project_x.0001.sql, project_x.0002.sql, etc.

[mike@mikes_computer db_scripts]$ pgdbsc checkout -u mike
Creating database project_x on localhost.
Created database project_x on localhost.
Updating to r0
Updating to r1
Updating to r1
...
Updating to r100
At revision 100
[mike@mikes_computer db_scripts]$

To copy data from the QA database, Mike can run

[mike@mikes_computer db_scripts]$ pgdbsc checkout -u mike -sourceDbServer qa-pg.mycompany.local -sourceUsername mike

When his coworkers make database changes, Mike gets the latest code from source control and runs

[mike@mikes_computer db_scripts]$ pgdbsc update -u mike
Updating to r101
Updating to r102
At revision 102
[mike@mikes_computer db_scripts]$

Check out the tutorial for a more detailed introduction.

Installing

dbsc comes in different flavors for different database engines. dbsc currently supports

  • PostgreSQL (pgdbsc)
  • Microsoft SQL Server (msdbsc)
  • MySQL (mydbsc)
  • MongoDB (mongodbsc)

dbsc runs on Windows and (except for msdbsc) Linux and Mac OS X using Mono.

There are .exe installers and .zip archives on the releases page. The .exe installers will install the given flavor of dbsc along with anything it needs to run and will add the installed version of dbsc to your PATH environment variable. This is the recommended way of installing in Windows.

Packages for Debian and Debian-based Linux distributions (such as Ubuntu) are available.

mongodbsc depends on the Mongo shell to run scripts. You may wish to follow the instructions at http://docs.mongodb.org/manual/tutorial/install-mongodb-on-debian/ or http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/ to add the mongodb.org APT repository to your system as a package source. This way you will get the latest stable release from mongodb.org instead of the (likely old) version that your distro repository has.

wget -O - http://apt.dbsourcecontrol.org/keys/[email protected] | sudo apt-key add -
echo 'deb http://apt.dbsourcecontrol.org dbsc main' | sudo tee /etc/apt/sources.list.d/dbsc.list
sudo apt-get update
sudo apt-get install pgdbsc # for the PostgreSQL version
sudo apt-get install mydbsc # for the MySQL version
sudo apt-get install mongodbsc # for the MongoDB version

Packages for Red Hat, CentOS, and Fedora are also available. You must have a version of mono-core of at least 2.10.8 available. At the time of this writing (2014-04-25), only Fedora has a version that meets that requirement in the default repositories. On CentOS and Red Hat you will have to find a version of Mono released within the last 3 years somewhere.

mongodbsc depends on the Mongo shell to run scripts. To get the official mongodb.org package instead of packages from your RH-family distro, you may wish to follow the instructions at http://docs.mongodb.org/manual/tutorial/install-mongodb-on-red-hat-centos-or-fedora-linux/ and install the mongodb-org-shell package.

sudo yum-config-manager --add-repo http://rpm.dbsourcecontrol.org/dbsc.repo
sudo yum makecache
sudo yum install pgdbsc # for the PostgreSQL version
sudo yum install mydbsc # for the MySQL version
sudo yum install mongodbsc # for the MongoDB version

On other Linux distributions and Unix systems and Mac OS X you can download a .zip from the releases page. You will need to have mono 2.10.8 or higher installed and run "mono " or make a shell script to do that for you.

Getting help

You can get the available command-line options with

pgdbsc -h

or

msdbsc -h

or

mydbsc -h

or

mongodbsc -h

depending on which flavor you are using.

Changelog

msdbsc 2.1.0

  • When using -importTableList to import data for only certain tables, don't clear tables that are not being imported.

pgdbsc 2.1.0

  • When using -importTableList to import data for only certain tables, don't clear tables that are not being imported.

mydbsc 2.1.0

  • When using -importTableList to import data for only certain tables, don't clear tables that are not being imported.

mongodbsc 2.1.0

  • When using -importTableList to import data for only certain tables, don't clear tables that are not being imported.

dbsc's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

nkabir

dbsc's Issues

Mac packages

I don't have a Mac to test on but fpm supports creating Mac packages. Maybe I can find someone to test for me.

Automated tests

Write an automated integration test suite for all 3 flavors.

Use a separate connection per script

Scripts could do things that affect connection state. For example, using SET in SQL Server to set options such as XACT_ABORT, QUOTED_IDENTIFIER, etc. Use a separate connection for each script to ensure each script gets a fresh environment.

Give assembly file name without ".exe" in message

Error: No operation specified. pgdbsc.exe -h for help.

On Linux with the (in progress) package, dbsc is invoked with a wrapper script which is named pgdbsc, mydbsc, or mongodbsc. Modify the above message to not include the ".exe".

DBSC for MongoDB

MongoDB may be schemaless but you still need to add indexes, add static data, migrate data from one "schema" to another, etc.

Scripts would be js instead of sql.

DBSC goes web scale!

Use COPY FREEZE for imports on PostgreSQL versions that support it

PostgreSQL 9.3 introduced a FREEZE option to the COPY command. COPY FREEZE does not speed up the import itself but makes operations after the import faster.

It does violate visibility rules by making the data visible to other transactions before the import transaction is committed, but that's not an issue for imports as you would not be importing data into a database in use.

Don't show postgres noise messages

Don't show notices that PostgreSQL displays when creating implicit indices on primary keys or creating implicit sequences for serial columns. They look like:

NOTICE: CREATE TABLE will create implicit sequence "mal_anime_synonym_mal_anime_
synonym_id_seq" for serial column "mal_anime_synonym.mal_anime_synonym_id"
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "mal_anime_synonym
_pkey" for table "mal_anime_synonym"

They got bumped down to DEBUG1 level instead of NOTICE in PostgreSQL 9.3 (http://www.postgresql.org/docs/devel/static/release-9-3.html) but it will be quite some time before most people will be running 9.3.

Check if snapshot transactions are enabled and use for imports if enabled

Doing an import of a SQL Server database that is in use can result in consistency issues or failures when re-enabling foreign keys after the import.

To allow for consistent imports of databases that are in use, check if snapshot isolation (http://msdn.microsoft.com/en-us/library/ms345124%28v=sql.90%29.aspx) is enabled on the source database:

SELECT snapshot_isolation_state_desc from sys.databases
where name='MyDatabase'

If snapshot isolation is enabled, wrap the entire import operation in a snapshot transaction on the source database.

Also document this behavior.

Don't use sp_msforeachtable

sp_msforeachtable is undocumented and is subject to quirks such as QUOTED_IDENTIFIER being OFF for SQL executed by it.

Cannot do anything other than create database in create db template for PostgreSQL

Trying to do anything other than the CREATE DATABASE statement in a create DB template in pgdbsc gives the error

Error: ERROR: 25001: CREATE DATABASE cannot be executed from a function or multi-command string

You would not be able to do much else interesting like setting permissions anyway because you would have to be connected to the target database instead of the "postgres" database.

Maybe have a command line parameter to specify a post-create script? Or the ability to specify an environment when creating the database and have scripts that only run for certain environments?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.