Git Product home page Git Product logo

postgrator-cli's Introduction

Postgrator CLI

Build Status npm package Coverage Status

Command line SQL database migration tool using SQL scripts. For PostgreSQL, MySQL and SQL Server.

Version control your SQL database using plain old SQL files.

Supports also undoing migrations.

Uses Postgrator node.js library developed by Rick Bergfalk.

Installation

As of postgrator-cli 6 Node.js version 14 or greater is required

npm install -g postgrator-cli

Or if you prefer to use it locally on your project using npm scripts of package.json:

npm install postgrator-cli --save-dev

And install the appropriate DB engine(s) if not installed yet:

npm install pg@8
npm install mysql@2
npm install mssql@6
npm install sqlite3@5

See the Postgrator documentation for more information about the supported engines.

Usage

SQL Files

Create a folder and stick some SQL scripts in there that change your database in some way. It might look like:

migrations/
  |- 001.do.sql
  |- 001.undo.sql
  |- 002.do.optional-description-of-script.sql
  |- 002.undo.optional-description-of-script.sql
  |- 003.do.sql
  |- 003.undo.sql
  |- 004.do.js
  |- 004.undo.js
  |- ... and so on

The files must follow the convention [version].[action].[optional-description].sql or [version].[action].[optional-description].js

Version must be a number, but you may start and increment the numbers in any way you'd like. If you choose to use a purely sequential numbering scheme instead of something based off a timestamp, you will find it helpful to start with 000s or some large number for file organization purposes.

Action must be either "do" or "undo". Do implements the version, and undo undoes it.

Optional-description can be a label or tag to help keep track of what happens inside the script. Descriptions should not contain periods.

SQL or JS You have a choice of either using a plain SQL file or you can also generate your SQL via a javascript module. The javascript module should export a function called generateSql() that returns back a string representing the SQL. For example:

module.exports.generateSql = function () {
  return (
    "CREATE USER transaction_user WITH PASSWORD '" +
    process.env.TRANSACTION_USER_PASSWORD +
    "'"
  )
}

You might want to choose the JS file approach, in order to make use (secret) environment variables such as the above.

The tool

You can specify all the parameters from command line (see below) but the easiest way is to:

  • Create .postgratorrc.json, or any config file supported by lilconfig. We also support .yml and .yaml files, and esm config files. For example:
{
    "migrationPattern": "migrations/*",
    "driver": "pg",
    "host": "127.0.0.1",
    "port": 5432,
    "database": "myDatabaseName",
    "username": "user",
    "password": "pass"
}

You can also set these settings with all ENV vars that node-postgres supports, though this may vary with your driver. (e.g. PGHOST PGHOST etc)

  • Migrate to latest version (it looks settings by default from .postgratorrc.json, etc):
$ postgrator
  • Migrate to version 004 (it knows current version and migrates up/down automatically):
$ postgrator 4

Synopsis

postgrator [[--to=]<version>] --database=<db> [--driver=<driver>] [--host=<host>] [--port=<port>] [--username=<username>] [--password=<password>] [--no-config]

postgrator [[--to=]<version>] [--config=<config>]

postgrator migrate [[--to=]version]

postgrator drop-schema [--config=<config>]

Options

  --to version                          Version number of the file to migrate to or 'max'. Default: 'max'
  -r, --driver pg|mysql|mssql|sqlite3   Database driver. Default: 'pg'.
  -h, --host hostname                   Host.
  -o, --port port                       Port.
  -d, --database database               Database name. (in the case of 'sqlite3' this is the filename, defaults to `:memory:`)
  -u, --username database               Username.
  -p, --password password               Password. If parameter without value is given, password will be asked.
  -m, --migration-pattern pattern       A pattern matching files to run migration files from. Default: 'migrations/*'
  -t --schema-table                     Table created to track schema version.
  --validate-checksum                   Validates checksum of existing SQL migration files already run prior to executing migrations.
  -s, --ssl                             Enables ssl connections. When using the mysql driver it expects a string containing name of ssl profile.
  -c, --config file                     Explicitly set the location of the config file to load.
  --no-config                           Do not load options from a configuration file.
  -v, --version                         Print version.
  -?, --help                            Print this usage guide.

Examples

  1. Specify parameters on command line                       postgrator 23 --host 127.0.0.1 --database sampledb
                                                              --username testuser --password testpassword
  2. Explicitly disable loading configuration file            postgrator 2 --no-config
  3. Use default configuration file to migrate to version 5   postgrator 5
  4. Migrate to latest version using default configuration    postgrator
  file (.postgratorrc.json, etc)
  5. Drop the schema table using configuration files          postgrator drop-schema

Tests

To run postgrator tests locally, run docker-compose up and then npm test.

postgrator-cli's People

Contributors

akanieski avatar bcomnes avatar dependabot[bot] avatar mattilehtinen avatar perrin4869 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

postgrator-cli's Issues

Provide connectionstring/dsn

Would be it be possible to support a single DSN argument rather than the host/db/username/pass vars?

(Might be more appropriate to open this issue at postgrator itself)

support pg@8?

Hi I am trying to run postgrator-cli with pg@8 and the cli returns me a message of Please raise issue to support pg > 7.x.x.

postgrator looks good to support pg@8 so I think the cli version should be ok too.

Default env vars?

Is there a set of default ENV vars postgrator-cli can look for when running migrations? I can pass them in easily as flags, but these tend to sneak into deployment logs which is less than ideal.

Add support for anvironment variables

Instead of configuration file or parameters, would be great if postgrator-cli supports environment variables, for example, for default "pg" engine they are PGUSER, PGHOST, PGPASSWORD, PGDATABASE
This will simplify a lot in multidocker environments.

Broken when running cli args with latest pg

With pg version 8.7.1, running postgrator without a postgrator.json file will error out:

TypeError [ERR_INVALID_ARG_TYPE]: The "string" argument must be of type string or an instance of Buffer or ArrayBuffer. Received an instance of Object

Downgrading to 8.2.1 solves the issue. I don't know which version introduces the change, but the issue is that pg no longer likes the options: { encrypt: boolean } option

CLI Warnings

Hi

Running postgrator (cli) with the example postgrator.json

{
  "migrationDirectory": "migrations",
  "driver": "pg",
  "host": "127.0.0.1",
  "port": 5433,
  "database": "postgres",
  "username": "postgres",
  "password": "postgres"
}

Lead to this output

postgrator deprecated Config option "port". Implement execQuery function instead. node_modules/postgrator-cli/postgrator-cli.js:222:26
postgrator deprecated Config option "host". Implement execQuery function instead. node_modules/postgrator-cli/postgrator-cli.js:222:26
postgrator deprecated Config option "username". Implement execQuery function instead. node_modules/postgrator-cli/postgrator-cli.js:222:26
postgrator deprecated Config option "password". Implement execQuery function instead. node_modules/postgrator-cli/postgrator-cli.js:222:26
postgrator deprecated Config option "migrationDirectory". use "migrationPattern" instead using glob match. e.g. path.join(__dirname, '/migrations/*') node_modules/postgrator-cli/postgrator-cli.js:222:26

The postgrator version is 4.3.1

I saw that these warnings have been introduced in 4.3.0
rickbergfalk/postgrator#131

Node 14 support - ERR_INVALID_ARG_TYPE error

The is the error on node 14 (possible also in node 12):

# postgrator 6 --host myhost --database mydb --username myuser --password mypass
buffer.js:728
    throw new ERR_INVALID_ARG_TYPE(
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "string" argument must be of type string or an instance of Buffer or ArrayBuffer. Received an instance of Object
    at Function.byteLength (buffer.js:728:11)
    at Writer.addCString (/usr/local/lib/node_modules/pg/node_modules/pg-protocol/dist/buffer-writer.js:41:30)
    at Object.startup (/usr/local/lib/node_modules/pg/node_modules/pg-protocol/dist/serializer.js:9:32)
    at Connection.startup (/usr/local/lib/node_modules/pg/lib/connection.js:124:33)
    at Connection.<anonymous> (/usr/local/lib/node_modules/pg/lib/client.js:121:13)
    at Connection.emit (events.js:315:20)
    at Socket.<anonymous> (/usr/local/lib/node_modules/pg/lib/connection.js:45:12)
    at Object.onceWrapper (events.js:421:28)
    at Socket.emit (events.js:327:22)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1132:10) {
  code: 'ERR_INVALID_ARG_TYPE'
}

# nodejs -v
v14.15.0

Possible issuer related to the one mentioned here: https://stackoverflow.com/a/62606989/2215679

Postgrator-cli not detecting changes to sql files

I have all my sql files in a sql folder. I started getting the following error in one of my sql files...
Error: Missing end comment mark '*/'.

Even though it should not be an error. I only have one /* and one */ in the entire file. I tried all sorts of things, and finally gave up. I deleted the entire contents of the file except...
use dbName
GO

I STILL get
Error: Missing end comment mark '*/'.

I've verified that I don't have those comments anywhere in any of my sql files, but I still get the error. I've tried changing the name of the files, etc, all to no avail. I always just get...
Error: Missing end comment mark '*/'.

Are the sql files cached somewhere? I would expect to be able to change a sql file that was giving an error, and have the change picked up and executed next time I run 'postgrator'.

command for generating do and undo files

Provide a command for generating the do and undo files automatically.

postgrator add create_users_table and automatically generates two files
201807051822.do.create_users_table.sql
201807051822.undo.create_users_table.sql

Add support for output from migration scripts

It would be nice if output from sql migration scripts would be printed in the terminal.
For example, currently when running postgrator 0020 the terminal output is as follows.

[7:01:25 PM] version of database is: 10
[7:01:25 PM] migrating up to 20
[7:01:25 PM] verifying checksum of migration 0010.do.create role admin_users.sql
[7:01:25 PM] running 0020.do.create role app_users.sql

In my 0020.do.create role app_users.sql file I have SELECT output like this (examples):

SELECT CONCAT(
   to_char(current_timestamp, '[HH12:MI:SS PM]'), ' Beginning script'
);
--do work
--do more work
SELECT CONCAT(
   to_char(current_timestamp, '[HH12:MI:SS PM]'), ' Beginning transaction'
);
--commit the transaction
SELECT CONCAT(
   to_char(current_timestamp, '[HH12:MI:SS PM]'), 'Transaction complete - 50 rows inserted'
);

It would be great if I could get this as output in the terminal (emphasizing the hypothetical last 3 lines):

[7:01:25 PM] version of database is: 10
[7:01:25 PM] migrating up to 20
[7:01:25 PM] verifying checksum of migration 0010.do.create role admin_users.sql
[7:01:25 PM] running 0020.do.create role app_users.sql
[7:01:25 PM] Beginning script
[7:01:28 PM] Beginning transaction
[7:02:06 PM] Transaction complete - 50 rows inserted

Can't use absolute directory path in -m option

It seems that current working directory is always prepended to the migrations directory provided by the -m option. It makes it harder to invoke this command from provision scripts (or any other scripts that can be executed from arbitrary directory).

Example:

postgrator -d mydatabase -m /mydb/migrations

I expect, that this command will run migrations from /mydb/migrations, but instead posgrator will look for migrations in /home/myuser/mydb/migrations if I run this command from /home/myuser.

Error: No migration files found from C:\dev\prototype\sqlmigrations\migrations

When I run "postgrator" from my c:\dev\prototype\sqlmigrations folder, I get the error in the title...

Error: No migration files found from C:\dev\prototype\sqlmigrations\migrations

I have postgrator-cli installed globally, and I have a postgrator.json sitting in sqlmigrations with the following contents...

{
    "migrationDirectory": "sql",
    "driver": "pg",
    "host": "(local)",
    "port": 5432,
    "database": "pdPrototyping",
    "username": "localuser",
    "password": "L0c4lp4ss"
}

I get the same error if I run
postgrator --config postgrator.json

It seems to be ignoring my postgrator.json file completely. I am putting together a proof of concept where there is nothing else in my sqlmigrations folder except the postgrator.json file, and my sql folder with a couple scripts in it.

Add option for single up and down migrations

Hello !

First off, thanks for this tool, I find it much easier to use and understand than others, and it's working well.

I was wondering if it would be possible to add a single up and down command, like this :

postgrator up

or something like

postgrator --migrationsToDo=1

The goal is to run migrations one at a time. A down version would be nice as well. It allows easier testing of migrations, as with those commands I can simply up and down quickly to test if my do/undo migrations are working (do and undo command line arguments could work as well).

This is also useful when migrating from one dev branch to the other on projects, so you can easily undo a migration on your test database before switching back to the main branch to test something else.

Thanks in advance for considering this feature.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.