Git Product home page Git Product logo

dbops's Introduction

master development
Build status Build status

DBOps

dbops

DBOps is a Powershell module that provides Continuous Integration/Continuous Deployment capabilities for SQL database deployments. In addition to easy-to-use deployment functions, it provides tracking functionality, ensuring that each script is deployed only once and in due order. It will also grant you with ability to organize scripts into builds and deploy them in a repeatable manner on top of any previously deployed version.

The deployment functionality of the module is provided by DbUp .Net library, which has proven its flexibility and reliability during deployments.

Currently supported RDBMS:

  • SQL Server
  • Oracle
  • PostgreSQL
  • MySQL

In addition it also supports Azure Synapse Analytics SQL Pools (both dedicated and serverless). Be aware that you cannot use the schemaversion table with the serverless SQL Pool - it needs to be explicitly disabled by setting SchemaVersionTable to $null. All scripts deployed in such scenario should be idempotent and re-runnable.

Features

The most notable features of the module:

  • Reliably deploy your scripts in a consistent and repeatable manner
  • Perform ad-hoc deployments with highly customizable deployment parameters
  • Run ad-hoc queries to any supported RDBMS on both Windows and Linux
  • Create ready-to-deploy versioned packages in a single command
  • Brings along all features of CI/CD pipelining functionality: builds, artifact management, deployment
  • Roll back the script (or a whole deployment!) in case of errors
  • Dynamically change your code based on custom variables using #{customVarName} tokens

System requirements

  • Powershell 5.0 or higher

Installation

Using git

git clone https://github.com/dataplat/dbops.git dbops
Import-Module .\dbops

Make sure to have the following modules installed as well:

Using PSGallery (Powershell 5+)

Install-Module dbops

Usage scenarios

  • Ad-hoc deployments of any scale without manual code execution
  • Delivering new version of the database schema in a consistent manner to multiple environments
  • Build/Test/Deploy scenarios inside the Continuous Integration/Continuous Delivery pipeline
  • Dynamic deployment based on modified files in the source folder
  • Versioned package deployment (e.g. Octopus Deployment)

Examples

Simple deployments and ad-hoc queries

Perform plain-text script deployments using a single command:

Install-DBOScript
(click to open the video)

Example code:

# Ad-hoc deployment of the scripts from a folder myscripts
Install-DBOScript -ScriptPath C:\temp\myscripts -SqlInstance server1 -Database MyDB

# Execute a list of files as an Ad-hoc query
Get-ChildItem C:\temp\myscripts | Invoke-DBOQuery -SqlInstance server1 -Database MyDB

Package management

dbops packages

Each package consists of multiple builds and can be easily deployed to the database, ensuring that each build is deployed in proper order and only once.

Add-DBOBuild
(click to open the video)

Example code:

# Deployment using packaging system
New-DBOPackage Deploy.zip -ScriptPath C:\temp\myscripts | Install-DBOPackage -SqlInstance server1 -Database MyDB

# Create new deployment package with predefined configuration and deploy it replacing #{dbName} tokens with corresponding values
New-DBOPackage -Path MyPackage.zip -ScriptPath .\Scripts -Configuration @{ Database = '#{dbName}'; ConnectionTimeout = 5 }
Install-DBOPackage MyPackage.zip -Variables @{ dbName = 'myDB' }

# Adding builds to the package
Add-DBOBuild Deploy.zip -ScriptPath .\myscripts -Type Unique -Build 2.0
Get-ChildItem .\myscripts | Add-DBOBuild Deploy.zip -Type New,Modified -Build 3.0

# Install package using internal script Deploy.ps1 - to use when module is not installed locally
Expand-Archive Deploy.zip '.\MyTempFolder'
.\MyTempFolder\Deploy.ps1 -SqlInstance server1 -Database MyDB

Configurations and defaults

There are multiple configuration options available, including:

  • Configuring default settings
  • Specifying runtime parameters
  • Using configuration files

Get-DBOConfig
(click to open the video)

Example code:

# Setting deployment options within the package to be able to deploy it without specifying options
Update-DBOConfig Deploy.zip -Configuration @{ DeploymentMethod = 'SingleTransaction'; SqlInstance = 'localhost'; Database = 'MyDb2' }
Install-DBOPackage Deploy.zip

# Generating config files and using it later as a deployment template
New-DBOConfig -Configuration @{ DeploymentMethod = 'SingleTransaction'; SqlInstance = 'devInstance'; Database = 'MyDB' } | Export-DBOConfig '.\dev.json'
Get-DBOConfig -Path '.\dev.json' -Configuration @{ SqlInstance = 'prodInstance' } | Export-DBOConfig '.\prod.json'
Install-DBOPackage Deploy.zip -ConfigurationFile .\dev.json

# Invoke package deployment using custom connection string
Install-DBOPackage -Path Deploy.zip -ConnectionString 'Server=myServerAddress;Database=myDataBase;Trusted_Connection=True;'

# Invoke package deployment to an Oracle database OracleDB
Install-DBOPackage -Path Deploy.zip -Server OracleDB -ConnectionType Oracle

# Get a list of all the default settings
Get-DBODefaultSetting

# Change the default SchemaVersionTable setting to null, disabling the deployment journalling by default
Set-DBODefaultSetting -Name SchemaVersionTable -Value $null

# Reset SchemaVersionTable setting back to its default value
Reset-DBODefaultSetting -Name SchemaVersionTable

CI/CD features

dbops CI/CD flow assumes that each package version is built only once and deployed onto every single environment. The successfull builds should make their way as artifacts into the artifact storage, from which they would be pulled again to add new builds into the package during the next iteration.

CI-CD flow

CI/CD capabilities of the module enable user to integrate SQL scripts into a package file using a single command and to store packages in a versioned package repository.

Invoke-DBOPackageCI
(click to open the video)

Example code:

# Invoke CI/CD build of the package MyPackage.zip using scripts from the source folder .\Scripts
# Each execution of the command will only pick up new files from the ScriptPath folder
Invoke-DBOPackageCI -Path MyPackage.zip -ScriptPath .\Scripts -Version 1.0

# Store the package in a DBOps package repository in a folder \\data\repo
Publish-DBOPackageArtifact -Path myPackage.zip -Repository \\data\repo

# Retrieve the latest package version from the repository and install it
Get-DBOPackageArtifact -Path myPackage.zip -Repository \\data\repo | Install-DBOPackage -Server MyDBServer -Database MyDB

Planned for future releases

  • Code analysis: know what kind of code makes its way into the package. Will find hidden sysadmin grants, USE statements and other undesired statements
  • Support for other RDBMS (eventually, everything that DbUp libraries can talk with)
  • Integration with unit tests (tSQLt/Pester/...?)
  • Module for Ansible (right now can still be used as a powershell task)
  • SQLCMD support
  • Deployments to multiple databases at once
  • Optional rollback scripts

Contacts

Submitting issues - GitHub issues

SQLCommunity Slack: https://sqlcommunity.slack.com #dbops or @nvarscar

dbops's People

Contributors

kevchant avatar kk-everbridge avatar lowlydba avatar nvarscar avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dbops's Issues

Improve output from Invoke-DBODeployment

Should return more than a simple object returned from DbUp. Should probably be a custom class that would be accepted through a pipeline by future helper functions: send emails, invoke tSQLt tests, PowerBI reports, etc.

variables not being replaced in Install-DBOPackage/SqlScript

SQL code:

INSERT INTO [dbo].[Course] ([Title], [Credits])
     VALUES ( N'#{Title}$Title$#{Title}#', #{Credit} )
GO

Powershell:

New-DBOPackage -Path C:\Temp\dbops_test.zip -ScriptPath C:\Temp\dbops_test -Force;
Add-DBOBuild -Path C:\Temp\dbops_test.zip -ScriptPath C:\Temp\dbops_test -Type Unique -Build 1.0.4
$pkg = Get-DBOPackage -Path C:\Temp\dbops_test.zip
$pkg | Install-DBOPackage -Build 1.0.4 -SqlInstance '.\SQL2017' -Database Test1 -CreateDatabase -Variables @{Title='hello';Credit='2'}

The columns with DBNull datatype fail to be converted

Cannot convert argument "type", with value: "", for "Add" to type "System.Type": "Cannot convert value "" to type "System.Type". Error: "Invalid cast from 'System.DBNull' to 'System.Type'.""
At /Users/kkravtsov/.local/share/powershell/Modules/dbops/0.5.4/functions/Invoke-DBOQuery.ps1:364 char:41
+ ...                          $null = $table.Columns.Add($name, $datatype)
+                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [], MethodException
+ FullyQualifiedErrorId : MethodArgumentConversionInvalidCastArgument

Package path is corrupted when using -Absolute

New-DBOPackage -Path C:\Temp\dbops_test_v00.zip -ScriptPath C:\Temp\dbops_test_v14 -Force -Absolute;
Get-DBOPackage -Path C:\Temp\dbops_test_v00.zip | Install-DBOPackage -SqlInstance '.\SQL2017' -Database Test1 -CreateDatabase 
Name                 Version   LastWriteTime            Builds                  
----                 -------   -------------            ------                  
dbops_test_v02.zip   2019.0... 30.8.2019 15.22.06       {[2019.08.30.152205]}   
File not found inside the package: content\2019.08.30.152205\C:\Temp\dbops_test_v14\00100_TIL_TAPAHTUMA.sql
At C:\Users\xxx\Documents\WindowsPowerShell\Modules\PSFramework\1.0.35\PSFramework.psm1:3141 char:23
+         if (-not $Cmdlet) { throw $records[0] }
+                             ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (C:\Temp\dbops_test_v02.zip:DBOpsPackage) [], Exception
    + FullyQualifiedErrorId : dbops_DBOpsPackage

New feature - packages without build version tracking

Packages that can contain one (or maybe more than one) build version, however, only the latest build is ever deployed. Build version is never recorded to the schemaversions table: every script is stored without a build prefix.
Pretty much, your regular Install-DBOScript deployment from a folder, but stored as a package.

I'm not sure this one is really needed yet, as similar behavior could be achieved by using the same exact build version every time when such package is getting created, but we'll see.

Ideas so far:

  • A new switch for New-DBOPackage (something like -Simple or -SingleBuild) that enables that behavior and enforces the version-less deployment history.
  • Single-build packages seem to make more sense to avoid confusion during the deployment.

New commands (Un)Register-DBODeployment

Allows to register/unregister scripts in the schema versions table to fine tune the deployment.
Should accept 2 parameter sets:

  • -Package $pkg -Build 1.0
  • -ScriptName "1.0\Folder\Script.sql"
    and use ad-hoc sql executor to add/remove entries to the schema versions table

New independent command for script deployment

Invoke-DBODeployment should become a fully internal command eventually, only accepting [DBOpsConfig] instead of all the variety of the configuration items. Wrapper functions Install-DBOPackage and Install-DBOScript (???) would prepare configuration objects and call Invoke-DBODeployment using one of the two available parameter sets: Package or Script.

Oracle tests hang when query timeout is reached

            $content = '
                DECLARE
                    in_time number := 3;
                BEGIN
                    DBMS_LOCK.sleep(in_time);
                END;'
            $file = Join-PSFPath -Normalize "$workFolder\delay.sql"
            $content | Set-Content $file
            $null = Install-DBOSqlScript -ScriptPath "$workFolder\delay.sql" @connParams -ExecutionTimeout 2

Add support for pre- and post-scripts

Add Pre- and -Post scripts to the packages and possibly to builds.

New-DBOPackage -ScriptPath .\foo -PreScriptPath .\bar -PostScriptPath .\foobar

New feature - retrieving the package file from the repo

Currently Get-DBOPackageArtifact only returns the path to the file. CI build should be done in a local folder, so there is a need for a convenient way of retrieving that file using single command. Copy-DBOPackageArtifact?

Unable to use SaveToFile on Get-DboConfig

First off, this is looking amazing already and I can't wait to start using it as an easy alternative to Flyway! Thanks for the work so far on a very much needed utility :)

dbops Version: 0.3.4
PS Version: 5.1.14393.2368
Issue: When attempting to export a config file, this is the error returned:

PS C:\> (Get-DBOConfig -Configuration @{ DeploymentMethod = 'SingleTransaction'; Schema = 'dbo'}).SaveToFile('test.json')
Method invocation failed because [DBOpsConfig] does not contain a method named 'SaveToFile'.
At line:1 char:1
+ (Get-DBOConfig -Configuration @{ DeploymentMethod = 'SingleTransactio ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : MethodNotFound

-Match filters out both files and folders

Expected behaviour:
New-DBOPackage ... -Match '\.sql$' includes files with the extension ".sql" from all subfolders.
Actual behavior:
It only includes files from the top folder. Recursive folders won't make the cut because they do not match the pattern.

Invoke-DBOQuery - unhandled datatype aclitem[]

Summary

Using Invoke-DBOQuery, queries returning non-null data of type aclitem[] (for example pg_database.datacl) throw an error. This data should be gracefully returned

Repro

Given Set-DBODefaultSetting -Name rdbms.Type -Value PostgreSQL,

PS > Invoke-DBOQuery 'select cast(null as aclitem[]);'                                                          

aclitem
-------


PS > Invoke-DBOQuery "select cast(null as aclitem[]), cast('{=c/postgres}' as aclitem[]);" -Verbose              
VERBOSE: [17:50:25][Initialize-ExternalLibrary] Npgsql was found among the loaded libraries, assuming that the library is fully loaded
VERBOSE: [17:50:25][Invoke-DBOQuery] Establishing connection with PostgreSQL localhost
VERBOSE: Performing the operation "Executing query select cast(null as aclitem[]), cast('{=c/postgres}' as aclitem[]);" on target "localhost".
VERBOSE: [17:50:25][Invoke-DBOQuery] Executing sub-query select cast(null as aclitem[]), cast('{=c/postgres}' as aclitem[]);
[17:50:25][Invoke-DBOQuery] 42883: no binary output function available for type aclitem
Exception calling "GetResult" with "0" argument(s): "42883: no binary output function available for type aclitem"
At /Users/pvandivier/.local/share/powershell/Modules/PSFramework/1.1.59/PSFramework.psm1:3304 char:23
+         if (-not $Cmdlet) { throw $records[0] }
+                             ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], Exception
    + FullyQualifiedErrorId : dbops_Invoke-DBOQuery
PS > 

Full Error

PS > Invoke-DBOQuery "select cast(null as aclitem[]), cast('{=c/postgres}' as aclitem[]);"
[17:53:06][Invoke-DBOQuery] 42883: no binary output function available for type aclitem
Exception calling "GetResult" with "0" argument(s): "42883: no binary output function available for type aclitem"
At /Users/pvandivier/.local/share/powershell/Modules/PSFramework/1.1.59/PSFramework.psm1:3304 char:23
+         if (-not $Cmdlet) { throw $records[0] }
+                             ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], Exception
    + FullyQualifiedErrorId : dbops_Invoke-DBOQuery
PS > $error[0] | select *         

PSMessageDetails      : 
Exception             : System.Exception: Exception calling "GetResult" with "0" argument(s): "42883: no binary output function available for type aclitem" ---> 
                        System.Management.Automation.MethodInvocationException: Exception calling "GetResult" with "0" argument(s): "42883: no binary output function available for type 
                        aclitem" ---> Npgsql.PostgresException: 42883: no binary output function available for type aclitem
                           at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.cs:line 1034
                        --- End of stack trace from previous location where exception was thrown ---
                           at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.cs:line 1034
                        --- End of stack trace from previous location where exception was thrown ---
                           at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming) in C:\projects\npgsql\src\Npgsql\NpgsqlDataReader.cs:line 526
                           at Npgsql.NpgsqlCommand.ExecuteDbDataReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) in 
                        C:\projects\npgsql\src\Npgsql\NpgsqlCommand.cs:line 1231
                           at CallSite.Target(Closure , CallSite , Object )
                           --- End of inner exception stack trace ---
                           at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
                           at lambda_method(Closure , Object[] , StrongBox`1[] , InterpretedFrame )
                           --- End of inner exception stack trace ---
TargetObject          : 
CategoryInfo          : NotSpecified: (:) [], Exception
FullyQualifiedErrorId : dbops_Invoke-DBOQuery
ErrorDetails          : 
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at Stop-PSFFunction, /Users/pvandivier/.local/share/powershell/Modules/PSFramework/1.1.59/PSFramework.psm1: line 3304
                        at Invoke-DBOQuery<Process>, /Users/pvandivier/repos/git/dbops/functions/Invoke-DBOQuery.ps1: line 386
                        at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}


PS > 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.