crossroad0201 / flyway-awslambda Goto Github PK

View Code? Open in Web Editor NEW

31.0 8.0 22.0 42 KB

AWS Lambda function for RDS migration using Flyway.

License: MIT License

Scala 92.17% Shell 7.83%

aws rds flyway p

flyway-awslambda's Introduction

Flyway AWS Lambda function.

What's this?

Lambda function for AWS RDS Migration using Flyway.

EC2 instance is not necessary for DB migration.

This Lambda function is supporting 2 migration methods.

Automatic migration by put SQL file into S3.
Manual migration by invoking yourself. (Since 0.2.0)

Setup

S3 bucket

Create S3 bucket and folder for Flyway resources.

Bucket structure

s3://my-flyway             <- Flyway migration bucket.
  - /my-application        <- Flyway resource folder(prefix).
    - flyway.conf          <- Flyway configuration file.
    - V1__create_foo.sql   <- SQL file(s)
    - V2__create_bar.sql

Flyway configuration

create Flyway configuration file named flyway.conf in resource folder.

flyway.url = jdbc:mysql://RDS_ENDPOINT/DATABSE_NAME
flyway.user = USER_NAME
flyway.password = PASSWORD

Supported options.

See Flyway - Config Files for option details.

flyway.outOfOrder
flyway.schemas
flyway.cleanOnValidationError

AWS Settings

VPC Endpoint

Require VPC Endpoint for access to S3 Bucket from Lambda function in VPC.

Deploy Lambda function

Code

Download jar module from releases
- Or build from source.(Require JDK, Scala, sbt)

sbt assembly

Upload flyway-awslambda-x.x.x.jar.

Configuration

	value
Runtime	`Java 8`
Handler	See `Handler` section.
Role	See `Role` section.
Timeout	`5 min.`
VPC	Same VPC as target RDS.

Handler

crossroad0201.aws.flywaylambda.S3EventMigrationHandler
Run migration automatically when put SQL file into S3 bucket.
crossroad0201.aws.flywaylambda.InvokeMigrationHandler (Since 0.2.0)
Run migration invoke Lambda function yourself.

Role

Require policies.

AmazonRDSFullAccess
AmazonS3FullAccess
AmazonLambdaVPCAccessExecutionRole

Triggers

Require setting trigger S3 to Lambda if using S3EventMigrationHandler.

	value	Example
Bucket	Your Flyway migration bucket.	`my-flyway`
Event type	`Object created`	-
Prefix	Your Flyway migration files location.	`my-application/`
Suffix	`sql`	-

Setup by CloudFormation

You can setup flyway-awslambda automatically using CloudFormation.
See sample templates in src/main/aws.

flyway-awslambda-x.x.x.jar module put in your any bucket.
Create stack by template 1-rds.yaml. Create RDS Aurora cluster.
Create stack by template 2-flyway-awslambda.yaml. Create flyway-awslambda function.
Put flyway.conf configuration file in Flyway migration bucket.

Note

Require delete ENI(Elastic Network Interface) entry for VPC Lambda before delete stack 1-rds.yaml.
A ENI entry for VPC Lambda create by stack 2-flyway-awslambda.yaml, but this ENI entry does not delete automatically.

Run

Using S3EventMigrationHandler

Put Flyway SQL file into S3 resource folder.(one by one!!!)

Invoke flyway-lambda automatically by S3 event.

Check migration-result.json in S3 resource folder for result, and CloudWatch log for more detail.

Using InvokeMigrationHandler

Put Flyway SQL file(s) into S3 resource folder.

And invoke flyway-lambda function yourself with the following json payload. (invoke by AWS console, CLI, any application...etc. see CLI example)

{
  "bucket_name": "my-flyway",
  "prefix": "my-application",
  "flyway_conf": "flyway.conf"  // Optional. Default 'flyway.conf'
}

Check result message or migration-result.json in S3 resource folder for result, and CloudWatch log for more detail.

flyway-awslambda's People

Contributors

Stargazers

Watchers

Forkers

yoshiyoshifujii gregluk locerito raghukumarc tsumug mustafa-rizvi-zocdoc zocdoc integritus slopyjoe navodar22 mclavel jensonjoseph brandonsaccount hirokikonishi dayagalanikhil swagatamaiti songyuqing-cloud

flyway-awslambda's Issues

java.nio.file.NoSuchFileException occurred.

TART RequestId: c3d89469-b5ce-11e6-b14e-e103d2c39e6b Version: $LATEST
Flyway migration start. by ObjectCreated:Put s3://l*******/flyway/V1__create_table.sql
java.nio.file.NoSuchFileException: /tmp/c3d89469-b5ce-11e6-b14e-e103d2c39e6b/flyway/V1__create_table.sql
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at java.nio.file.Files.newOutputStream(Files.java:216)
at java.nio.file.Files.copy(Files.java:3016)
at crossroad0201.aws.flywaylambda.S3EventMigrationHandler.createSqlFile$1(S3EventMigrationHandler.scala:78)
at crossroad0201.aws.flywaylambda.S3EventMigrationHandler.deployInternal$1(S3EventMigrationHandler.scala:90)
at crossroad0201.aws.flywaylambda.S3EventMigrationHandler.$anonfun$migrate$1(S3EventMigrationHandler.scala:103)
at scala.util.Try$.apply(Try.scala:209)
at crossroad0201.aws.flywaylambda.S3EventMigrationHandler.deployFlywayResources$1(S3EventMigrationHandler.scala:58)
at crossroad0201.aws.flywaylambda.S3EventMigrationHandler.migrate(S3EventMigrationHandler.scala:142)
at crossroad0201.aws.flywaylambda.S3EventMigrationHandler.handleRequest(S3EventMigrationHandler.scala:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.handleRequest(EventHandlerLoader.java:456)
at lambdainternal.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:375)
at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:1140)
at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:285)
at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:57)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94)
END RequestId: c3d89469-b5ce-11e6-b14e-e103d2c39e6b
REPORT RequestId: c3d89469-b5ce-11e6-b14e-e103d2c39e6b	Duration: 11836.20 ms	Billed Duration: 11900 ms Memory Size: 1536 MB	Max Memory Used: 115 MB

lambda not working for mssql

Hello,
The lambda is not working with mssql. What changes is required to make it working?
Also, if there are place holders i.e. variables in sql script then how to pass it's value?

Licence type

What's the licence for the code inside this repository?

NullPointerException with repeatable scripts

Flyway has the ability to deploy repeatable scripts (by default with the prefix "R__"), which are deployed when the contents of the file has changed.

https://flywaydb.org/getstarted/repeatable

When a repeatable script is deployed, the record inserted in the schema_version table is null. I'm guessing that the null pointer exception is being thrown when the code is trying to get the migration info from the schema_version table and the version is unexpectedly null.

Below is a screen shot from my CloudWatch logs, the exception is being thrown after the deployment has proceeded successfully.

limit of number of files the flyway lambda can execute

I have around 2000 DB schema files. But flyway lambda fails with error saying cannot find flyway.conf file. But when i reduce the file count to below 200 in the S3, it successfully deploys.

so what should we do if more files present. Any suggestions on running the scripts in batches