Git Product home page Git Product logo

stackup's Introduction

stackup

Gem Version Build Status

Stackup provides a CLI and a simplified Ruby API for dealing with AWS CloudFormation stacks.

Why?

Stackup provides some advantages over using awscli or aws-sdk directly:

  • It treats stack changes as synchronous, streaming stack events until the stack reaches a stable state.

  • A Stack#up facade for create/update frees you from having to know whether your stack already exists or not.

  • Changes are (mostly) idempotent: "no-op" operations - e.g. deleting a stack that doesn't exist, or updating without a template change - are handled gracefully (i.e. without error).

Installation

$ gem install stackup

Command-line usage

The entry-point is the "stackup" command.

Most commands operate in the context of a named stack:

$ stackup STACK-NAME ...

Called with --list, it will list stacks:

$ stackup --list
foo-bar-test
zzz-production

Stack create/update

Use sub-command "up" to create or update a stack, as appropriate:

$ stackup myapp-test up -t template.json

This will:

  • update (or create) the named CloudFormation stack, using the specified template
  • monitor events until the stack update is complete

Requests will retry 3 times by default. After this limit is exceeded, ERROR: Rate exceeded failures will be logged. You can increase the limit using the --retry-limit option, or by setting the $AWS_API_RETRY_LIMIT environment variable.

For more details on usage, see

$ stackup STACK up --help

Specifying parameters

Stack parameters can be read from a file, e.g.

$ stackup myapp-test up -t template.json -p parameters.json

These files can be either JSON or YAML format, see YAML support for more information.

Parameters can be specified as simple key-value pairs:

{
  "IndexDoc": "index.html"
}

but also supports the extended JSON format used by the AWS CLI:

[
  {
    "ParameterKey": "IndexDoc",
    "ParameterValue": "index.html",
    "UsePreviousValue": false
  }
]

You may specify -p multiple times; stackup will read and merge all the files:

$ stackup myapp-test up -t template.json \
  -p defaults.json \
  -p overrides.json

Or, you can specify one or more override parameters on the command-line, using -o with -p:

$ stackup myapp-test up -t template.json \
  -p defaults.json \
  -o IndexDoc=index-override.html
  -o ContentDoc=content-override.html

Specifying tags

Stack tags can be read from a file, e.g.

$ stackup myapp-test up -t template.json --tags tags.json

These files can be either JSON or YAML format, see YAML support for more information.

Tags are specified as simple key-value pairs:

{
  "environment": "dev"
}

Acknowledging Capabilities

CloudFormation requires that some stacks explicitly acknowledge certain capabilities before creation. This helps to prevent the creation of stacks with unintended privileges.

If your stack includes IAM resources, you must specify either the CAPABILITY_IAM capability, or the CAPABILITY_NAMED_IAM capability if they have custom names.

If your stack template contains macros or nested stacks, you must specify the CAPABILITY_AUTO_EXPAND capability.

Capabilities can be provided via the --capability CLI option.

$ stackup myapp-test up -t template.json \
  --capability CAPABILITY_NAMED_IAM \
  --capability CAPABILITY_AUTO_EXPAND

stackup includes defaults to including CAPABILITY_NAMED_IAM capability if, and only if, no capabilities are specified. This is to provide backwards compatibility with previously deployed stacks and may be removed in a future release.

YAML support

stackup supports input files (template, parameters, tags) in YAML format, as well as JSON.

It also supports the abbreviated YAML syntax for Cloudformation functions, though unlike the AWS CLI, Stackup (by default) normalises YAML input to JSON before invoking CloudFormation APIs.

If you don't want normalisation of the YAML input to JSON, then use the --preserve-template-formatting flag to the up or change-set create commands.

Note: normalisation of S3 / HTTP URL stored templates is never done, as Cloudformation collects these directly.

AWS credentials

The stackup command-line looks for AWS credentials in the standard environment variables.

You can also use the --with-role option to temporarily assume a different IAM role, for stack operations:

$ stackup myapp-test up -t template.json \
  --with-role arn:aws:iam::862905684840:role/deployment

You can use the --service-role-arn option to pass a specific IAM service role for CloudFormation to use for stack operations:

$ stackup myapp-test up -t template.json \
    --service-role-arn arn:aws:iam::862905684840:role/cloudformation-role

(for more information on CloudFormation service roles, see AWS' documentation).

Using URLs as inputs

You can use either local files, or HTTP URLs, to specify inputs; stack template, parameters, etc.

$ stackup mystack up \
  -t https://s3.amazonaws.com/mybucket/stack-template.json

Where a template URL references an object in S3, stackup leverages CloudFormation's native support for such URLs, enabling use of much larger templates.

Non-S3 URLs are also supported, though in that case stackup must fetch the content itself:

$ stackup mystack up \
  -t https://raw.githubusercontent.com/realestate-com-au/stackup/main/examples/template.yml

Stack deletion

Sub-command "delete" deletes the stack.

Stack inspection

Inspect details of a stack with:

$ stackup myapp-test status
$ stackup myapp-test resources
$ stackup myapp-test outputs

Change-set support

You can also create, list, inspect, apply and delete change sets using stackup.

$ stackup myapp-test change-sets
$ stackup myapp-test change-set create -t template.json
$ stackup myapp-test change-set inspect
$ stackup myapp-test change-set apply

The change-set name defaults to "pending", but can be overridden using --name.

The change-set create subcommand, like the up command, supports --service-role-arn to specify a service role.

It is impossible to create a change set with no changes. By default, stackup will only return successfully if a change set was actually created, and will otherwise fail. If the --no-fail-on-empty-change-set option is provided, stackup will return successfully if a change set was created or if no change set was created because no changes were needed.

Programmatic usage

Get a handle to a Stack object as follows:

stack = Stackup.stack("my-stack")

You can pass an Aws::CloudFormation::Client, or client config, to Stackup, e.g.

stack = Stackup(credentials).stack("my-stack")

See {Stackup::Stack} for more details.

Rake integration

Stackup integrates with Rake to generate handy tasks for managing a stack, e.g.

require "stackup/rake_tasks"

Stackup::RakeTasks.new("app") do |t|
  t.stack = "my-app"
  t.template = "app-template.json"
end

providing tasks:

rake app:diff       # Show pending changes to my-app stack
rake app:down       # Delete my-app stack
rake app:inspect    # Show my-app stack outputs and resources
rake app:up         # Update my-app stack

Parameters and tags may be specified via files, or as a Hash, e.g.

Stackup::RakeTasks.new("app") do |t|
  t.stack = "my-app"
  t.template = "app-template.json"
  t.parameters = "production-params.json"
  t.tags = { "environment" => "production" }
end

Docker image

Stackup is also published as a Docker image. Basic usage is:

docker run --rm \
    -v "`pwd`:/cwd" \
    -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \
    -e AWS_DEFAULT_REGION \
    realestate/stackup:latest ...

If you're sensible, you'll replace "latest", with a specific version.

The default working-directory within the container is /cwd; hence the volume mount to make files available from the host system.

IAM Permissions

up

This policy grants the principal all actions required by stackup up for any cloudformation stack:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudformation:CreateStack",
                "cloudformation:DeleteStack",
                "cloudformation:DescribeStackEvents",
                "cloudformation:DescribeStackResource",
                "cloudformation:DescribeStacks",
                "cloudformation:SetStackPolicy",
                "cloudformation:UpdateStack"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Development

Running tests

auto/test will run the tests in a Docker container. auto/lint will run the linter in a Docker container.

Releasing

Releasing is done mostly by CI. The automated process will push tags to GitHub, push the docker images to DockerHub. It won't yet push the gem to rubygems.org, but we're working ok it..

Prerequisites:

  • You must have a rubygems account with permission to push to the stackup gem. (auto/release-gem will ask for your username and password)
  • You have bumped the version number in lib/stackup/version.rb.
  • You have checked the CHANGES.md file to make sure it is reasonable and has the changes for this new version.

To release the rubygem.

  1. auto/release-gem
  2. On REA's internal CI tool of choice, find the build for stackup-ci and trigger a new build from HEAD. This will always grab the latest code from here and release the rest of the parts.

stackup's People

Contributors

ahmed-shash-rea avatar aidansteele avatar amcinnes avatar djmattyg007 avatar jitran avatar jnewbigin avatar katherinelim avatar kiefmaster avatar kunday avatar logan-han avatar lukeck avatar mbelekar avatar mdub avatar mukaibot avatar ps-jay avatar stevenringo avatar stiak avatar thetaylorhicks avatar tigris avatar toothbrush avatar wilvk avatar zadowsmash avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

stackup's Issues

Tags as AWS style array of hashes doesn't seem to work for me

With a .json file of tags in AWS format ... I get errors from stackup some-stack up ...:

Tags file:

[
  {
    "Key": "Name",
    "Value": "some-stack"
  },
  {
    "Key": "Environment",
    "Value": "test"
  },
  {
    "Key": "Owner",
    "Value": "Delivery Engineering"
  },
  {
    "Key": "Criticality",
    "Value": "Low"
  },
  {
    "Key": "CostCentre",
    "Value": "1000"
  },
  {
    "Key": "PublicResource",
    "Value": "Yes"
  }
]

Errors:

/usr/local/bundle/gems/aws-sdk-core-3.68.1/lib/aws-sdk-core/param_validator.rb:33:in `validate!': parameter validator found 24 errors: (ArgumentError)
  - missing required parameter params[:tags][0][:key]
  - missing required parameter params[:tags][0][:value]
  - unexpected value at params[:tags][0]["Key"]
  - unexpected value at params[:tags][0]["Value"]
  - missing required parameter params[:tags][1][:key]
  - missing required parameter params[:tags][1][:value]
  - unexpected value at params[:tags][1]["Key"]
  - unexpected value at params[:tags][1]["Value"]
  - missing required parameter params[:tags][2][:key]
  - missing required parameter params[:tags][2][:value]
  - unexpected value at params[:tags][2]["Key"]
  - unexpected value at params[:tags][2]["Value"]
  - missing required parameter params[:tags][3][:key]
  - missing required parameter params[:tags][3][:value]
  - unexpected value at params[:tags][3]["Key"]
  - unexpected value at params[:tags][3]["Value"]
  - missing required parameter params[:tags][4][:key]
  - missing required parameter params[:tags][4][:value]
  - unexpected value at params[:tags][4]["Key"]
  - unexpected value at params[:tags][4]["Value"]
  - missing required parameter params[:tags][5][:key]
  - missing required parameter params[:tags][5][:value]
  - unexpected value at params[:tags][5]["Key"]
  - unexpected value at params[:tags][5]["Value"]

I realise these come from the aws-sdk .. so it may not be Stackup's issue ?

In hash style, it works fine. e.g.

{
  "Name": "some-stack",
  "Environment": "test",
  "Owner": "Delivery Engineering",
  "Criticality": "Low",
  "CostCentre": "1000",
  "PublicResource": "Yes"
}

FYI, I'm using the Stackup docker container:
realestate/stackup 1.4.5 247f28127230 2 months ago 51MB

rake tasks - some params from ENV

I just started poking around at this gem. Looks interesting.

I can't seem to figure out how I can get some params into environment variables.

  t.stack = "NetworkPipeline"
  t.template = "pipelines/networkPipeline.yaml"
  t.parameters = "pipelines/networkPipelineConfig.yaml"
  t.tags = {
    "builtby" => `whoami`
  }

in the param file, I don't want to have my GitHub access token. Is there a way to inject this?

Is there a way to do this if I was to write a ruby script instead of the rake task?

I am poking around at some of the cli - just a bit of friendly feedback as well - the subtasks don't have documentation on the command line so it's a little tricky - and there aren't examples of using the CLI in the examples folder so I have to go digging around in the code.

Assume role expired less than 2 hours

Since our stack creation would run at least last 2h 30m, we are now using --with-role to extend the assumed role credential, but it looks like the assumed role still expired in 2 hours.

Here is part of log:
21-Nov-2016 21:44:53: start to create the stack;
21-Nov-2016 23:35:02 ERROR: The security token included in the request is expired
21-Nov-2016 23:35:02 An error occurred (ExpiredToken) when calling the DescribeStacks operation: The security token included in the request is expired

Needs Ruby 2.1+

In lib/stackup/yaml.rb#L27, there is the line:

CloudFormationToRuby.create.accept(tree)

Psych::Visitors::ToRuby::create appears to have been introduced in the YAML library bundled with Ruby 2.1.x. It fails with a NoMethodError on earlier Rubies.

This probably isn't a problem on account of the earlier Rubies being EOL'd, but Ruby 2.0.0 is bundled with even the latest macOS, so perhaps a note in the README is warranted? Or I could submit a PR with a hack to check whether the method exists, otherwise fall back to ::new. Your call ๐Ÿ‘

Templates over 51kb in size should use S3

Attempting to create stacks with templates over 51kb (after JSON conversion) result in errors from the AWS API.

Instead, these templates need to be uploaded to an S3 bucket, and then a pre-signed URL passed to cloudformation with the template-url parameter, rather than template-body.

Parsing Error with [ in !Sub

I'm trying to create a CFN init object with Files like this :

          files:
            /etc/awslogs/awscli.conf:
              content: !Sub: |
                  [default]
                  region = ${AWS::Region}
                  [plugins]
                  cwlogs = cwlogs

This will break the parsing, and cause the gem to spit out the entire thing.

  /home/imran/.rbenv/versions/2.3.1-railsexpress/lib/ruby/2.3.0/json/common.rb:156:in `parse'
  /home/imran/.rbenv/versions/2.3.1-railsexpress/lib/ruby/2.3.0/json/common.rb:156:in `parse'
  /home/imran/.rbenv/versions/2.3.1-railsexpress/lib/ruby/gems/2.3.0/gems/multi_json-1.12.1/lib/multi_json/adapters/json_common.rb:14:in `load'
  /home/imran/.rbenv/versions/2.3.1-railsexpress/lib/ruby/gems/2.3.0/gems/multi_json-1.12.1/lib/multi_json/adapter.rb:21:in `load'
  /home/imran/.rbenv/versions/2.3.1-railsexpress/lib/ruby/gems/2.3.0/gems/multi_json-1.12.1/lib/multi_json.rb:122:in `load'


But if I put a \ infront of the [ - seems to work.

Please document required IAM permissions for `up` command

I'm porting a build onto stackup and are discovering one by one the IAM permissions that stackup's up command requires.

            "cloudformation:CreateStack",
            "cloudformation:UpdateStack",
            "cloudformation:DescribeStacks",
            "cloudformation:DescribeStackEvents",

Would be good to mention these (and any others Ive yet to discover) in the docs around credentials.

Multi-OS Docker image

G'day @mdub,

I've found myself in the unfortunate position where I occasionally have to run stackup from a Windows host. Ruby is quite the pain on Windows and I've got an uphill fight if I want to convince coworkers to go through that pain to enjoy the bliss that is stackup.

How amenable would you be to a PR that adds multi-OS support to the Docker images for stackup? Now with Docker supporting both Linux and Windows containers (but not both at the same time without massive pain), it would be nice if users could just docker run -it stackup regardless of which mode is currently active. I'd be happy to do the work, just want to know if you'd accept something like that spoiling this otherwise wonderful repo.

"Rate exceeded" with default retry limit

My team has some projects which involve deploying about 10 stacks in parallel to the same AWS account and region. If we deploy multiple of these projects at once we see stackup failing with "Rate exceeded" error.

Our developers and/or per-application deployment scripts ideally shouldn't have to think about setting a specific --retry-limit option when they deploy, as this is a generic CloudFormation API concern rather than an application-specific concern. I would like stackup to keep retrying for a very long time, with exponential backoff, so that it gracefully handles the case where many deployments are happening at once.

Related to #55 .

Fails to create change sets when a stack is in the REVIEW_IN_PROGRESS state

Using stackup 1.2.0

If I do change-set create against a stack that doesn't exist, a stack is created in the REVIEW_IN_PROGRESS state, as expected.

If I then try to create another change set, I get ERROR: no such stack.

I believe this is because stackup uses a change set type of UPDATE when the stack already exists -- but for a stack in the REVIEW_IN_PROGRESS state, it's appropriate to use a change set type of CREATE instead.

Feature request: do not normalise to JSON before invoking Cloudformation API's

As per https://github.com/realestate-com-au/stackup/blob/v1.4.5/README.md#yaml-support, and discussions in #31 & #74 - Stackup normalises YAML input to JSON before invoking Cloudformation API's.

This functionality is because Stackup supported YAML before Cloudformation did.

As I understand it, it also makes diff'ing templates accurate when doing stackup diff ....

I am however, still wondering if we could change Stackup to (optionally?) upload templates in their original format. In particular, because the YAML format is much easier to read - and it's nice to be able to read a stack's template within the AWS web-console.

What are the challenges in supporting this?
Where can I help?
Would this feature request be accepted at all?

Logging of missing parameters when using change-set's

We are using the change-set feature to help diff the impact of our cloudformation changes, and it is working really well. We have just run into an edge condition when we add new parameters to our stack.

The root cause is operator error, but stackup could help narrow down the issue with better logging.

When using the stack up command with missing parameters we see:

$ stackup db-dev-steve up -t cloudformation/db/master-stack.json -p parameters/db/common-params.json -o Environment=dev-steve
ERROR: ParameterValue for ParameterKey Account is required

But doing this via change-set create it doesn't say which parameter is in error.

$ stackup db-dev-steve change-set create -t cloudformation/db/master-stack.json -p parameters/db/common-params.json -o Environment=dev-steve
ERROR: Cannot specify UsePreviousValue on Parameters when ChangeSetType is CREATE

Would be great if the change-set create could output which parameter is in error.

Escaped slashes break stackup

For some reason, a template with escaped slashes, e.g. "Description": "DC\/OS AWS CloudFormation Template" will break stackup.

#-โฏ docker run -v $PWD:/cwd \
  --region ap-southeast-2 \
  -e AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY \
  -e AWS_SECURITY_TOKEN \
  -e AWS_SESSION_TOKEN \
  realestate/stackup:0.9.5 \
  dpearce-mesos up -t mesos.json -p dpearce-mesos.json

/usr/local/lib/ruby/2.3.0/psych.rb:377:in `parse': (mesos.json): found unknown escape character while parsing a quoted scalar at line 3 column 18 (Psych::SyntaxError)

At first I thought ruby was having trouble with that JSON string in general. But:

#-โฏ ruby -rjson -e 'puts JSON.parse(File.read("./mesos.json")).keys'
AWSTemplateFormatVersion
Description
Metadata
Parameters
Mappings
Resources
Outputs

So then I thought it was something to do with docker volumes, but:

#-โฏ docker run -v $PWD:/cwd ruby:2.3 ruby -rjson -e 'puts JSON.parse(File.read("/cwd/mesos.json")).keys'
AWSTemplateFormatVersion
Description
Metadata
Parameters
Mappings
Resources
Outputs

Lastly, I verified I was testing on the correct version of ruby:

#-โฏ docker run --entrypoint ruby realestate/stackup:0.9.5 -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]

Support new YAML template features

CloudFormation Now Supports YAML.

Looks like stackup will need some design and then development given its strict JSON template handling: https://github.com/realestate-com-au/stackup/blob/master/lib/stackup/stack.rb#L115 (as an example)

Im going to have a look into converting a small stack in the next week or so, and will try attempt a YAML stackup refactor with that stack in both JSON and YAML. I'll at least try and provide any relevant info if and when possible.

References

Jeff's blog and release notes:
https://aws.amazon.com/blogs/aws/aws-cloudformation-update-yaml-cross-stack-references-simplified-substitution/

CloudFormation Formats documentation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-formats.html

Stack delete fails with `stack_status variable not found.`

Logs:

/var/lib/gems/2.2.0/gems/stackup-0.0.7/lib/stackup/stack.rb:88:in `block in wait_for_events': undefined local variable or method `stack_status' for #<Stackup:
:Stack:0x00000001670bc8> (NameError)
        from /var/lib/gems/2.2.0/gems/stackup-0.0.7/lib/stackup/stack.rb:86:in `loop'
        from /var/lib/gems/2.2.0/gems/stackup-0.0.7/lib/stackup/stack.rb:86:in `wait_for_events'
        from /var/lib/gems/2.2.0/gems/stackup-0.0.7/lib/stackup/stack.rb:46:in `delete'
        from /vagrant/rea/cbde/jinkun/lib/jinkun/stack/master.rb:19:in `delete'

Allow parameters to be created from file and/or command line

It would be cool if I could specify a parameters file and also set some parameters on the command line eg. stackup mystack up -t template -p paramsfile -p buildnumber=1.

This way I could change something like the version or build number from the command line and keep all other items in the params file. Currently I need to use sed and a placeholder in the params file to change this.

stdout not appearing until the cfn has finished

It might be something I have done, but my status output doesnt appear until after the CFN has finished running. Is there a STDOUT.sync = true anywhere to make sure the output is flushed to stdout as it happens?

Can remote parameters files be accessed?

Documentation says so You can use either local files, or HTTP URLs, to specify inputs; stack template, parameters, etc. but when I attempt to access a parameters file from s3 I get the following error:

ERROR: cannot read "https://s3-ap-southeast-2.amazonaws.com/my-bucket-12345/dev-parameters.yml" - 403 Forbidden

Feedback when change set creation fails due to no changes

I want to set up a CI plan that will create a change set, show the user what the changes are, wait for the user to click a button, and then apply the changes.

When updating a stack normally (without change sets), stackup will say "No update required" and exit with code 0 if there are no changes. This makes it easy to implement idempotent deployments.

But when using change-sets, stackup will fail (exit code 1) with:

ERROR: The submitted information didn't contain changes. Submit different information to create a change set.
ERROR: change-set creation failed

It would be useful if there was some way (exit code) for my script to differentiate between "change set creation failed due to an error" and "change set creation failed due to no changes", so my build can be green when there are no changes needed.

Multiple params files.

Currently if you want to maintain a deployment parameter, such as Version or something, in a params file, your only option is to include this in the same params.json file as all your other (probably environment) params.

This makes automating the update of that param harder, since there might be 20+ params in there, you are using a tool like jq or something to manipulate the file, hoping you don't accidentally change any other params.

It would be handy if stackup up could take multiple -p arguments. That way, the Version param you are maintaining can be in a file all of it's own, and you don't necessarily need to use a tool like jq to edit it. A simple echo '{"Version": "..."}' > version.json will suffice, no external json parsing tools required.

Obviously with this change comes the issue of which params file should the real param come from. As an end-user I would assume the last one. E.g. up -p params.json -p version.json ..., then if params.json also had a Version in it, it would get overridden. Much like a -o Version=... will also override both of them.

Create and Update functions should not mutate the options parameter

The functions create() and update() are currently mutating the options parameter. Similar to the create_or_update() function, they should duplicate the options parameter into a local variable called options, i.e. options = options.dup (https://github.com/realestate-com-au/stackup/blob/master/lib/stackup/stack.rb#L115)

For example, we're invoking the create_or_update() function like so to create a stack:

    stack = Stackup.stack(stack_name)
    stack.create_or_update(
      ..
      template: template,
      disable_rollback: true
    )

We have a working CloudFormation template and a set of parameters that are expected to fail the stack creation.

Here's a sequence of events that show how a stack creation with disable_rollback=true behaves inconsistently:

  1. [SUCCESS] Invoke the code to create a brand new stack. The stack creation fails but as instructed, the resources are not rolled back, and the stack ends with a status of 'CREATE_FAILED'.
  2. [FAILED] Invoke the code with the same template and params to create the same stack using the same name. Stackup successfully deletes the existing stack and then proceeds to create a new stack.
    1. This time around the stack creation fails as expected, but all the resources were rolled back, and the stack ends with a status of 'ROLLBACK_COMPLETE'.
    2. Looking through the code, it appears:
      1. L132 delete if ALMOST_DEAD_STATUSES.include?(status) was responsible for deleting the existing failed stack from step 1
      2. L133 update(options) was executed;
        1. it mutated the options param by removing disable_rollback on L293 options.delete(:disable_rollback)
        2. as the stack had just been deleted, a NoSuchStack exception was thrown and caught by L134 of the create_or_update() function
        3. create(options) is finally executed to create the stack, but this time around, disable_rollback is no longer set in the options hash, as it was deleted in the update function, hence the stack creation failure followed by a rollback

Can the create() and update() functions be updated to duplicate the options parameter into a local variable instead?

Thanks

Unhelpful error message when parameters file contains unknown keys

I'm getting this error:

/usr/local/bundle/gems/stackup-1.1.2/lib/stackup/parameters.rb:56:in `public_send': undefined method `stack_name=' for #<Stackup::ParameterStruct:0x005620462f23a0> (NoMethodError)
	from /usr/local/bundle/gems/stackup-1.1.2/lib/stackup/parameters.rb:56:in `block in initialize'
	from /usr/local/bundle/gems/stackup-1.1.2/lib/stackup/parameters.rb:52:in `each'
	from /usr/local/bundle/gems/stackup-1.1.2/lib/stackup/parameters.rb:52:in `initialize'
	from /usr/local/bundle/gems/stackup-1.1.2/lib/stackup/parameters.rb:19:in `new'
	from /usr/local/bundle/gems/stackup-1.1.2/lib/stackup/parameters.rb:19:in `block (2 levels) in hashify'
	from /usr/local/bundle/gems/stackup-1.1.2/lib/stackup/parameters.rb:18:in `each'
	from /usr/local/bundle/gems/stackup-1.1.2/lib/stackup/parameters.rb:18:in `block in hashify'
	from /usr/local/bundle/gems/stackup-1.1.2/lib/stackup/parameters.rb:17:in `tap'
	from /usr/local/bundle/gems/stackup-1.1.2/lib/stackup/parameters.rb:17:in `hashify'
	from /usr/local/bundle/gems/stackup-1.1.2/lib/stackup/parameters.rb:10:in `new'
	from /usr/local/bundle/gems/stackup-1.1.2/bin/stackup:134:in `block in parameters_from_files'
	from /usr/local/bundle/gems/stackup-1.1.2/bin/stackup:133:in `map'
	from /usr/local/bundle/gems/stackup-1.1.2/bin/stackup:133:in `parameters_from_files'
	from /usr/local/bundle/gems/stackup-1.1.2/bin/stackup:207:in `parameters'
	from /usr/local/bundle/gems/stackup-1.1.2/bin/stackup:189:in `execute'
	from /usr/local/bundle/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
	from /usr/local/bundle/gems/stackup-1.1.2/bin/stackup:58:in `run'
	from /usr/local/bundle/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
	from /usr/local/bundle/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
	from /usr/local/bundle/gems/stackup-1.1.2/bin/stackup:58:in `run'
	from /usr/local/bundle/gems/clamp-1.1.2/lib/clamp/command.rb:132:in `run'
	from /usr/local/bundle/gems/clamp-1.1.2/lib/clamp.rb:6:in `Clamp'
	from /usr/local/bundle/gems/stackup-1.1.2/bin/stackup:18:in `<top (required)>'
	from /usr/local/bundle/bin/stackup:22:in `load'
	from /usr/local/bundle/bin/stackup:22:in `<main>'

when running this command:

docker run --rm \
        -v `pwd`:/cwd \
        -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \
        -e AWS_DEFAULT_REGION="$REGION" \
        realestate/stackup:latest "$STACK_NAME" up -t "$TEMPLATE" -p "$PARAMETERS"

I'm not sure exactly what I'm doing wrong, and the error message certainly doesn't make it obvious.

Does stackup support cloudformation stackset?

Hi Team,

I have been using stackup and it's been worked very well, now I have a requirement to create cloudformation stackset, wondering does stackup also support creating stackset?

GEM CLI very slow in 1.3.1

I've updated my gems and now stackup v1.3.1 takes 30s to do anything on the command line. While it's "loading" I've got a ruby CPU thread at 100%. Same behaviour with other tools using that gem (shipper, swa).

$ time stackup --version
stackup v1.3.1
stackup --version  21.88s user 3.98s system 98% cpu 26.254 total

$ gem uninstall stackup
Successfully uninstalled stackup-1.3.1

$ gem install stackup -v "1.3.0"
Successfully installed stackup-1.3.0

$ time stackup --version
stackup v1.3.0
stackup --version  0.35s user 0.09s system 93% cpu 0.472 total

allow CAPABILITY_NAMED_IAM through the cli

Hi,

I can see that you can set a capability through using the ruby code, but I'm using stackup as a cli tool, and would like to be able to pass CAPABILITY_NAMED_IAM as a parameter which at the moment I'm unable to do.

There was discussion on #28 but it doesn't look like it's resulted in any changes.

Thanks

Sam

Separate into "core" and "cli" gems?

Some users might find the stackup CLI handy; others may just want to use the Stackup::Stack abstraction, without the overhead of the gems that support the CLI (clamp, diffy, multi_json).

Should we split this into separate gems? I was thinking:

  • stackup-core - the core Ruby API
  • stackup-cli - the CLI

@kunday? @lukeck?

Stackup should gracefully handle Rate Limiting

We use stackup for deploying some large stacks that can take an hour+ to deploy.
With increased development effort on our AWS account we often run into AWS Rate Limit issues.

We are seeing ERROR: Rate exceeded failures on some of our longer running deployments.

INFO: [10:21:56] BastionStack - UPDATE_COMPLETE
INFO: [10:21:57] StagingStack - UPDATE_IN_PROGRESS
ERROR: Rate exceeded
make: *** [run_stackup] Error 1

Named IAM role capabilities

lib/stackup/stack.rb#L129 has the line:

options[:capabilities] ||= ["CAPABILITY_IAM"]

There's a new capability named CAPABILITY_NAMED_IAM, but there appears to be no way to use it with stackup. It would be great if this was added to the defaults (as above) or some way to specify capabilities was provided.

Enable use of service role ARN when using change sets

I'm currently using stackup to build stacks across account. To do this I'm using both an assumed role (cross-account), and a service role which is passed to Cloudformation to perform the stack operations.

The standard up command supports the parameter --service-role-arn, but when using change sets this option isn't available.

I've created a PR to add this functionality.

Produce a message when there are no changes to be done

I notice that if cloudformation has nothing to do, stackup exits with no screen output. Is it possible to add some logs to say something like "There are no updates to be applied." or if you cant know that even a simple "Stack up completed successfully". This way I will know that it was not an error, or mistype and that it was simply there were no changes to be made.

Mismatch between env variable in readme file and what stackup expects

In the readme there is a reference to an env variable to handle the --retry-limit behaviour. That env variable in the readme is $AWS_RETRY_LIMIT where stackup expects $AWS_API_RETRY_LIMIT.

readme:

You can increase the limit using the `--retry-limit` option, or by setting the `$AWS_RETRY_LIMIT` environment variable.

stackup: https://github.com/realestate-com-au/stackup/blob/main/lib/stackup/main_command.rb#L34

I noticed this because I was specifying the env variable and wasn't getting the expected result (ie it doesn't throw an error so it's a bit of a silent error)

Applying change-set to create stack fails

Hi - me again :D

Found another issue when using change-set's - this time it's with using change-sets to create new stacks.

After successfully creating a change-set in create mode, when we go to apply the stack it creates correctly but fails with an unexplainable error.

stackup cdn-dev-f change-set apply \
            --debug \
                --retry-limit 10 \
                --wait-poll-interval 10
DEBUG: [Aws::CloudFormation::Client 200 0.214019 0 retries] describe_stack_events(stack_name:"cdn-dev-f")
DEBUG: [Aws::CloudFormation::Client 200 0.21583 0 retries] execute_change_set(stack_name:"cdn-dev-f",change_set_name:"pending")
DEBUG: [Aws::CloudFormation::Client 200 0.090555 0 retries] describe_stack_events(stack_name:"cdn-dev-f")
INFO: [21:55:02] cdn-dev-f - CREATE_IN_PROGRESS - User Initiated
DEBUG: [Aws::CloudFormation::Client 200 0.108543 0 retries] describe_stacks(stack_name:"cdn-dev-f")
DEBUG: stack_status=CREATE_IN_PROGRESS
DEBUG: [Aws::CloudFormation::Client 200 0.196638 0 retries] describe_stack_events(stack_name:"cdn-dev-f")
INFO: [21:55:06] ImageResizeLambdaStack - CREATE_IN_PROGRESS
INFO: [21:55:06] ImagePropStack - CREATE_IN_PROGRESS
INFO: [21:55:07] ImageResizeLambdaStack - CREATE_IN_PROGRESS - Resource creation Initiated
INFO: [21:55:07] ImagePropStack - CREATE_IN_PROGRESS - Resource creation Initiated
DEBUG: [Aws::CloudFormation::Client 200 0.069392 0 retries] describe_stacks(stack_name:"cdn-dev-f")
DEBUG: stack_status=CREATE_IN_PROGRESS
DEBUG: [Aws::CloudFormation::Client 200 0.188284 0 retries] describe_stack_events(stack_name:"cdn-dev-f")
INFO: [21:55:55] ImageResizeLambdaStack - CREATE_COMPLETE
INFO: [21:55:55] ImagePropStack - CREATE_COMPLETE
INFO: [21:55:58] S3ImageBuckets - CREATE_IN_PROGRESS
INFO: [21:55:59] S3ImageBuckets - CREATE_IN_PROGRESS - Resource creation Initiated
DEBUG: [Aws::CloudFormation::Client 200 0.081156 0 retries] describe_stacks(stack_name:"cdn-dev-f")
DEBUG: stack_status=CREATE_IN_PROGRESS
DEBUG: [Aws::CloudFormation::Client 200 0.206843 0 retries] describe_stack_events(stack_name:"cdn-dev-f")
INFO: [21:57:18] cdn-dev-f - CREATE_COMPLETE
DEBUG: [Aws::CloudFormation::Client 200 0.069191 0 retries] describe_stacks(stack_name:"cdn-dev-f")
DEBUG: stack_status=CREATE_COMPLETE
ERROR: update failed

You can see that the final stack_status is CREATE_COMPLETE, but then we get this ERROR: update failed. This returns non-zero and so borks out our deploy process.

Happy to provide more details to help identify this issue.

Add support for S3 source when using "diff"

Hi Guys,

Currently stackup up supports the use of an S3 source, but stackup diff does not.
Would it be possible to add support for S3 when running the diff sub-command?

Thanks
Duncan

Use CloudFormation Service Role

Firstly, thanks for the great work! Stackup can simplify the CloudFormation workflow a lot! But I cannot replace my Bash scripts for deploying CloudFormation stacks with Stackup yet.

It is because I cannot set the Service Role of the stacks as described in here https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-iam-servicerole.html.

I understand that Stackup allows me to specify a role to assume for the stack operation. But it is different from the CloudFormation service role. To me, the key difference is that I can grant the CloudFormation service role to have very broad privileges without directly granting those privileges to the users' role, and the users cannot assume the service role directly. This means that the users are forced to use CloudFormation to manage the AWS resources which is good in terms of IaC compliance.

It will be great if Stackup can support CloudFormation service role. Thanks!

!Sub not substituting pseudovariables in YAML CFN templates

See YAML template that uses !Sub to substitute pseudo parameters.

Output observed in build implies substitution not performed.

INFO: [15:41:10] SnsPermissionAddBuyProfile - UPDATE_FAILED - 2 validation errors detected: Value 'arn:aws:sns:${AWS::Region}:${AWS::AccountId}:*' at 'sourceArn' failed to satisfy constraint: Member must satisfy regular expression pattern: arn:aws:([a-zA-Z0-9\-])+:([a-z]{2}-[a-z]+-\d{1})?:(\d{12})?:(.*); Value 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:*' at 'functionName' failed to satisfy constraint: Member must satisfy regular expression pattern: (arn:aws:lambda:)?([a-z]{2}-[a-z]+-\d{1}:)?(\d{12}:)?(function:)?([a-zA-Z0-9-_]+)(:(\$LATEST|[a-zA-Z0-9-_]+))?

Stack creation broken

In v0.9.0, I broke deployment of new stacks! Fails with:

$ stackup woollyams-test up --debug -t examples/template.json -p examples/parameters-terse.yml 
DEBUG: [Aws::CloudFormation::Client 400 0.323207 0 retries] describe_stacks(stack_name:"woollyams-test") Aws::CloudFormation::Errors::ValidationError Stack with id woollyams-test does not exist
DEBUG: [Aws::CloudFormation::Client 400 0.104306 0 retries] describe_stack_events(stack_name:"woollyams-test") Aws::CloudFormation::Errors::ValidationError Stack [woollyams-test] does not exist
ERROR: Stack [woollyams-test] does not exist

What a moron.

stackup -p errors out when file is not found

It sounds trivial that the file must exist before using it during an override, but if the stackup command can deal with a missing file then it would be great.

Example -
$ stackup myapp-test up -t template.json
-p defaults.json
-p overrides.json
-p filedoesnotexist.json

the above command should not fail

Take template from stdin

Is it possible to provide your json template via standard in eg, cfndsl myrubycfntemplate.rb | stackup mystack up -p params_file.

This would be really handy if you use a tool like cfndsl, cfoo, troposphere etc and dont want to have to have an intermediary step of writing the json to a file.

Updating stack using --tags option, ArgumentError

All thats needed to re-create this error is an existing stack that you want to update using the --tags argument as a file path. When running the command below there is an error in the AWS sdk. Really annoying cause the stack level tags are great!

I know this isn't an issue with stackup, but was hoping those who created this might be able to debug better than I can. Going into the param_validator.rb and trying to chase down the seahorse shape stuff is head ache inducing.

I have had to resort back to manually tagging resources for now. Creating the stack works perfectly its just the update that is raising.

trace:

/Users/dave/.rvm/gems/ruby-2.3.1/gems/aws-sdk-core-2.2.18/lib/aws-sdk-core/param_validator.rb:26:in `validate!': unexpected value at params[:tags] (ArgumentError)
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/aws-sdk-core-2.2.18/lib/aws-sdk-core/param_validator.rb:11:in `validate!'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/aws-sdk-core-2.2.18/lib/aws-sdk-core/plugins/param_validator.rb:20:in `call'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/aws-sdk-core-2.2.18/lib/aws-sdk-core/plugins/logging.rb:39:in `call'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/aws-sdk-core-2.2.18/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/aws-sdk-core-2.2.18/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/aws-sdk-core-2.2.18/lib/seahorse/client/plugins/response_target.rb:21:in `call'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/aws-sdk-core-2.2.18/lib/seahorse/client/request.rb:70:in `send_request'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/aws-sdk-core-2.2.18/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/aws-sdk-resources-2.2.18/lib/aws-sdk-resources/request.rb:24:in `call'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/aws-sdk-resources-2.2.18/lib/aws-sdk-resources/operations.rb:41:in `call'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/aws-sdk-resources-2.2.18/lib/aws-sdk-resources/operation_methods.rb:19:in `block in add_operation'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/lib/stackup/stack.rb:264:in `block in update'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/lib/stackup/stack.rb:293:in `block (2 levels) in modify_stack'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/lib/stackup/error_handling.rb:16:in `handling_validation_error'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/lib/stackup/stack.rb:292:in `block in modify_stack'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/lib/stackup/stack.rb:232:in `watch'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/lib/stackup/stack.rb:291:in `modify_stack'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/lib/stackup/stack.rb:263:in `update'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/lib/stackup/stack.rb:120:in `create_or_update'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/bin/stackup:147:in `block in execute'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/bin/stackup:92:in `report_change'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/bin/stackup:146:in `execute'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/bin/stackup:50:in `run'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/clamp-1.0.0/lib/clamp/subcommand/execution.rb:11:in `execute'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/bin/stackup:50:in `run'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/clamp-1.0.0/lib/clamp/command.rb:133:in `run'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/clamp-1.0.0/lib/clamp.rb:6:in `Clamp'
    from /Users/dave/.rvm/gems/ruby-2.3.1/gems/stackup-0.8.2/bin/stackup:16:in `<top (required)>'
    from /Users/dave/.rvm/gems/ruby-2.3.1/bin/stackup:23:in `load'
    from /Users/dave/.rvm/gems/ruby-2.3.1/bin/stackup:23:in `<main>'
    from /Users/dave/.rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in `eval'
    from /Users/dave/.rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in `<main>'

command:

bundle exec stackup ${APP} up \
  -t ${APP}.json \
  -p ${DEPLOY_ENV}.params.json \
  --tags tags.json \
  --region ${AWS_REGION}

Support templates pulled from S3

Can templates existing in either S3 or Github be referenced? Can't see anything in the documentation and tried referencing a template in S3 but got a ERROR: no such file response.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.