Git Product home page Git Product logo

bora's Introduction

Gem Build Status

Bora

This Ruby gem contains a command line utility and rake tasks that help you define and work with CloudFormation stacks.

In a single YAML file you define your templates, the stack instances built from those templates (eg: dev, uat, staging, prod, etc), and the parameters for those stacks. Parameters can even refer to outputs of other stacks. Templates can be written with plain CloudFormation JSON or cfndsl.

Given this config, Bora then provides commands (or Rake tasks) to work with those stacks (create, update, delete, diff, etc).

Installation

This gem requires Ruby 2.1 or greater.

If you're using Bundler, add this line to your application's Gemfile:

gem 'bora'

And then run bundle install.

Alternatively, install directly with gem install bora.

Quick Start

Create a file bora.yml in your project directory, something like this:

templates:
  example:
    template_file: example.json
    stacks:
      uat:
        params:
          InstanceType: t2.micro
      prod:
        params:
          InstanceType: m4.xlarge

Now run bora apply example-uat to create your "uat" stack. Bora will wait until the stack is complete (or failed), and return stack events to you as they happen. To get a full list of available commands, run bora help.

Alternatively if you prefer using Rake, add this to your Rakefile:

require 'bora'
Bora.new.rake_tasks

Then run rake example-uat:apply. To get a full list of available tasks run rake -T.

File Format Reference

The example below is a bora.yml file showing all available options:

# Optional. The default region for all stacks in the file.
# See below for further information.
default_region: us-east-1

# A map defining all the CloudFormation templates available.
# A "template" is effectively a single CloudFormation JSON (or cfndsl template).
templates:
  # A template named "app"
  app:
    # This template is a plain old CloudFormation JSON file
    template_file: app.json
    # Tag Key / Value pairs for the Cloudformation stack
    # Tags are inherited by stack resources
    tags:
      Name: my-app
    # Optional create stack parameters
    # See - http://docs.aws.amazon.com/sdkforruby/api/Aws/CloudFormation/Client.html#create_stack-instance_method
    capabilities: [CAPABILITY_IAM] # An array of "capabilities" to be passed to the CloudFormation API
    on_failure: DO_NOTHING # See CloudFormation API docs for valid values and their meanings; "disable_rollback" is also supported
    # Optional. The default region for all stacks in this template.
    # Overrides "default_region" at the global level.
    # See below for further information.
    default_region: us-west-2

    # A map defining all the "stacks" associated with this template
    # for example, "uat" and "prod"
    stacks:
      # The "uat" stack
      uat:

        params:
          InstanceType: t2.micro
          AMI: ami-11032472
      # The "prod" stack
      prod:
        # Overrides template level
        on_failure: DELETE
        # Tags here are merged with the template
        tags:
          Environment: uat
        # The CloudFormation parameters to pass into the stack
        # Optional. The stack name to use in CloudFormation
        # If you don't supply this, the name will be the template
        # name concatenated with the stack name as defined in this file,
        # eg: "app-prod".
        cfn_stack_name: prod-application-stack
        # Optional. Default region for this stack.
        # Overrides "default_region" at the template level.
        # See below for further information.
        default_region: ap-southeast-2
        params:
          InstanceType: m4.xlarge
          AMI: ami-11032472

  # A template named "web"
  web:
    # This template is using cfndsl. Bora treats any template ending in
    # ".rb" as a cfndsl template.
    template_file: "web.rb"
    stacks:
      uat:
        # The CloudFormation parameters to pass into the stack.
        # You can define both cfndsl parameters and traditional CloudFormation
        # parameters here. Cfndsl will receive all of them, but only those
        # actually defined in the "Parameters" section of the template will be
        # passed through to CloudFormation when the stack is applied.
        params:
          dns_zone: example.com

          # You can use complex data structures with cfndsl parameters:
          users:
            - id: joe
              name: Joe Bloggs
            - id: mary
              name: Mary Bloggs

          # You can refer to outputs of other stacks using "${}" notation too.
          # See below for further details.
          app_url: http://${cfn://app-uat/outputs/Domain}/api

          # Traditional CloudFormation parameters
          InstanceType: t2.micro
          AMI: ami-11032472

      prod: {}

Command Reference

The following commands are available through the command line and rake tasks.

  • apply - Creates the stack if it doesn't exist, or updates it otherwise
  • changeset - Manage CloudFormation change sets for the stack
  • delete - Deletes the stack
  • diff - Provides a visual diff between the local template and the currently applied template in AWS. The diff also shows the changes that CloudFormation will apply as reported by the CloudFormation Change Set API.
  • events - Outputs the latest events from the stack
  • list - Outputs a list of all stacks defined in the config file
  • outputs - Shows the outputs from the stack
  • parameters - Shows the parameters from the stack
  • recreate - Recreates (deletes then creates) the stack
  • show - Shows the local template in JSON, generating it if necessary
  • show_current - Shows the currently applied template in AWS
  • status - Displays the current status of the stack
  • validate - Validates the template using the AWS CloudFormation "validate" API call

Change Sets

Bora provides full support for working with CloudFormation Change Sets via the "changeset" command. The following subcommands are available (eg: bora changeset <subcommand> ...). Run bora changeset help for more information.

  • changeset create - Creates a change set for the stack
  • changeset list - Lists all available change sets for the stack
  • changeset show - Shows the details of a particular change set
  • changeset apply - Applies a change set. Any other available change sets will be automatically deleted by AWS after this action.
  • changeset delete - Deletes a particular change set for the stack

Note that bora diff will also show you a summary of the change set that will be applied. It does this by creating (and automaticlly deleting) a temporary change set in order to get the change actions to display.

Note that change set funtionality is not available via the Rake tasks at this time.

Command Line

Run bora help to see all available commands.

bora help [command] will show you help for a particular command, eg: bora help apply.

Rake Tasks

To use the rake tasks, simply put this in your Rakefile:

require 'bora'
Bora.new.rake_tasks

To get a full list of available tasks run rake -T.

Specifying Regions

You can specify the region in which to create a stack in a few ways. The order of precedence is as follows (first non-empty value found wins):

  • The --region parameter on the command line (only available in the CLI, not in the Rake tasks)
  • The default_region setting within the stack section in bora.yml
  • The default_region setting within the template section in bora.yml
  • The default_region setting at the top level of bora.yml
  • The default region as determined by the AWS Ruby SDK.

Parameter Substitution

Bora supports looking up parameter values from various locations and interpolating them into stack parameters. This is useful so that you don't have to hard-code values into your stack parameters that may change across regions or over time. For example, you might have a VPC template that creates a subnet and returns the subnet ID as a stack output. You could then have an application template that creates an EC2 instance in that subnet, with the subnet ID parameter looked up dynamically from the VPC stack.

These lookup parameters are specified using ${} syntax within the parameter value, and the lookup target is a URI.

For example:

params:
  api_url: http://${cfn://api-stack/outputs/Domain}/api

This will look up the Domain output from the stack named api-stack and substitute it into the api_url parameter. The URI "scheme" (cfn in the above example) controls which resolver will handle the lookup. The format of the rest of the URI is dependent on the resolver.

There are a number of resolvers that come with Bora (documented below), or you can write your own.

Parameter Lookup

Any substitution that does not specify a "scheme" is treated as a reference to another parameter value. For example:

params:
  domain: example.com
  url: http://${domain}/foo

This even works within array or hash parameters, although you can only look up values from the top level params.

params:
  domain: example.com
  site_config:
    url: http://${domain}/foo

Stack Output Lookup

You can look up outputs from stacks in the same region.

For example:

# Look up output "MyOutput" from stack "my-stack" in the same region as the current stack.
${cfn://my-stack/outputs/MyOutput}

# Look up an output from a stack in another region
${cfn://my-stack.ap-southeast-2/outputs/MyOutput}

CredStash Key Lookup

CredStash is a utility for storing secrets using AWS KMS. You can pass these secrets as parameters to your stack. If you do so, you should use a CloudFormation parameter with the "NoEcho" flag to true, so as to not expose the secret in the template.

For example:

# Simple key lookup in same region as the stack. Note 3 slashes. Will run `credstash get mykey`.
${credstash:///mykey}

# Lookup with a key context. Will run `credstash get mykey app=webapp`.
${credstash:///mykey?app=webapp}

# Lookup a credstash in another region.
${credstash://ap-southeast-2/mykey?app=webapp}

Route53 Hosted Zone ID Lookup

Looks up the Route53 hosted zone ID given a hosted zone name (eg: example.com). Also allows you to specify if you want the private or public hosted zone for a given name, which can be useful if you have set up split-view DNS with both public and private zones for the same name.

${hostedzone://example.com}
${hostedzone://example.com/public}
${hostedzone://example.com/private}

AMI Lookup

Looks up an AMI given a name prefix which may contain wildcards. If query returns multiple images the latest is used.

Owners takes a query string list of AWS account ID, self (owner is the sender of the request), or an AWS owner alias (valid values are amazon | aws-marketplace | microsoft). Omitting this option defaults to self

${ami://amzn-ami-hv*x86_64-gp2?owner=amazon}
${ami://my-windows-soe}
${ami://my-windows-soe?owner=1234567890,self}

Overriding Stack Parameters from the Command Line

Some commands accept a list of parameters that will override those defined in the YAML file.

If you are using the Bora command line, you can pass these parameters like this:

$ bora apply web-uat --params 'instance_type=t2.micro' 'ami=ami-11032472'

For rake, he equivalent is:

$ rake web-uat:apply[instance_type=t2.micro,ami=ami-11032472]

Creating Multiple Instances of a Stack

Sometimes it can be useful to create multiple instances of a single stack. For example, you may define a single "qa" stack with all the settings for a testing environment. Then you might want to stand up this stack multiple times so you can have multiple testing environments, eg "qa1", "qa2", etc.

Bora makes this possible by allowing you to override the name of the stack that gets created in CloudFormation. For example:

$ bora apply web-qa --cfn-stack-name "web-qa-1"
$ bora apply web-qa --cfn-stack-name "web-qa-2"

Remember that if you use this functionality you must remember to pass in the stack name to every command or you will get unexepected results.

$ bora outputs web-qa --cfn-stack-name "web-qa-1"

Work is underway to improve how Bora handles this use case. If this is of interest to you, please have a look at the GitHub issue for this functionality.

Related Projects

The following projects provided inspiration for or are similar to Bora. If Bora doesn't meet your needs, one of these might.

  • CfnDsl - A Ruby DSL for CloudFormation templates
  • StackMaster - Very similar in goals to Bora
  • CloudFormer - Rake tasks for CloudFormation
  • Cumulus - A Python YAML based tool for working with CloudFormation

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/ampedandwired/bora.

bora's People

Contributors

ampedandwired avatar herebebogans avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

bora's Issues

AMI Resolver?

Would there be interest in an AMI resolver ..?

some ideas for the syntax

${ami://OPTIONAL_ACCOUNT_ID/NAME_PREFIX_WILDCARD/}

Lookup latest by creation time

${ami://77777777/my_aws_linux*?latest}

Possible lookups by tags / filters

${ami://my_aws_linux?state=certified

diff of parameters

Sorry for all the feature requests.

It would be really useful for the diff method to show the changes in stack parameters of the current stack vs new.

Add some integration tests

Should have some proper end-to-end tests, as the specs don't catch everything.

Using awsspec would provide a good way to test things. I'm imagining a bunch of specs that set up a simple stack (one that costs nothing, eg with just a few security groups) and then asserts that the resources created by the stack match what you expect from the bora.yml.

Windows Ruby CA certificate error - potential fix?

This is not a Bora issue, but I couldn't find a workaround for Bora to work.

$ bora show_current example-qa
D:/dev/tools/ruby-2.3.1-x64-mingw32/lib/ruby/2.3.0/net/http.rb:933:in `connect_nonblock': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (Seahorse::Client::NetworkingError)
        from D:/dev/tools/ruby-2.3.1-x64-mingw32/lib/ruby/2.3.0/net/http.rb:933:in `connect'
        from D:/dev/tools/ruby-2.3.1-x64-mingw32/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
        from D:/dev/tools/ruby-2.3.1-x64-mingw32/lib/ruby/2.3.0/net/http.rb:858:in `start'
        from D:/dev/tools/ruby-2.3.1-x64-mingw32/lib/ruby/2.3.0/delegate.rb:83:in `method_missing'

I have been Googled on this error seems to be related to Ruby on Windows, that doesn't come with ca-bundle anymore. A work around is to add 'Aws.use_bundled_cert! if Gem.win_platform?' somewhere. Any idea where I can put this for Bora to work on Windows?

bora apply stack (update stack) error with 'on_failure' params

Create stack works fine but apply stack (update stack) failed with error:

aws-sdk-core/param_validator.rb:28:in `validate!': unexpected value at params["on_failure"] (ArgumentError)

'on_failure' is not an argument for aws update-stack command but would be good for Bora to not pass this to update-stack.

ruby: 2.4.1p111 (windows)
gem: 2.6.11
aws-sdk-core-2.9.38

Allow pass in AWS profile name as parameter

Hi. we have nonprod and prod environment as separated AWS account. It would be handy if bora can take a parameter for us to launch stacks in a specified account.
Currently I am using export AWS_PROFILE to switch but easily forgot if not careful.

Stack Tags merging bug

  timtest:
    template_file: templates/global/alb.rb
    on_failure: DO_NOTHING
    tags:
      CostCentre: EDS
    capabilities: [CAPABILITY_IAM]
    stacks:
      qa:
        tags:
          Account: NONPROD
          Environment: QA
        params:
          Environment: QA
          AlbSchema: internal
          targetgroups:
            XXXXX:
              healthcheck_path: '/healthcheck.html'
              listenerrules:
                - priority: 100
                  conditions:
                    host-header: 'xxxxx.x.x'
      stg:
        tags:
          Account: NONPROD
          Environment: STG
        params:
          Environment: STG
          AlbSchema: internal
          targetgroups:
            XXXXX:
              healthcheck_path: '/healthcheck.html'
              listenerrules:
                - priority: 100
                  conditions:
                    host-header: 'xxxxx.x.x'

I run bora apply timtest-qa, the stack it created has Tags:

  CostCentre: EDS
  Account: NONPROD
  Environment: STG  <--- Should be QA

Based on my testing, Bora always uses the last stack Tags.

cfndsl pretty json output

First thought I'd say that this is a really useful tool!

Would be nice to have a command line option to pass the equivalent of the -p (pretty json output) flag to cfndsl.

Additional layer of abstraction

Hi @ampedandwired,

Thanks for this gem!

I was wondering your thoughts on implementing an additional layer of abstraction.

For example. I have application 'example' comprised of multiple stacks that essentially build up separate components of aws infrastructure.

I would want the ability of bora apply example or rake example:apply which will then use the apply method to build those stacks listed within the bora.yml with each stack having a separate template file.

Thanks,
thebeefcake

Diff - default parameters

Looks like the new diff parameter feature doesn't consider default parameters in the template (cfndsl in my test case) vs the current running stack?

eg If I have a running stack with parameter

Foo = Bar

and a template with

"Parameters": {
"Foo": {
"Type": "String",
"Description": "Bar"
},

then a bora diff will show

Parameters
----------
-Foo - Bar

Does Bora convert Number from Yaml to String when passing parameters to the Templates?

First of all, Bora looks great. I am trying this out at the moment. One issue I came across, not sure if my environment set up or else, but: When running 'bora apply example-qa':

/aws-sdk-core-2.9.36/lib/aws-sdk-core/param_validator.rb:28:in 'validate!': parameter validator found 4 errors: (ArgumentError)
  - expected params[:parameters][2][:parameter_value] to be a String, got value 20 (class: Fixnum) instead.
  - expected params[:parameters][4][:parameter_value] to be a String, got value 1 (class: Fixnum) instead.
  - expected params[:parameters][5][:parameter_value] to be a String, got value 1 (class: Fixnum) instead.
  - expected params[:parameters][6][:parameter_value] to be a String, got value 1 (class: Fixnum) instead.

This is my bora.yaml params:

        params:
          Environment: qa
          InstanceType: m4.xlarge
          EbsSize: 20
          AMI: ami-b82622db
          MinNumberOfInstance: 1
          DesiredNumberOfInstance: 1
          MaxNumberOfInstance: 1

This is my CFNDSL parameter definitions:

  Parameter('EbsSize') do
    Type('Number')
    Default 20
  end

  Parameter('MinNumberOfInstance') do
    Type('Number')
    Default 1
  end

  Parameter('DesiredNumberOfInstance') do
    Type('Number')
    Default 1
  end

  Parameter('MaxNumberOfInstance') do
    Type('Number')
    Default 1
  end

After commenting out those params in bora.yaml file, the stack got created and no errors reported.

Rubocop PR

Tracking issue for a Rubocop fixup PR.

cfn-stack-name parameter is not working with rake

$ rake example-qa:status --cfn-stack-name=example-qa-6
invalid option: --cfn-stack-name=example-qa-6

$ rake example-qa:status[cfn-stack-name=example-qa-6]
Stack does not exist

$ bora status example-qa --cfn-stack-name "example-qa-6"
example-qa-6 - CREATE_COMPLETE

Delete stack with no event polling

Hi

Thoughts on an override option (--no-events?) for delete to not poll the event output?

There's times when you just want to

Aws::CloudFormation::Client.delete_stack()

and not block on the event output.

dynamic list of stacks

Hey mate,

We have a requirement to run the near same stack many times but with a slightly different config. That is a dynamic list really. I want to pass in and have a stack run a set config (say dev) but then override with my runtime params and most importantly give it an aws cfn name rather than a static one in bora.yml. So something like

bora apply my-stack-dev --cfn-stack-id=my-stack-dev-1 --params 'example=1'

Looked at the code - think it is possible if I treat cfn-stack-id as a new type of param and in stack.rb would have to do some work so that:

@cfn_stack_name = stack_config['stack_name'] || @stack_name

can also be set by a 3rd option (which takes precedence)

But do you want to take it in this direction as it kinda adds a new concept and is a bit of a largish change. Have to deal with all the other tasks, look at resolvers etc

Otherwise I'll use output the and fall back to aws tooling.

Stack tags on creation

There's currently no support for creating tag / value pairs for the CF stack itself?

Thoughts on extending the yaml to something like.

templates:
  mystack:
    template_file: mystack.rb
    stacks:
      tags:
        - Name: Name
          Value: My application stack
      uat:
        tags:
          - Name: Environment
            Value: uat
          - Name: Cost_Center
            Value: Development
        params:
          InstanceType: t2.micro
      prod:
        tags:
          - Name: Environment
            Value: prod
          - Name: Cost_Center
            Value: Business
        params:
          InstanceType: m4.xlarge

The tags would then be passed to the create_stack / update_stack api.

Allow stacks to be instantiated multiple times

See #6 for background. The basic idea is taking a stack, say "web-uat" and being able to instantiate that many times ("web-uat-1", "web-uat-2", etc).

At the moment this is possible by using the --cfn-stack-name command line option, however this should really be supported as a first-class citizen in Bora, as it is a powerful way of working with stacks.

bora in bamboo

Hi Guys,

When I run bora in Bamboo tasks, it doesn't show the stack creation progress event logs until the whole task is finished (the output is being buffered somewhere). It works fine on my PC command line.
Just wondering do you have similar problems when run bora in Bamboo / Jenkins and what is your workaround?

Support for CloudFormation custom resources using Lambda

CloudFormation custom resources are really powerful, but not often used because it's a fair bit of messing around to get them set up and working.

Bora could allow you to write custom resources in Ruby code alongside your templates, then package them up and deploy them automatically to Lambda for you.

Change set

Hi Charles,

Can I make a feature request. Can you add or please help me add the ability to generate change sets, using change_set api call

Cheers,

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.