Git Product home page Git Product logo

awscfncli's Introduction

AWS CloudFormation CLI

The missing CloudFormation CLI.

Official cfncli is not designed to manage stacks at this point.

[TOC]

Introduction

awscfncli helps build and manage AWS CloudFormation stacks.

Highlights:

  • Manage stacks in different accounts and regions use single YAML config file.
  • Cross-stack parameter reference works cross-region and cross-account.
  • Organize stack using stages and blueprints.
  • Automatically package and upload template resources.
  • Push button SAM deployment using stack sync command.
  • Display and track stack events in the CLI.
  • List stack resources, outputs and exports in the CLI.

Install

Install from pypi:

pip install --user --pre awscfncli2 

When install globally, use pipx is recommended:

pipx install awscfncli2 

Usage

Quickstart

cfn-cli [OPTIONS...] COMMAND SUBCOMMAND [ARGS...]

To view a list of available subcommands, use:

cfn-cli COMMAND --help

Options:

  • -f, --file: Specify an alternate config file.
  • -s, --stack: Specify stacks to operate on, defined by STAGE_NAME.STACK_NAME, default value is *, which means all stacks in all stages.
  • --profile: Override AWS profile specified in the config or environment variable AWS_PROFILE.
  • --region: Override AWS region specified in the config.
  • --artifact-store: Override bucket used for template transform/packaging specified in the config.
  • --verbose: Be more verbose.

Options can also be specified using environment variables:

CFN_STACK=Default.Table1 cfn-cli stack deploy

By default, cfn-cli tries to locate cfn-cli.yml or cfn-cli.yaml file in current directory, override this use -f.

Stack Selector

Individual stack can be selected using full qualified name:

cfn-cli -s Default.Table2 status

Or, select stacks use Unix globs:

cfn-cli -s Default.Table* status
cfn-cli -s Def*.Table1 status

If . is missing from stack selector, cfn-cli will assume stage name * is specified.

Commands

Use --help to see help on a particular command.

  • generate - Generate sample configuration file.
  • status - Print stack status and resources.
  • validate - Validate template file.
  • stack - Stack operations.
    • sync -Apply changes using ChangeSets
    • deploy - Deploy new stacks.
    • update - Update existing stacks.
    • tail - Print stack events.
    • delete - Delete stacks.
    • cancel - Cancel stack update.
  • drift - Drift detection.
    • detect - Detect stack drifts.
    • diff - Show stack resource drifts.

Auto Completion

Auto completion is supported by click_completion, supported shells are: bash, zsh , fish and Powershell.

To install auto completion, run this in target shell:

> cfn-cli --install-completion
fish completion installed in /Users/Bob/.config/fish/completions/cfn-cli.fish

Supported completion:

  • Commands and sub commands:

    > cfn-cli drift d<TAB><TAB> 
    detect  (Detect stack drifts.)  diff  (Show stack resource drifts.)
    
  • Options and parameters:

    > cfn-cli stack deploy --<TAB> <TAB>
    --disable-rollback  (Disable rollback if stack creation failed. You can specify ei…)
    --help                                                 (Show this message and exit.)
    --ignore-existing               (Don't exit with error if the stack already exists.)
    --no-wait                                (Exit immediately after deploy is started.)
    --on-failure  (Determines what action will be taken if stack creation fails. This …)
    --timeout-in-minutes  (The amount of time in minutes that can pass before the stac…)
    
  • Parameter choices:

    > cfn-cli stack deploy --on-failure <TAB> <TAB>
      DELETE  DO_NOTHING  ROLLBACK  
    
  • Dynamic complete for --profile by search profile name in awscli config:

    > cfn-cli -p <TAB><TAB>
    default
    prod
    staging
    
  • Dynamic complete for --stack by search stack name in cfn-cli config:

    > cfn-cli -s <TAB><TAB>
    Develop.ApiBackend-Develop           (ApiBackend-Develop)
    Production.ApiBackend-Production  (ApiBackend-Production)
    Staging.ApiBackend-Staging           (ApiBackend-Staging)
    

Automatic Packaging

If a template contains property which requires a S3 url or text block, Set stack Package parameter to True tells cfn-cli to package the resource automatically and upload to a S3 artifact bucket, and S3 object location is inserted into the resource location.

This feature is particular useful when your property is a lambda source code, SQL statements or some kind of configuration.

By default, the artifact bucket is awscfncli-${AWS_ACCOUNT_ID}-${AWS_RERION}, and it will be created automatically on first run. Override the default bucket using ArtifactStore parameter.

The following resource property are supported by awscfncli and official aws cloudformation package command:

  • BodyS3Location property for the AWS::ApiGateway::RestApi resource
  • Code property for the AWS::Lambda::Function resource
  • CodeUri property for the AWS::Serverless::Function resource
  • ContentUri property for the AWS::Serverless::LayerVersion resource
  • DefinitionS3Location property for the AWS::AppSync::GraphQLSchema resource
  • RequestMappingTemplateS3Location property for the AWS::AppSync::Resolver resource
  • ResponseMappingTemplateS3Location property for the AWS::AppSync::Resolver resource
  • DefinitionUri property for the AWS::Serverless::Api resource
  • Location parameter for the AWS::Include transform
  • SourceBundle property for the AWS::ElasticBeanstalk::ApplicationVersion resource
  • TemplateURL property for the AWS::CloudFormation::Stack resource
  • Command.ScriptLocation property for the AWS::Glue::Job resource

To package a template build by awssamcli, point Template parameter to sam build output.

Configuration

awscfncli uses a YAML config file to manage which stacks to deploy and how to deploy them. By default, it is cfn-cli.yml.

Anatomy

The config is composed of the following elements, Version, Stages and Blueprints.

  • Version (required): Version of cfn-cli config, support 2 and 3 now.
  • Stages (required): Definition of the stack to be deployed.
  • Blueprints (optional): Template of the stack.

The following is a simple example of a typical config:

Version: 3

Stages:
  Default:
    DDB:
      Template: DynamoDB_Table.yaml
      Region: us-east-1
      Parameters:
        HashKeyElementName: id
    DDB2ndIdx:
      Template: DynamoDB_Secondary_Indexes.yaml
      Region: us-east-1
      StackPolicy: stack_policy.json
      ResourceTypes:
        - AWS::DynamoDB::Table
      Parameters:
        ReadCapacityUnits: 10

A stage could have multiple stacks. In the above example, Stage Default have two stacks DDB and DDB2ndIdx. Stack name could be customized and should contain only alpha and numbers.

Each stack may have the following attributes.

  • Attributes introduced by awscfncli:
    • Profile: Profile name of your aws credential
    • Region: Eg. us-east-1
    • Package: Automatically package your template or not
    • ArtifactStore: Name of S3 bucket to store packaged files
    • Order: Deployment order of stacks
    • Extends: Extend a blueprint
  • Attributes introduced by boto3:

Blueprints and Inheritance

Blueprint serves as a template of a common stack. A stack could extends a stack and override its attributes with its own attributes.

  • Inheritance behaviors:

    • scalar value: replace
    • dict value: update
    • list value: extend
  • Special attributes:

    • Capabilities: replace

For example, please refer to Blueprints Example

Stages and Ordering

Stage and stacks could be deployed according to the order you specified. Order numbers are positive integers. cfn-cli will deploy stacks in stages with lower order first and in each stage stacks with lower order will be deployed first.

  • Stage Order
  • Stack Order
    Stages:
        Stage1:
            Order: 1
            Stack1:
                Order: 1
            Stack2:
                Order: 2
        Stage2:
            Order: 2

For examples, please refer to Order Example

Cross Stack Reference

In many cases, a stack's input parameter depends on output from other stacks during deployment. Cross stack reference allows stacks collect their inputs from outputs form other stacks, including stacks deployed to other region and account.

An stack parameter can reference ouputs of another stack in same configuration file by using the following syntax:

Stack1:
    Parameters:
        VpcId: ${StageName.StackName.OutputName}

This feature make managing related cross-account and/or cross-region stacks much easier. See VPC peering and CodePipeline for example.

Note: Take care of the order of deployment so eferenced stack is deployed first.

Breaking Changes in 3.0

Generally only major version changes cli and config syntax, and support of last config version is gaunteered.

CLI

  • stack describe is depecated, use status instead.
  • sync now defaults to --confirm, use --no-confirm to overwrite this.

Breaking Changes in 2.1

CLI

  • cfn is renamed to cfn-cli to avoid conflict with troposphere.
  • template command is removed.
  • changeset command is removed, replaced by sync command.
  • Because config file supports multiple stages and stacks, stack selector must be specified when you want to operate a subset of stacks.

Config

"Cross stack reference" feature requires version 3:

Version: 3
Stages:
  Default:
    ...

Parameter NotificationARNs, ResourceTypes, RollbackConfiguration are supported now but no changes is required if old config file is not using them.

Breaking Changes in 2.0

New configuration file supports multiple stages and stacks, to convert an 0.x configure file to current version, do following:

  1. Add following block to the head of conf file and indent the rest properly:
Version: 3
Stages:
  Default:
    << old config file >>
  1. Change any TemplateURL or TemplateBody parameter to Template:

Old:

Stack:
  TemplateURL:          https://s3.amazonaws.com/...
  Region:               us-east-1
  StackName:            SampleIAMUsersGroupsAndPolicies
  Capabilities:         [CAPABILITY_IAM]

New:

Version: 2
Stages:
  Default:
    Stack:
      Template:          https://s3.amazonaws.com/...
      Region:               us-east-1
      StackName:            SampleIAMUsersGroupsAndPolicies
      Capabilities:         [CAPABILITY_IAM]

awscfncli's People

Contributors

glieseray avatar kotaimen avatar lianqiu-cao-bose avatar mkielar avatar weiliangqian avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

awscfncli's Issues

YAML configure overwrite nested dictionary

Assume following YAML:

Blueprints:
  Foo: &Foo
    A: a
    B:
      C: c
Environments:
  Default:
    Bar:
      <<: *Foo
      B:
        D: d

Expected value for Bar.B is {'C':'c', 'D':'d'} but will get {'D':'d'} instead.
Seems YAML anchors not working as expected?

Support `cloudformation package`

We're working with nested stacks a lot, effectively having 50+ template files. So far, we've been using cloudformation package with --output-template-file option, followed by cloudformation update-stack pointed to the local file generated by package.

The problem is that cloudformation package generates file names that look like 3492974862e431dc045af45c71555b17.template, including the master template. This makes it really hard to point the awscfncli to such file. Instead I'd like to:

  • either be able to package & deploy in one go with awscfncli
  • or be able to point awscfncli with TemplateURL parameter to a file on local machine.

Move sync command to stack sync

The command sync actually works as another way to deploy a CloudFormation Template. So it may be better to put it under stack subcommand as an alternative way to deploy.

Should throw a ConfigError

When cfn-cli.yml contains an invalid config parameter, it should throw a ConfigError like "invalid parameter" instead of TypeError.

TypeError: _create_stack_config() got an unexpected keyword argument 'StackPolicyBody'

Improve error message when `changestack create` fails when no changes

When I run cfn changestack create on a simple stack (no nested stacks) and Cloudformation figures out there are no changes, it fails with:

Region: eu-west-2
Stack Name: dummy-stck
Template: C:\repositories\cloudformation\dummy-stack.yaml
Creating change set...
ChangeSet Name: mkitest1
ChangeSet ARN: arn:aws:cloudformation:eu-west-2:000000000000:changeSet/mkitest1/9189df70-92b0-49e3-97f7-d1fd173161b4
Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state
Aborted!

The thing is, that when I go to AWS Console => CloudFormation => dummy-stack => Change Sets => Click on mkitest1, I go into the change set details, and see the real reason for the failure, which is:

Status: FAILED - The submitted information didn't contain changes. Submit different information to create a change set.

I'd expect to see that error message returned by cfn, as currently there is no way of knowing if the changeset stack failed because of no changes, or because of something else...

Proposal for Cross account output parameter reference

Stages:

  Production:
    Order: 2
    DDBTable1:
      Order: 1
      Extends: DDB
      StackName: DynamoDbTable1
      Region: us-west-2
      Tags:
        Environment: staging
      Parameters:
        ReadCapacityUnits: 10
      Outputs:
        - OutputKeyInTheTemplateOfDDBTable1

    DDBTable2:
      Order: 2
      Extends: DDB2Idx
      StackName: DynamoDbTable1
      Region: us-east-1
      Tags:
        Environment: staging
      Parameters:
        ReadCapacityUnits: 10
        WriteCapacityUnits:
          !GetAtt Production.DDBTable1.Outputs.OutputKeyInTheTemplateOfDDBTable1

Use same syntax as AWS cloudformation.
Explicitly declare parameter output and reference

Initial create ChangeSet failure.

When creating new stack with ChangeSet, and the ChangeSet failed, the stack will be stuck in REVIEW_IN_PROGRESS state and the stack must be deleted.
In this situation,boto3 always "stack doesn't not exist" in create_change_set call, which is confusing.
We can add some code to recognize this situation and return a "stack state is invalid" error to the user.

New Proposal for the way to extend stack config

For now, awscfncli uses anchor of YAML to reuse configuration shared with multiple stacks. It works however the syntax looks a little tricky and not very straight forward. So Here is a proposal for a new way to share configuration with multiple stacks. For example:

Suppose we have a Config template:

StackConfigTemplate:
  Region: us-east-1
  Tags:
    Project:  Bob
    CostUnit: 180

And we are going to extend this template:

Stack1:
  Extends: StackConfigTemplate
  Region: us-west-1
  Tags:
    CostUnit: 190

Then the result stack config will be:

Stack1:
  Region: us-west-1
  Tags:
    Project:  Bob
    CostUnit: 190

General rule for extension:

  1. Scalar: Replace
  2. Array: Append
  3. Dictionary: Merge

For properties such as Capabilities, special actions will be taken to ensure the completeness of extension. (For Capabilities, maybe replace is a more appropriate approach.)

cfn-cli no longer working with latest version of awscli (version 1.16.26)

Hi,

Workaround

If you are using virtualenv, you can specify the following versions in requirements.txt. This will fix this issue for now.

awscfncli2==2.0.1
awscli==1.16.21
boto3==1.9.11
botocore==1.12.11

If you are using user level or system level python packages, running the following commands will fix this issue for now.

pip install awscfncli2==2.0.1
pip install awscli==1.16.21
pip install boto3==1.9.11
pip install botocore==1.12.11

Issue

Using awscfncli 2.0.1 has been working great, until recently when my version of awscli got upgraded to the latest version of 1.16.26 Now awscfncli is throwing the following error (see below)

It was working ok at least until awscli 1.16.21

#$ cfn-cli --profile myprofile -f cfn/mycfnfile.yml status

Traceback (most recent call last):
  File "/home/user/workspace/myproject/.venv/bin/cfn-cli", line 7, in <module>
    from awscfncli2.__main__ import main
  File "/home/user/workspace/myproject/.venv/lib/python3.6/site-packages/awscfncli2/__main__.py", line 26, in <module>
    from .cli import cfn_cli
  File "/home/user/workspace/myproject/.venv/lib/python3.6/site-packages/awscfncli2/cli/__init__.py", line 2, in <module>
    from .main import cfn_cli
  File "/home/user/workspace/myproject/.venv/lib/python3.6/site-packages/awscfncli2/cli/main.py", line 10, in <module>
    from .utils.context import ContextObject
  File "/home/user/workspace/myproject/.venv/lib/python3.6/site-packages/awscfncli2/cli/utils/__init__.py", line 8, in <module>
    from .package import run_packaging
  File "/home/user/workspace/myproject/.venv/lib/python3.6/site-packages/awscfncli2/cli/utils/package.py", line 10, in <module>
    from awscli.customizations.cloudformation.artifact_exporter import Template, \
ImportError: cannot import name 'EXPORT_DICT'

list stacks command

I love awscfncli! Thanks so much for making this!

I'd like a way to list existing stacks (whether or not they have stack configs). The AWS CLI's list-stacks shows all stacks that ever existed and it's not very useful.

Exception when specified TerminationProtection parameter

Exception raised:

NotImplementedError: Termination protection is not supported for current version of boto. Please upgrade to a new version.

Condition:

  • Specified TerminationProtection parameter,
  • Executing cfn-cli sync command,
  • And the stack is not yet created (changeset_type == 'CREATE')

Confirm region/profile config inhertince order

@GlieseRay We need to confirm configuration inheritance order during testing. I'm not sure current situation works as expected.

region config order (last wins):

  • region config in awscli named profile, which defaults to the AWS account's home region
  • AWS_REGION environment variable
  • cfn-cli config file
  • CFN_REGION environment variable
  • cfn-cli option --region

profile config order (last wins):

  • default
  • AWS_PROFILE environment variable
  • Profile parameter in cfn-cli config file
  • CFN_PROFILE environment variable
  • cfn-cli option --profile

Parameter Override Issue

Blueprints:
  Bob180:
    Foo: bar

Stages:
  Default:
    Bob190:
      Extends: Bob180
      Parameters:
        Bob: 190

    Bob200:
      Extends: Bob180
      Parameters:
        Bob: 200

Parameters will both be set to Bob190 in this configuration file.
Suspecting this has something to do with "extends" logic.

Support ordering when execute command on multiple stacks

@GlieseRay Originally my plan is defer this feature to 2.1 and discover the stack dependency automatically. However this is a must have feature when deploy a set of related stacks in a single stage, otherwise the "stack select" option is less useful since you have to deploy separate stacks manually.

Now I opt for manually specify deployment order of stacks and stage in the configuration file, the order is decided by:

  1. DeployOrder parameter of stage and stack,
  2. Order of stage name and stack name.

For example:

Stages:

  Foundation:
    DeployOrder: 1  
    VPC:

  Develop:
    DeployOrder: 2
      Database: 
        DeployOrder: 1
      Service: 
        DeployOrder: 2

  Production:
    DeployOrder: 3
      1_Database: 
         Parameter: Value
      2_Service: 
         Parameter: Value

Deployment order for the config is:

  1. Foundation.VPC
  2. Develop.Database
  3. Develop.Service
  4. Production.1_Database
  5. Production.2_Service

New CLI

Purposing new CLI:

Before:

cfn stack update --override-policy=ALLOW_ALL

After:

cfn-cli -f CONFIG_FILE -s STACK_SELECTOR update --override-policy=ALLOW_ALL

Major changes:

  • cfn -> cfn-cli
  • Config file and stack selection move to click options
  • Use a stack selector to select stacks in the configuration file
    • Purposing STAGE_PATTERN.STACK_PATTERN
    • Default is Default.*
  • "Collapse" command subcommand structure to use only commands, eg:
    • stack deploy -> deploy
    • stack update -> update
    • changeset sync -> sync
  • Autocomplete (depends on click support)

Deal with same StackName with different profile in the same stage.

Currently, stacks should have different name in the same stage. However, in some circumstance, stacks with different profile should be allowed to have same name. (eg. different regions, accounts).

To deal with these conditions, here are some proposals for the config schema:

  1. Use list instead of dict
Stages:
  Dev:
    - StackName: StackA
      Region: us-east-1
      Profile: bob
    - StackName: StackA
      Region: us-east-2
      Profile: ray
  1. Use a name space:
Stages:
  Dev:
    bob.us-east-1.StackA:
      Template: test.template.yaml
    ray.us-east-2.StackA:
      Template: test.template.yaml

Looks like solution 1 is more simple and straight forward.

Return proper `ERRORLEVEL` / `$?` values

Let's say I'm running the command like this:

cfn changeset execute ./stack-config.yaml dummy-changeset

Then it fails with:

Region: eu-west-2
Stack Name: dev-stack
Template: C:\repositories\cloudformation\dev\master.yaml.packaged
An error occurred (ChangeSetNotFound) when calling the ExecuteChangeSet operation: ChangeSet [dummy-changeset] does not exist

The issue is, the %ERRORLEVEL% is 0 in that case in cmd.exe, and similarly under bash, the $? is also 0. This makes it hard to implement fail-fast bash scripts...

Above has been tested on Python 3.6.4 in Windows 10 with cmd.exe and Cygwin Bash.

awscfncli2/config/schema_v2.json is missing in pip package

Hi,

Steps to re-create the issue:

  1. Install awscfncli2 by running pip install --upgrade --user awscfncli2
  2. Run the following cfn-cli command: cfn-cli --profile myawsprofile -f ./cfnconfig.yml stack deploy
  3. The following error is received:
FileNotFoundError: [Errno 2] No such file or directory: '/home/user/.local/lib/python3.5/site-packages/awscfncli2/config/schema_v2.json'

Upon further investigation, the files awscfncli2/config/schema_v1.json and awscfncli2/config/schema_v2.json are in the master branch in github for this project, but they are missing in in pypi (at least they are missing when I download the 2.0.0.tar.gz from here and check inside)

I've attached the full traceback in the traceback.txt file.
traceback.txt

Costs

The AWS CLI has an estimate-template-costs command, and I think it would be awesome to be able to do cfn stack costs that would:

  • estimate costs of stacks using the estimate-template-costs api
  • evaluate actual costs incurred by the given stack (even if just on a monthly basis).

I could probably implement the estimates, but I don't know enough about the AWS billing apis to know if the second part is even possible, any ideas?

Add missing stack configure parameters.

Unsupported parameters as of now:

  • StackPolicyURL - StackPolicy supports local file now
  • ResourceTypes
  • NotificationARNs
  • RollbackConfiguration
  • Also supports Disable Rollback and 'Timeout` in cli option, which overrides config

Support new custom packaging:

  • AWS::Athena::NamedQuery
  • DefinitionBody property for AWS::Serverless::Api resource
  • PolicyDocument property for IAM resources
  • DefinitionString property for AWS::StepFunctions::StateMachine resource
  • Sql property for IoT, Kinesis analytics and Glue resource

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.