kotaimen / awscfncli Goto Github PK

The command sync actually works as another way to deploy a CloudFormation Template. So it may be better to put it under stack subcommand as an alternative way to deploy.

Should printout resolved value of SSM parameter.

Also cleanup some very chatty config documentation by ray

Assume following YAML:

Blueprints:
  Foo: &Foo
    A: a
    B:
      C: c
Environments:
  Default:
    Bar:
      <<: *Foo
      B:
        D: d

Expected value for Bar.B is {'C':'c', 'D':'d'} but will get {'D':'d'} instead.
Seems YAML anchors not working as expected?

For now, awscfncli uses anchor of YAML to reuse configuration shared with multiple stacks. It works however the syntax looks a little tricky and not very straight forward. So Here is a proposal for a new way to share configuration with multiple stacks. For example:

Suppose we have a Config template:

StackConfigTemplate:
  Region: us-east-1
  Tags:
    Project:  Bob
    CostUnit: 180

And we are going to extend this template:

Stack1:
  Extends: StackConfigTemplate
  Region: us-west-1
  Tags:
    CostUnit: 190

Then the result stack config will be:

Stack1:
  Region: us-west-1
  Tags:
    Project:  Bob
    CostUnit: 190

General rule for extension:

1. Scalar: Replace
2. Array: Append
3. Dictionary: Merge

For properties such as Capabilities, special actions will be taken to ensure the completeness of extension. (For Capabilities, maybe replace is a more appropriate approach.)

For example

EnableTerminationProtection: True (not work)
EnableTerminationProtection: true (works)

cfn-cli stack delete will delete all stacks in all stages, add a warning requires user confirm by default and a --quiet argument to suppress the warning.

• AWS::Athena::NamedQuery
• DefinitionBody property for AWS::Serverless::Api resource
• PolicyDocument property for IAM resources
• DefinitionString property for AWS::StepFunctions::StateMachine resource
• Sql property for IoT, Kinesis analytics and Glue resource

Unsupported parameters as of now:

• StackPolicyURL - StackPolicy supports local file now
• ResourceTypes
• NotificationARNs
• RollbackConfiguration
• Also supports Disable Rollback and 'Timeout` in cli option, which overrides config

@GlieseRay Originally my plan is defer this feature to 2.1 and discover the stack dependency automatically. However this is a must have feature when deploy a set of related stacks in a single stage, otherwise the "stack select" option is less useful since you have to deploy separate stacks manually.

Now I opt for manually specify deployment order of stacks and stage in the configuration file, the order is decided by:

1. DeployOrder parameter of stage and stack,
2. Order of stage name and stack name.

For example:

Stages:

  Foundation:
    DeployOrder: 1  
    VPC:

  Develop:
    DeployOrder: 2
      Database: 
        DeployOrder: 1
      Service: 
        DeployOrder: 2

  Production:
    DeployOrder: 3
      1_Database: 
         Parameter: Value
      2_Service: 
         Parameter: Value

Deployment order for the config is:

1. Foundation.VPC
2. Develop.Database
3. Develop.Service
4. Production.1_Database
5. Production.2_Service

• decide how environment variable works
• optional: RollbackConfiguration
• optional: automatically handle SQL
• optional: Transform::Include

Stages:

  Production:
    Order: 2
    DDBTable1:
      Order: 1
      Extends: DDB
      StackName: DynamoDbTable1
      Region: us-west-2
      Tags:
        Environment: staging
      Parameters:
        ReadCapacityUnits: 10
      Outputs:
        - OutputKeyInTheTemplateOfDDBTable1

    DDBTable2:
      Order: 2
      Extends: DDB2Idx
      StackName: DynamoDbTable1
      Region: us-east-1
      Tags:
        Environment: staging
      Parameters:
        ReadCapacityUnits: 10
        WriteCapacityUnits:
          !GetAtt Production.DDBTable1.Outputs.OutputKeyInTheTemplateOfDDBTable1

Use same syntax as AWS cloudformation.
Explicitly declare parameter output and reference

The website https://kotaimen.github.io/awscfncli/ containing the awscfncli Tutorial documentation is throwing a 404 error.

TerminationProtection is enabled before stack creation completes thus if stack deployment failed user. have to manually disable it in the Console.

...and probable cleanup the sample config in the process.

Hi,

Workaround

If you are using virtualenv, you can specify the following versions in requirements.txt. This will fix this issue for now.

awscfncli2==2.0.1
awscli==1.16.21
boto3==1.9.11
botocore==1.12.11

If you are using user level or system level python packages, running the following commands will fix this issue for now.

pip install awscfncli2==2.0.1
pip install awscli==1.16.21
pip install boto3==1.9.11
pip install botocore==1.12.11

Issue

Using awscfncli 2.0.1 has been working great, until recently when my version of awscli got upgraded to the latest version of 1.16.26 Now awscfncli is throwing the following error (see below)

It was working ok at least until awscli 1.16.21

#$ cfn-cli --profile myprofile -f cfn/mycfnfile.yml status

Traceback (most recent call last):
  File "/home/user/workspace/myproject/.venv/bin/cfn-cli", line 7, in <module>
    from awscfncli2.__main__ import main
  File "/home/user/workspace/myproject/.venv/lib/python3.6/site-packages/awscfncli2/__main__.py", line 26, in <module>
    from .cli import cfn_cli
  File "/home/user/workspace/myproject/.venv/lib/python3.6/site-packages/awscfncli2/cli/__init__.py", line 2, in <module>
    from .main import cfn_cli
  File "/home/user/workspace/myproject/.venv/lib/python3.6/site-packages/awscfncli2/cli/main.py", line 10, in <module>
    from .utils.context import ContextObject
  File "/home/user/workspace/myproject/.venv/lib/python3.6/site-packages/awscfncli2/cli/utils/__init__.py", line 8, in <module>
    from .package import run_packaging
  File "/home/user/workspace/myproject/.venv/lib/python3.6/site-packages/awscfncli2/cli/utils/package.py", line 10, in <module>
    from awscli.customizations.cloudformation.artifact_exporter import Template, \
ImportError: cannot import name 'EXPORT_DICT'

Hi,

Steps to re-create the issue:

1. Install awscfncli2 by running pip install --upgrade --user awscfncli2
2. Run the following cfn-cli command: cfn-cli --profile myawsprofile -f ./cfnconfig.yml stack deploy
3. The following error is received:

FileNotFoundError: [Errno 2] No such file or directory: '/home/user/.local/lib/python3.5/site-packages/awscfncli2/config/schema_v2.json'

Upon further investigation, the files awscfncli2/config/schema_v1.json and awscfncli2/config/schema_v2.json are in the master branch in github for this project, but they are missing in in pypi (at least they are missing when I download the 2.0.0.tar.gz from here and check inside)

I've attached the full traceback in the traceback.txt file.
traceback.txt

Proposal:

Defaults:
    Vpc:
        Profile: default
Environments:
    staging:
        Vpc1:
            from: Vpc
        Vpc2:
            from: Vpc

Currently, stacks should have different name in the same stage. However, in some circumstance, stacks with different profile should be allowed to have same name. (eg. different regions, accounts).

To deal with these conditions, here are some proposals for the config schema:

1. Use list instead of dict

Stages:
  Dev:
    - StackName: StackA
      Region: us-east-1
      Profile: bob
    - StackName: StackA
      Region: us-east-2
      Profile: ray

2. Use a name space:

Stages:
  Dev:
    bob.us-east-1.StackA:
      Template: test.template.yaml
    ray.us-east-2.StackA:
      Template: test.template.yaml

Looks like solution 1 is more simple and straight forward.

An error occurred (ValidationError) when calling the CreateChangeSet operation: TemplateURL must be an Amazon S3 URL is reported instead of "temple file not found" when Template parameter set to a local SAM template which does not exist.

When creating new stack with ChangeSet, and the ChangeSet failed, the stack will be stuck in REVIEW_IN_PROGRESS state and the stack must be deleted.
In this situation,boto3 always "stack doesn't not exist" in create_change_set call, which is confusing.
We can add some code to recognize this situation and return a "stack state is invalid" error to the user.

For now, we only support .yml

Artifact bucket should be created automatically if it does not exist.
(same behavior as previous version)

An error occurred (Throttling) when calling the DescribeStackEvents operation (reached max retries: 4): Rate exceeded

Let's say I'm running the command like this:

cfn changeset execute ./stack-config.yaml dummy-changeset

Then it fails with:

Region: eu-west-2
Stack Name: dev-stack
Template: C:\repositories\cloudformation\dev\master.yaml.packaged
An error occurred (ChangeSetNotFound) when calling the ExecuteChangeSet operation: ChangeSet [dummy-changeset] does not exist

The issue is, the %ERRORLEVEL% is 0 in that case in cmd.exe, and similarly under bash, the $? is also 0. This makes it hard to implement fail-fast bash scripts...

Above has been tested on Python 3.6.4 in Windows 10 with cmd.exe and Cygwin Bash.

Blueprints:
  Bob180:
    Foo: bar

Stages:
  Default:
    Bob190:
      Extends: Bob180
      Parameters:
        Bob: 190

    Bob200:
      Extends: Bob180
      Parameters:
        Bob: 200

Parameters will both be set to Bob190 in this configuration file.
Suspecting this has something to do with "extends" logic.

Exception raised:

NotImplementedError: Termination protection is not supported for current version of boto. Please upgrade to a new version.

Condition:

• Specified TerminationProtection parameter,
• Executing cfn-cli sync command,
• And the stack is not yet created (changeset_type == 'CREATE')

@GlieseRay plz fix ASAP.

Docker build files only tests --version thus this is slipped from testing. I'll start a hot fix branch and fix test first.

@GlieseRay We need to confirm configuration inheritance order during testing. I'm not sure current situation works as expected.

region config order (last wins):

• region config in awscli named profile, which defaults to the AWS account's home region
• AWS_REGION environment variable
• cfn-cli config file
• CFN_REGION environment variable
• cfn-cli option --region

profile config order (last wins):

• default
• AWS_PROFILE environment variable
• Profile parameter in cfn-cli config file
• CFN_PROFILE environment variable
• cfn-cli option --profile

We're working with nested stacks a lot, effectively having 50+ template files. So far, we've been using cloudformation package with --output-template-file option, followed by cloudformation update-stack pointed to the local file generated by package.

The problem is that cloudformation package generates file names that look like 3492974862e431dc045af45c71555b17.template, including the master template. This makes it really hard to point the awscfncli to such file. Instead I'd like to:

• either be able to package & deploy in one go with awscfncli
• or be able to point awscfncli with TemplateURL parameter to a file on local machine.

Validate not working when checking a SAM template for China region.
(template is general but the cloudformation api seems only supports us-east-1 region).

The AWS CLI has an estimate-template-costs command, and I think it would be awesome to be able to do cfn stack costs that would:

• estimate costs of stacks using the estimate-template-costs api
• evaluate actual costs incurred by the given stack (even if just on a monthly basis).

I could probably implement the estimates, but I don't know enough about the AWS billing apis to know if the second part is even possible, any ideas?

When I run cfn changestack create on a simple stack (no nested stacks) and Cloudformation figures out there are no changes, it fails with:

Region: eu-west-2
Stack Name: dummy-stck
Template: C:\repositories\cloudformation\dummy-stack.yaml
Creating change set...
ChangeSet Name: mkitest1
ChangeSet ARN: arn:aws:cloudformation:eu-west-2:000000000000:changeSet/mkitest1/9189df70-92b0-49e3-97f7-d1fd173161b4
Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state
Aborted!

The thing is, that when I go to AWS Console => CloudFormation => dummy-stack => Change Sets => Click on mkitest1, I go into the change set details, and see the real reason for the failure, which is:

Status: FAILED - The submitted information didn't contain changes. Submit different information to create a change set.

I'd expect to see that error message returned by cfn, as currently there is no way of knowing if the changeset stack failed because of no changes, or because of something else...

• Process of convert config to boto3 call
• Abstract stack Command objects
• Separate boto3 call from click call

EnableTerminationProtection parameter is available in http://boto3.readthedocs.io/en/latest/reference/services/cloudformation.html#CloudFormation.Client.create_stack but not in stack configuration structure.

When cfn-cli.yml contains an invalid config parameter, it should throw a ConfigError like "invalid parameter" instead of TypeError.

TypeError: _create_stack_config() got an unexpected keyword argument 'StackPolicyBody'

Purposing new CLI:

Before:

cfn stack update --override-policy=ALLOW_ALL

After:

cfn-cli -f CONFIG_FILE -s STACK_SELECTOR update --override-policy=ALLOW_ALL

Major changes:

• cfn -> cfn-cli
• Config file and stack selection move to click options
• Use a stack selector to select stacks in the configuration file
  • Purposing STAGE_PATTERN.STACK_PATTERN
  • Default is Default.*
• "Collapse" command subcommand structure to use only commands, eg:
  • stack deploy -> deploy
  • stack update -> update
  • changeset sync -> sync
• Autocomplete (depends on click support)

I love awscfncli! Thanks so much for making this!

I'd like a way to list existing stacks (whether or not they have stack configs). The AWS CLI's list-stacks shows all stacks that ever existed and it's not very useful.