remind101 / stacker_blueprints Goto Github PK

Right now DockerImage is set to master which is doesn't do anything.

We should set this to a stable version of empire or make it a required parameter.

just a preference thing, i don't really like _ in repo names and always go with -

Reopen from Stacker issue:

I got this error using the latest released version and latest master version. Not exactly sure where to look to fix this issue. The same configuration and environment files worked two months ago.

[2017-09-17T12:09:25] INFO stacker.commands.stacker:32(configure): Using Default AWS Provider
[2017-09-17T12:09:25] DEBUG stacker.plan:288(outline): Plan "Create/Update stacks":
[2017-09-17T12:09:25] DEBUG stacker.plan:296(outline): - step: 1: target: "vdt-dev-vpc", action: "_launch_stack"
[2017-09-17T12:09:25] DEBUG stacker.plan:296(outline): - step: 2: target: "vdt-dev-empireDB", action: "_launch_stack"
[2017-09-17T12:09:25] DEBUG stacker.plan:296(outline): - step: 3: target: "vdt-dev-empireController", action: "_launch_stack"
[2017-09-17T12:09:25] DEBUG stacker.plan:296(outline): - step: 4: target: "vdt-dev-bastion", action: "_launch_stack"
[2017-09-17T12:09:25] DEBUG stacker.plan:296(outline): - step: 5: target: "vdt-dev-empireMinion", action: "_launch_stack"
[2017-09-17T12:09:25] DEBUG stacker.plan:296(outline): - step: 6: target: "vdt-dev-empireDaemon", action: "_launch_stack"
[2017-09-17T12:09:25] DEBUG stacker.actions.build:303(run): Launching stacks: vdt-dev-vpc, vdt-dev-empireDB, vdt-dev-empireController, vdt-dev-bastion, vdt-dev-empireMinion, vdt-dev-empireDaemon
[2017-09-17T12:09:25] INFO stacker.plan:358(_check_point): Plan Status:
[2017-09-17T12:09:25] INFO stacker.plan:380(_check_point): vdt-dev-vpc: pending
[2017-09-17T12:09:25] INFO stacker.plan:380(_check_point): vdt-dev-empireDB: pending
[2017-09-17T12:09:25] INFO stacker.plan:380(_check_point): vdt-dev-empireController: pending
[2017-09-17T12:09:25] INFO stacker.plan:380(_check_point): vdt-dev-bastion: pending
[2017-09-17T12:09:25] INFO stacker.plan:380(_check_point): vdt-dev-empireMinion: pending
[2017-09-17T12:09:25] INFO stacker.plan:380(_check_point): vdt-dev-empireDaemon: pending
[2017-09-17T12:09:25] DEBUG stacker.util:58(retry_with_backoff): Calling <bound method CloudFormation.describe_stacks of <botocore.client.CloudFormation object at 0x103c193d0>>, attempt 1.
[2017-09-17T12:09:25] DEBUG stacker.actions.build:231(_launch_stack): Resolving stack vdt-dev-vpc
[2017-09-17T12:09:25] DEBUG stacker.actions.build:234(_launch_stack): Launching stack vdt-dev-vpc now.
[2017-09-17T12:09:25] DEBUG stacker.blueprints.base:448(import_mappings): Adding mapping AmiMap.
[2017-09-17T12:09:25] DEBUG stacker.blueprints.base:369(setup_parameters): No parameters defined.
Traceback (most recent call last):
File "/Users//dev/projects/viditure/vt_env/bin/stacker", line 4, in
import('pkg_resources').run_script('stacker==1.0.4', 'stacker')
File "/Users//dev/projects/viditure/vt_env/lib/python2.7/site-packages/pkg_resources/init.py", line 748, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Users//dev/projects/viditure/vt_env/lib/python2.7/site-packages/pkg_resources/init.py", line 1524, in run_script
exec(script_code, namespace, namespace)
File "/Users//dev/projects/viditure/vt_env/lib/python2.7/site-packages/stacker-1.0.4-py2.7.egg/EGG-INFO/scripts/stacker", line 9, in

File "build/bdist.macosx-10.12-intel/egg/stacker/commands/stacker/build.py", line 52, in run
File "build/bdist.macosx-10.12-intel/egg/stacker/actions/base.py", line 127, in execute
File "build/bdist.macosx-10.12-intel/egg/stacker/actions/build.py", line 304, in run
File "build/bdist.macosx-10.12-intel/egg/stacker/plan.py", line 267, in execute
File "build/bdist.macosx-10.12-intel/egg/stacker/plan.py", line 227, in _single_run
File "build/bdist.macosx-10.12-intel/egg/stacker/plan.py", line 74, in run
File "build/bdist.macosx-10.12-intel/egg/stacker/actions/build.py", line 235, in _launch_stack
File "build/bdist.macosx-10.12-intel/egg/stacker/actions/base.py", line 101, in s3_stack_push
File "build/bdist.macosx-10.12-intel/egg/stacker/actions/base.py", line 25, in stack_template_key_name
File "build/bdist.macosx-10.12-intel/egg/stacker/blueprints/base.py", line 491, in version
File "build/bdist.macosx-10.12-intel/egg/stacker/blueprints/base.py", line 461, in render_template
File "/Users//dev/projects/viditure/vt_env/lib/python2.7/site-packages/troposphere-1.9.5-py2.7.egg/troposphere/init.py", line 609, in to_json
return json.dumps(self.to_dict(), indent=indent,
File "/Users//dev/projects/viditure/vt_env/lib/python2.7/site-packages/troposphere-1.9.5-py2.7.egg/troposphere/init.py", line 606, in to_dict
return encode_to_dict(t)
File "/Users//dev/projects/viditure/vt_env/lib/python2.7/site-packages/troposphere-1.9.5-py2.7.egg/troposphere/init.py", line 63, in encode_to_dict
props[name] = encode_to_dict(prop)
File "/Users//dev/projects/viditure/vt_env/lib/python2.7/site-packages/troposphere-1.9.5-py2.7.egg/troposphere/init.py", line 63, in encode_to_dict
props[name] = encode_to_dict(prop)
File "/Users//dev/projects/viditure/vt_env/lib/python2.7/site-packages/troposphere-1.9.5-py2.7.egg/troposphere/init.py", line 54, in encode_to_dict
return encode_to_dict(obj.to_dict())
File "/Users//dev/projects/viditure/vt_env/lib/python2.7/site-packages/troposphere-1.9.5-py2.7.egg/troposphere/init.py", line 226, in to_dict
self.validate()
File "/Users//dev/projects/viditure/vt_env/lib/python2.7/site-packages/troposphere-1.9.5-py2.7.egg/troposphere/ec2.py", line 344, in validate
exactly_one(self.class.name, self.properties, gateway_conds)
File "/Users//dev/projects/viditure/vt_env/lib/python2.7/site-packages/troposphere-1.9.5-py2.7.egg/troposphere/validators.py", line 201, in exactly_one
specified_count = mutually_exclusive(class_name, properties, conditionals)
File "/Users//dev/projects/viditure/vt_env/lib/python2.7/site-packages/troposphere-1.9.5-py2.7.egg/troposphere/validators.py", line 196, in mutually_exclusive
class_name, ', '.join(conditionals)))
ValueError: Route: only one of the following can be specified: EgressOnlyInternetGatewayId, GatewayId, InstanceId, NatGatewayId, NetworkInterfaceId, VpcPeeringConnectionId

Since cloudformation and troposphere support nat gateways now, should they be used instead of nat instances? I'm working on trying to modify vpc.py to support that, but I keep doing something wrong, so I don't have a pull request ready yet. Is this a direction in which you are moving, or are you planning on sticking with nat instances?

Also, thanks for doing some great work with this and empire!

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-elasticache-parameter-group.html

It's just confusing at this point. Need to think about existing examples (empire) that use the old rds.

I was going to create an s3 stack, but I don't see how to create the role that setting ReadWriteRoles would add a policy to.

I can work on a PR for this. Does it make more sense to have a blueprint that creates a bunch of ec2 and/or lambda roles, or just a single one.

Hello guys,
First of all, thank you for the excellent work.

Today I tried to change an old stack of mine and got the following error:

File "...github.com_remind101_stacker_blueprints/stacker_blueprints/sns.py", line 136, in create_topic

sqs_subs = [sub for sub in topic_subs if sub["Protocol"] == "sqs"]
TypeError: 'NoneType' object is not iterable

Here is my stack:

- name: sns                                                                   
  class_path: stacker_blueprints.sns.Topics                                   
  enabled: true                                                               
  variables:                                                                  
    Topics:                                                                   
      LifeCycleHookTopic:                                                     
      DisplayName: "cassandra-lifecycle-hook"

If I change the stack to define an empty Subscription list, it works

- name: sns                                                                   
  class_path: stacker_blueprints.sns.Topics                                   
  enabled: true                                                               
  variables:                                                                  
    Topics:                                                                   
      LifeCycleHookTopic:                                                     
      DisplayName: "cassandra-lifecycle-hook"
      Subscription: []

I don't think Ref("PublicSubnets") is sufficiently configurable. I'm going to work on a patch for this too.

As an RDS user
I'd like to get io1 or gp2 disks depending on if iops is declared
So that I don't use slow spinning disks

Current:
When StorageType is not set, and iops == 0, magnetic disks are provisioned

Expected:
When StorageType is not set, and iops == 0, gp2 SSD disks are provisioned

I'm guessing this is RDS default behavior for Ref("AWS::NoValue") I'd recommend using something like StorageType = variables['StorageType'] or 'io1' if get_piops() > 0 else 'gp2'

When attempting to upgrade a Postgres RDS instance from 9.5.x to 9.6.1, the stack updates the DBParameterGroup and OptionGroup, but then proceeds to fail on AWS::RDS::DBInstance with the following error:

The Parameter Group stage-classchirp-com-shortymasterdb-parametergroup-z95umv2qmwhi with DBParameterGroupFamily postgres9.5 cannot be used for this instance. Please use a Parameter Group with DBParameterGroupFamily postgres9.6

This is despite having the family set to postgres9.6.

MariaDB is now supported as a DB engine, but there is no corresponding blueprint.

Should have the same rules as stacker

Looks like the elasticache package only supports redis

As a network admin
I want to expose the bastion security group as an output
So I can pass it into other stacks which add ingress rules allowing traffic from the bastions

The template conditionally does not create the ParameterGroup and OptionGroup based on whether the aforementioned variables are empty, but refers to them anyway and causes errors:

botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the CreateStack operation: Template format error: Unresolved resource dependencies [OptionGroup, ParameterGroup] in the Resources block of the template

Result:

API: ec2:disassociateAddress You do not have permission to access the specified resource.

--
Template ends up in a 'UPDATE_ROLLBACK_COMPLETE' state.

Dupe the issue.

Deploy VPC template with UseNatGateway: True, then attempt to do a stacker build with UseNatGateway: False

As a lazy admin
I want a predicable DNS name for my bastion box(s)
So that I have a reliable endpoint to connect to.

I'm not sure how to accomplish this, especially for users that choose to run more than one bastion. You'd likely have to add an ELB forwarding port 22, and deal with host keys.

The default cloudformation dynamodb stack is really rigid. Updating AttributeDefinitions requires a replacement: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dynamodb-table.html#cfn-dynamodb-table-attributedef and you can't have a global secondary index that references a non-attribute definition.

You CAN update a table and add attribute definitions/GSIs with the boto3 api: http://boto3.readthedocs.io/en/latest/reference/services/dynamodb.html#DynamoDB.Client.update_table.

A custom resource that just piped args to the create_table and update_table would make it much easier to evolve dynamodb schema.

I think we should have some standards around parameters and outputs.

ie. sometimes a security group is called SecurityGroup other times its EmpireMinionSG.

Maybe there is another way to do this, but this should allow other stacks to give the minion permission to access resources.

The specific thing I'm trying to handle: I have an SNS topic I want to allow a service running within Empire to publish to without having to pass AWS credentials via. environmental variables to the task.

I tried setting timeout: <some val> in two different regions with ClusterParameters and both threw an Internal Error within CF. Have you ever set anything with this?

we should allow passing a value for: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-desiredcapacity