cloudfoundry / bosh-bootloader Goto Github PK

Command line utility for standing up a BOSH director on an IAAS of your choice.

License: Apache License 2.0

Go 77.11% Shell 5.01% HCL 17.51% Dockerfile 0.37%

bosh-bootloader's Introduction

bosh-bootloader

Also known as bbl (pronounced: "bubble"), bosh-bootloader is a command line utility for standing up BOSH on an IaaS. bbl currently supports AWS, GCP, Microsoft Azure, Openstack and vSphere.

CI
Tracker

Docs

Prerequisites

Install Dependencies

The following should be installed on your local machine

bosh-cli
bosh create-env dependencies
terraform >= 0.11.0
ruby (necessary for bosh create-env)

Install bosh-bootloader using a package manager

Mac OS X

$ brew tap cloudfoundry/tap
$ brew install bosh-cli
$ brew install bbl

Usage

IaaS-Specific Getting Started Guides

Managing state

The bbl state directory contains all of the files that were used to create your bosh director. You will need the entire bbl state in order to update, delete, or run bosh commands against the environment, so you should keep it someone anyone on your team can access it. However, it is important to note that the bbl state directory contains very senstive IAAS credentials, so you should keep it somewhere you feel comfortable storing these credentials, such as an encrypted bucket.

filename	contents
`bbl-state.json`	Environment name, and bbl version metadata
`terraform/`	The terraform templates bbl used to pave your IaaS. See docs/advanced-configuration for information on modifying this.
`vars/`	This is where bbl will store environment specific variables. Consider storing this outside of version control.
`jumpbox-deployment/`	The latest jumpbox-deployment that has been tested with your version of bbl.
`create-jumpbox.sh`	The BOSH cli command bbl will use to create your jumpbox.
`bosh-deployment/`	The latest bosh-deployment that has been tested with your version of bbl
`create-director.sh`	The BOSH cli command bbl will use to create your director when you run `bbl up`. See docs/advanced-configuration for help with modifying this.
`cloud-config/`	The cloud-config yaml that bbl will upload to the director to map IAAS resources to BOSH resources.
`delete-director.sh`	The BOSH cli command bbl will use to delete your director.
`delete-jumpbox.sh`	The BOSH cli command bbl will use to delete your jumpbox.

Troubleshooting

To turn on debugging traces:

BBL: use bbl --debug flag.
BOSH: export the BOSH_LOG_LEVEL=debug and the BOSH_LOG_PATH=bosh.log environment variables in the related bbl shell script, such as create-jumpbox.sh.
TERRAFORM: export the TF_LOG=TRACE and the TF_LOG_PATH=terraform.log environment variables in the related bbl shell script, such as create-jumpbox.sh.
CPI: override or set cpi configuration, e.g. for vsphere cpi within jumpbox-deployment/vsphere/cpi.yml set http_logging: true according to vsphere cpi documentation

Tearing down an environment

Once you are done kicking the tires on CF and BOSH, clean up your environment to save IaaS costs:

You must first delete any deployments on BOSH. e.g. bosh -d cf delete-deployment
bbl destroy with your IaaS user/account information.

Automating the automation tool

In order to use bbl in your concourse pipelines, the current supported way for cf-deployment is to use the cf-deployment-concourse-tasks.

There is a work-in-progress concourse resource for bbl: bbl-state-resource.

bosh-bootloader's People

Contributors

Stargazers

Watchers

Forkers

amit-pivotallabs rosenhouse xtreme-sameer-vohra vfonsecapv mendix digideskio ekcasey benmoss crhino tylerphelan degaurab idoru dsabeti flavorjones timani keaty border-buddy skmichaelson gifflen micahyoung opusdude edwardstudy sahilm niroyb dgruber kkallday sjolicoeur alex-slynko reneighbor cjcjameson crdant charleshansen matthewcosgrove oozie gcapizzi lancefrench joshzarrabi arthomps dkoper chrisp-cbh sneal rcw5 jmcarp dbbaskette genevieve andyliuliming ktpv tebuta dazjones christianang ishustava gthiruva jenspinney cagiti aaronjwood clhynfield mcnichol norshtein bingosummer loggregator molambo shashikantk nwmahoney cwlbraa mirahimage bsnchan brylex418 rkoster guoyu07 sandyg1 mouellet calebwashburn mcwumbly ashishapy evandbrown acarrigan-pivotal henderjm pivotal-saman-alvi leeloodub orange-cloudfoundry bengtrj athornton2012 garimasharma titilambert zankich akshaymankar scottbri bpada00 cxhercules manifaust squeedee acnenierga kefeiw sapient007 myminseok crawsible vadimb111 axelaris navin205 leungandrew

bosh-bootloader's Issues

The deployed BOSH director should have a meaningful name

Currently the BOSH director is given the name my-bosh. It would be nice if the BOSH director was given the same name as the stack name.

deploy seems to put state file in the wrong place

I ran the following command:

± |master S:3 ✗| → bbl \
--aws-access-key-id=$(cat ~/workspace/deployments-rmd/bbl-concourse/aws_access_key_id) \
--aws-secret-access-key=$(cat ~/workspace/deployments-rmd/bbl-concourse/aws_secret_access_key) \
--state-dir="~/workspace/deployments-rmd/bbl-concourse/state" \
--aws-region=us-east-1 \
unsupported-deploy-bosh-on-aws-for-concourse

and received the following error:


step: generating cloud config
step: applying cloud config


open ~/workspace/deployments-rmd/bbl-concourse/state/state.json: no such file or directory

Taking a look at what is on the filesystem, it seemed the --state-dir was not respected, and put the file in the wrong place, meaning it couldn't find it later:

 2016-04-08 19:44:41 ☆ ruby 2.1.8p440 rmd-macbook in ~/workspace/deployments-rmd
± |master S:3 ✗| → gst
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

    new file:   bbl-concourse/aws_access_key_id
    new file:   bbl-concourse/aws_secret_access_key
    new file:   state.json

Running the command without the --state-dir flag seemed to succeed.

Changing the director configuration

Hi,

I asked this question on slack, but putting it here as well for further discussion.

Recently I had a bosh environment set up with bbl, and I wanted to configure the health monitor to enable datadog reporting. Normally I would have updated the bosh manifest and done a bosh-init deploy.

How would I do it if I'm using bbl? I ended up just updating the director manually, but that's not a proper long term solution, since another director deployment would overwrite my configuration.

This morning I tried another method. The state.json that bbl created contains all the info I need to create a bosh.yml, bosh-state.yml and the director private key.

I created those files, then used bosh-init to update the director, and set the proper health-monitor options.

This worked but was a pain to do. bbl seems to hide the director configuration and makes it impossible to change after setup.

I'm not sure what the roadmap for bbl is. Will bbl manage the director configuration in the future? To me it would make more sense if bbl bootstrapped the environment, then provided me the template and state so I can continue configuring the director as I want with bosh-init.

Document AWS policy

I created a new AWS user and gave it Administrator Access policy. When I use the corresponding secret access id and key I get the following error:

± |master ?:2 ✗| → bbl --aws-access-key-id=$(cat ~/workspace/deployments-rmd/bbl-concourse/aws_access_key_id) --aws-secret-access-key=$(cat ~/workspace/deployments-rmd/bbl-concourse/aws_secret_access_key) --state-dir="~/workspace/deployments-rmd/bbl-concourse/state" --aws-region=us-east-1 unsupported-deploy-bosh-on-aws-for-concourse
step: creating keypair


UnauthorizedOperation: You are not authorized to perform this operation.
    status code: 403, request id:

Can we document what permissions are expected of the user associated with the keypair?

Argument parsing order

Upon initial use, i attempted to run the command

bbl unsupported-deploy-bosh-on-aws-for-concourse --aws-access-key-id="KEY"

when bbl wanted

bbl --aws-access-key-id="KEY" unsupported-deploy-bosh-on-aws-for-concourse

feel free to blame user error here, but given that amit emailed me the commands in the incorrect order as well i think it seems intuitive to put the commands in an order bbl doesn't recognise

Unable to ssh into Bosh director

We are using an environment that was setup with bbl on AWS. It was created in concourse, and is running our CI. We recently tried to ssh into the director using the bbl credentials which were checked into our repo as part of the CI in the following way:

++ pwd
+ DIR=/tmp/build/be72c34d
+ trap commit_bbl_state_file EXIT
+ pushd env-repo
/tmp/build/be72c34d/env-repo /tmp/build/be72c34d
+ set +x
bbl     --state-dir bbl-infrastructure     --aws-access-key-id [REDACTED]     --aws-secret-access-key [REDACTED]     --aws-region us-east-1     unsupported-deploy-bosh-on-aws-for-concourse
step: using existing keypair
step: generating cloudformation template
step: updating cloudformation stack
step: finished applying cloudformation template
step: generating bosh-init manifest
step: deploying bosh director
Deployment manifest: '/tmp/bosh-init283134969/bosh.yml'
Deployment state: '/tmp/bosh-init283134969/bosh-state.json'

Started validating
  Downloading release 'bosh'... Finished (00:00:03)
  Validating release 'bosh'... Finished (00:00:09)
  Downloading release 'bosh-aws-cpi'... Finished (00:00:00)
  Validating release 'bosh-aws-cpi'... Finished (00:00:00)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Downloading stemcell... Finished (00:00:00)
  Validating stemcell... Finished (00:00:00)
Finished validating (00:00:14)
No deployment, stemcell or release changes. Skipping deploy.
step: generating cloud config
step: applying cloud config
+ bbl --state-dir bbl-infrastructure unsupported-create-lbs --type=cf --cert=/tmp/bbl-cert --key=/tmp/bbl-key --skip-if-exists
lb type "cf" exists, skipping...
+ bbl --state-dir bbl-infrastructure unsupported-update-lbs --cert=/tmp/bbl-cert --key=/tmp/bbl-key
no updates are to be performed
+ popd
/tmp/build/be72c34d
+ commit_bbl_state_file
+ pushd /tmp/build/be72c34d/env-repo
/tmp/build/be72c34d/env-repo /tmp/build/be72c34d
++ git status --porcelain
+ [[ -n '' ]]
+ popd
/tmp/build/be72c34d
+ pushd /tmp/build/be72c34d
/tmp/build/be72c34d /tmp/build/be72c34d
+ shopt -s dotglob
+ cp -R env-repo/.git env-repo/bbl-infrastructure env-repo/cf env-repo/diego updated-env-repo/
+ popd
/tmp/build/be72c34d

Our credentials were rejected and we were unable to debug our problem on the director:

ssh -vvv -i /tmp/hagrid-key [email protected]
OpenSSH_6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /Users/pivotal/.ssh/config
debug1: /Users/pivotal/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 52.206.238.7 [52.206.238.7] port 22.
debug1: Connection established.
debug1: identity file /tmp/hagrid-key type 1
debug1: key_load_public: No such file or directory
debug1: identity file /tmp/hagrid-key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 52.206.238.7:22 as 'root'
debug3: hostkeys_foreach: reading file "/Users/pivotal/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/pivotal/.ssh/known_hosts:52
debug3: load_hostkeys: loaded 1 keys from 52.206.238.7
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: [email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
debug2: kex_parse_kexinit: [email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,hmac-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,hmac-sha1
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug1: kex: server->client aes128-ctr [email protected] none
debug1: kex: client->server aes128-ctr [email protected] none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: [REDACTED]
debug3: hostkeys_foreach: reading file "/Users/pivotal/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/pivotal/.ssh/known_hosts:52
debug3: load_hostkeys: loaded 1 keys from 52.206.238.7
debug1: Host '52.206.238.7' is known and matches the ECDSA host key.
debug1: Found key in /Users/pivotal/.ssh/known_hosts:52
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /Volumes/jaresty/.ssh/work_id_rsa (0x7f916c801430),
debug2: key: /tmp/hagrid-key (0x7f916bc22660), explicit
debug3: input_userauth_banner
Unauthorized use is strictly prohibited. All access and activity
is subject to logging and monitoring.
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Volumes/jaresty/.ssh/work_id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: /tmp/hagrid-key
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

Here is a log of our cloudformation events:


2016-08-17  Status  Type    Logical ID  Status reason
16:20:44 UTC-0700   UPDATE_COMPLETE AWS::CloudFormation::Stack  bbl-aws-KVJis   
16:20:43 UTC-0700   UPDATE_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack  bbl-aws-KVJis   
16:20:38 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet4RouteTableAssociation    
16:20:37 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup CFRouterInternalSecurityGroup   
16:20:36 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterInternalSecurityGroup   Resource creation Initiated
16:20:35 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup CFSSHProxyInternalSecurityGroup 
16:20:34 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet3RouteTableAssociation    
16:20:34 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet1RouteTableAssociation    
16:20:33 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxyInternalSecurityGroup Resource creation Initiated
16:20:32 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet2RouteTableAssociation    
16:20:23 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet4RouteTableAssociation    Resource creation Initiated
16:20:22 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet4RouteTableAssociation    
16:20:20 UTC-0700   CREATE_COMPLETE AWS::ElasticLoadBalancing::LoadBalancer CFRouterLoadBalancer    
16:20:20 UTC-0700   CREATE_COMPLETE AWS::ElasticLoadBalancing::LoadBalancer CFSSHProxyLoadBalancer  
16:20:20 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterInternalSecurityGroup   
16:20:19 UTC-0700   CREATE_IN_PROGRESS  AWS::ElasticLoadBalancing::LoadBalancer CFSSHProxyLoadBalancer  Resource creation Initiated
16:20:19 UTC-0700   CREATE_IN_PROGRESS  AWS::ElasticLoadBalancing::LoadBalancer CFRouterLoadBalancer    Resource creation Initiated
16:20:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet3RouteTableAssociation    Resource creation Initiated
16:20:18 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet1RouteTableAssociation    Resource creation Initiated
16:20:18 UTC-0700   CREATE_IN_PROGRESS  AWS::ElasticLoadBalancing::LoadBalancer CFRouterLoadBalancer    
16:20:18 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet3RouteTableAssociation    
16:20:18 UTC-0700   CREATE_IN_PROGRESS  AWS::ElasticLoadBalancing::LoadBalancer CFSSHProxyLoadBalancer  
16:20:17 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet1RouteTableAssociation    
16:20:17 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxyInternalSecurityGroup 
16:20:17 UTC-0700   CREATE_COMPLETE AWS::EC2::Route LoadBalancerRoute   
16:20:16 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet2RouteTableAssociation    Resource creation Initiated
16:20:15 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet2RouteTableAssociation    
16:20:13 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup CFRouterSecurityGroup   
16:20:13 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet1 
16:20:12 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup CFSSHProxySecurityGroup 
16:20:12 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet4 
16:20:12 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet3 
16:20:12 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet2 
16:20:11 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxySecurityGroup Resource creation Initiated
16:20:11 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterSecurityGroup   Resource creation Initiated
16:20:01 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route LoadBalancerRoute   Resource creation Initiated
16:19:59 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route LoadBalancerRoute   
16:19:56 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet1 Resource creation Initiated
16:19:56 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    LoadBalancerRouteTable  
16:19:56 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet4 Resource creation Initiated
16:19:56 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet1 
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxySecurityGroup 
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet3 Resource creation Initiated
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet2 Resource creation Initiated
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet4 
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    LoadBalancerRouteTable  Resource creation Initiated
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet3 
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet2 
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterSecurityGroup   
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    LoadBalancerRouteTable  
16:19:49 UTC-0700   UPDATE_IN_PROGRESS  AWS::CloudFormation::Stack  bbl-aws-KVJis   User Initiated
16:09:23 UTC-0700   CREATE_COMPLETE AWS::CloudFormation::Stack  bbl-aws-KVJis   
16:09:20 UTC-0700   CREATE_COMPLETE AWS::EC2::EIP   NATEIP  
16:09:04 UTC-0700   CREATE_COMPLETE AWS::EC2::Route InternalRoute   
16:08:48 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route InternalRoute   Resource creation Initiated
16:08:48 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   NATEIP  Resource creation Initiated
16:08:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route InternalRoute   
16:08:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   NATEIP  
16:08:44 UTC-0700   CREATE_COMPLETE AWS::EC2::Instance  NATInstance 
16:07:56 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Instance  NATInstance Resource creation Initiated
16:07:55 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH 
16:07:55 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH 
16:07:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Instance  NATInstance 
16:07:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH Resource creation Initiated
16:07:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH Resource creation Initiated
16:07:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH 
16:07:53 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH 
16:07:50 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup BOSHSecurityGroup   
16:07:50 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup NATSecurityGroup    
16:07:48 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup BOSHSecurityGroup   Resource creation Initiated
16:07:48 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup NATSecurityGroup    Resource creation Initiated
16:07:46 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    
16:07:46 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    
16:07:46 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    
16:07:45 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation 
16:07:45 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    
16:07:44 UTC-0700   CREATE_COMPLETE AWS::EC2::Route BOSHRoute   
16:07:44 UTC-0700   CREATE_COMPLETE AWS::EC2::EIP   BOSHEIP 
16:07:34 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf 
16:07:33 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf 
16:07:33 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf Resource creation Initiated
16:07:32 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf 
16:07:32 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup BOSHSecurityGroup   
16:07:32 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf Resource creation Initiated
16:07:32 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf 
16:07:32 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup NATSecurityGroup    
16:07:30 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    Resource creation Initiated
16:07:30 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    Resource creation Initiated
16:07:30 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    Resource creation Initiated
16:07:29 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    
16:07:29 UTC-0700   CREATE_COMPLETE AWS::IAM::AccessKey BOSHUserAccessKey   
16:07:29 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation Resource creation Initiated
16:07:29 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    Resource creation Initiated
16:07:29 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    
16:07:29 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::AccessKey BOSHUserAccessKey   Resource creation Initiated
16:07:29 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    
16:07:28 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup InternalSecurityGroup   
16:07:28 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route BOSHRoute   Resource creation Initiated
16:07:28 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   BOSHEIP Resource creation Initiated
16:07:28 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation 
16:07:28 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    
16:07:28 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route BOSHRoute   
16:07:28 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   BOSHEIP 
16:07:27 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::AccessKey BOSHUserAccessKey   
16:07:27 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   Resource creation Initiated
16:07:26 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet3 
16:07:26 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet2 
16:07:26 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet4 
16:07:25 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet1 
16:07:25 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    BOSHSubnet  
16:07:24 UTC-0700   CREATE_COMPLETE AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
16:07:23 UTC-0700   CREATE_COMPLETE AWS::IAM::User  BOSHUser    
16:07:12 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    InternalRouteTable  
16:07:11 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  Resource creation Initiated
16:07:10 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  
16:07:10 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    BOSHRouteTable  
16:07:10 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 Resource creation Initiated
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet3 Resource creation Initiated
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 Resource creation Initiated
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  Resource creation Initiated
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 Resource creation Initiated
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  Resource creation Initiated
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet3 
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    Resource creation Initiated
16:07:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 
16:07:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  
16:07:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 
16:07:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
16:07:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  
16:07:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   
16:07:05 UTC-0700   CREATE_COMPLETE AWS::EC2::VPC   VPC 
16:07:03 UTC-0700   CREATE_COMPLETE AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
16:06:48 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPC   VPC Resource creation Initiated
16:06:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPC   VPC 
16:06:47 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::User  BOSHUser    Resource creation Initiated
16:06:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   Resource creation Initiated
16:06:46 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::User  BOSHUser    
16:06:46 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
16:06:42 UTC-0700   CREATE_IN_PROGRESS  AWS::CloudFormation::Stack  bbl-aws-KVJis   User Initiated

As the result of this, we are planning to destroy and re-setup our infrastructure. Let us know if you guys need anymore information.

@jaresty && Chunyi

Setup-infrastructure failed when creating the stack, leaving behind a VM in status `ROLLBACK_COMPLETE`

> bbl --version
bbl 0.0.1

Git sha 33a7fa56f3d0657c6fc25e91434c6d4841fd237d

  > bbl \
    --state-dir bbl-infrastructure \
    --aws-access-key-id "$AWS_ACCESS_KEY_ID" \
    --aws-secret-access-key "$AWS_SECRET_ACCESS_KEY" \
    --aws-region us-east-1 \
    unsupported-deploy-bosh-on-aws-for-concourse

>  echo "$BBL_LB_CERT" > /tmp/bbl-cert
>  echo "$BBL_LB_KEY" > /tmp/bbl-key
>  set -x

>  bbl \
    --state-dir bbl-infrastructure \
    unsupported-create-lbs \
    --type=cf \
    --cert=/tmp/bbl-cert \
    --key=/tmp/bbl-key \
    --skip-if-exists
aws-for-concourse
step: creating keypair
step: generating cloudformation template
step: creating cloudformation stack
•••••••••••••••••••

CloudFormation failure on stack 'bbl-aws-jbpN1'.
Check the AWS console for error events related to this stack,
and/or open a GitHub issue at https://github.com/pivotal-cf-experimental/bosh-bootloader/issues.
/tmp/build/be72c34d/env-repo /tmp/build/be72c34d/env-repo /tmp/build/be72c34d
/tmp/build/be72c34d/env-repo /tmp/build/be72c34d
/tmp/build/be72c34d /tmp/build/be72c34d/env-repo /tmp/build/be72c34d
/tmp/build/be72c34d/env-repo /tmp/build/be72c34d

Cloudformation logs:

2016-08-22  Status  Type    Logical ID  Status reason
15:49:05 UTC-0700   ROLLBACK_COMPLETE   AWS::CloudFormation::Stack  bbl-aws-jbpN1   
15:49:03 UTC-0700   DELETE_COMPLETE AWS::IAM::User  BOSHUser    
15:48:35 UTC-0700   DELETE_COMPLETE AWS::EC2::VPC   VPC 
15:48:18 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::VPC   VPC 
15:48:16 UTC-0700   DELETE_COMPLETE AWS::EC2::Subnet    BOSHSubnet  
15:48:07 UTC-0700   DELETE_COMPLETE AWS::EC2::SecurityGroup InternalSecurityGroup   
15:48:05 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   
15:48:02 UTC-0700   DELETE_COMPLETE AWS::EC2::SecurityGroup NATSecurityGroup    
15:48:00 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SecurityGroup NATSecurityGroup    
15:48:00 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  
15:47:58 UTC-0700   DELETE_COMPLETE AWS::EC2::Instance  NATInstance 
15:47:57 UTC-0700   DELETE_COMPLETE AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
15:47:41 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
15:47:40 UTC-0700   DELETE_COMPLETE AWS::EC2::Subnet    InternalSubnet4 
15:47:40 UTC-0700   DELETE_COMPLETE AWS::EC2::Subnet    InternalSubnet2 
15:47:39 UTC-0700   DELETE_COMPLETE AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
15:47:39 UTC-0700   DELETE_COMPLETE AWS::EC2::Subnet    InternalSubnet3 
15:47:39 UTC-0700   DELETE_COMPLETE AWS::EC2::Subnet    InternalSubnet1 
15:47:25 UTC-0700   DELETE_COMPLETE AWS::EC2::RouteTable    BOSHRouteTable  
15:47:24 UTC-0700   DELETE_COMPLETE AWS::EC2::RouteTable    InternalRouteTable  
15:47:24 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  
15:47:23 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 
15:47:23 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 
15:47:23 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  
15:47:23 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet3 
15:47:23 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Instance  NATInstance 
15:47:23 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
15:47:23 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 
15:47:22 UTC-0700   DELETE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation 
15:47:21 UTC-0700   DELETE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    
15:47:21 UTC-0700   DELETE_COMPLETE AWS::EC2::Route BOSHRoute   
15:47:21 UTC-0700   DELETE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    
15:47:21 UTC-0700   DELETE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    
15:47:21 UTC-0700   DELETE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    
15:47:20 UTC-0700   DELETE_COMPLETE AWS::EC2::Route InternalRoute   
15:47:11 UTC-0700   DELETE_COMPLETE AWS::EC2::SecurityGroup BOSHSecurityGroup   
15:47:09 UTC-0700   DELETE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf 
15:47:08 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SecurityGroup BOSHSecurityGroup   
15:47:06 UTC-0700   DELETE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf 
15:47:06 UTC-0700   DELETE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH 
15:47:06 UTC-0700   DELETE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH 
15:47:06 UTC-0700   DELETE_COMPLETE AWS::EC2::EIP   BOSHEIP 
15:47:05 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation 
15:47:05 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    
15:47:05 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH 
15:47:05 UTC-0700   DELETE_IN_PROGRESS  AWS::IAM::User  BOSHUser    
15:47:05 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf 
15:47:05 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    
15:47:05 UTC-0700   DELETE_COMPLETE AWS::EC2::EIP   NATEIP  
15:47:05 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Route BOSHRoute   
15:47:05 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    
15:47:04 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH 
15:47:04 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Route InternalRoute   
15:47:04 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf 
15:47:04 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    
15:47:04 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::EIP   BOSHEIP 
15:46:33 UTC-0700   ROLLBACK_IN_PROGRESS    AWS::CloudFormation::Stack  bbl-aws-jbpN1   The following resource(s) failed to create: [BOSHUser, InternalRoute, NATEIP]. . Rollback requested by user.
15:46:32 UTC-0700   CREATE_FAILED   AWS::IAM::User  BOSHUser    Resource creation cancelled
15:46:32 UTC-0700   CREATE_FAILED   AWS::EC2::Route InternalRoute   Resource creation cancelled
15:46:30 UTC-0700   CREATE_FAILED   AWS::EC2::EIP   NATEIP  The maximum number of addresses has been reached.
15:46:30 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route InternalRoute   Resource creation Initiated
15:46:30 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   NATEIP  
15:46:30 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route InternalRoute   
15:46:26 UTC-0700   CREATE_COMPLETE AWS::EC2::Instance  NATInstance 
15:46:22 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::User  BOSHUser    Resource creation Initiated
15:45:39 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH 
15:45:38 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Instance  NATInstance Resource creation Initiated
15:45:38 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH 
15:45:38 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH Resource creation Initiated
15:45:38 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH 
15:45:37 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    
15:45:37 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH Resource creation Initiated
15:45:37 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Instance  NATInstance 
15:45:37 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH 
15:45:33 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup NATSecurityGroup    
15:45:33 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup BOSHSecurityGroup   
15:45:32 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    
15:45:32 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup NATSecurityGroup    Resource creation Initiated
15:45:31 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup BOSHSecurityGroup   Resource creation Initiated
15:45:31 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation 
15:45:30 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    
15:45:30 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    
15:45:30 UTC-0700   CREATE_COMPLETE AWS::EC2::EIP   BOSHEIP 
15:45:30 UTC-0700   CREATE_COMPLETE AWS::EC2::Route BOSHRoute   
15:45:21 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    Resource creation Initiated
15:45:20 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    
15:45:17 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    Resource creation Initiated
15:45:17 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf 
15:45:16 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf 
15:45:16 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    
15:45:16 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf Resource creation Initiated
15:45:16 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup NATSecurityGroup    
15:45:15 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf 
15:45:15 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup BOSHSecurityGroup   
15:45:15 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf Resource creation Initiated
15:45:15 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation Resource creation Initiated
15:45:15 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf 
15:45:14 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    Resource creation Initiated
15:45:14 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    Resource creation Initiated
15:45:14 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   BOSHEIP Resource creation Initiated
15:45:14 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation 
15:45:14 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route BOSHRoute   Resource creation Initiated
15:45:14 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    
15:45:13 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   BOSHEIP 
15:45:13 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    
15:45:13 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route BOSHRoute   
15:45:11 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet4 
15:45:11 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet2 
15:45:11 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup InternalSecurityGroup   
15:45:11 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet1 
15:45:10 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet3 
15:45:10 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    BOSHSubnet  
15:45:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   Resource creation Initiated
15:45:09 UTC-0700   CREATE_COMPLETE AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
15:44:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 Resource creation Initiated
15:44:55 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    InternalRouteTable  
15:44:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 Resource creation Initiated
15:44:55 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    BOSHRouteTable  
15:44:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 
15:44:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet3 Resource creation Initiated
15:44:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 Resource creation Initiated
15:44:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  Resource creation Initiated
15:44:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 
15:44:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  Resource creation Initiated
15:44:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  Resource creation Initiated
15:44:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet3 
15:44:53 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    Resource creation Initiated
15:44:53 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 
15:44:53 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  
15:44:53 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  
15:44:53 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   
15:44:53 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
15:44:53 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  
15:44:50 UTC-0700   CREATE_COMPLETE AWS::EC2::VPC   VPC 
15:44:49 UTC-0700   CREATE_COMPLETE AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
15:44:33 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   Resource creation Initiated
15:44:33 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
15:44:32 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPC   VPC Resource creation Initiated
15:44:32 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::User  BOSHUser    
15:44:31 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPC   VPC 
15:44:27 UTC-0700   CREATE_IN_PROGRESS  AWS::CloudFormation::Stack  bbl-aws-jbpN1   User Initiated

@ChunyiLyu && jaresty

Setup-infrastructure failed

bbl --version
bbl 0.0.1

Ran:
bbl --state-dir bbl-infrastructure --aws-access-key-id [REDACTED] --aws-secret-access-key [REDACTED] --aws-region us-east-1 unsupported-deploy-bosh-on-aws-for-concourse

Cloudformation output:

2016-08-17  Status  Type    Logical ID  Status reason
14:59:17 UTC-0700   UPDATE_ROLLBACK_COMPLETE    AWS::CloudFormation::Stack  bbl-aws-hR3w8   
14:59:16 UTC-0700   DELETE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet4 
14:59:16 UTC-0700   DELETE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet1 
14:59:15 UTC-0700   DELETE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet2 
14:59:15 UTC-0700   DELETE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet3 
14:59:00 UTC-0700   DELETE_COMPLETE AWS::EC2::RouteTable    LoadBalancerRouteTable  
14:59:00 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet4 
14:58:59 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet2 
14:58:59 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::RouteTable    LoadBalancerRouteTable  
14:58:59 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet1 
14:58:59 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet3 
14:58:57 UTC-0700   DELETE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet4RouteTableAssociation    
14:58:57 UTC-0700   DELETE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet2RouteTableAssociation    
14:58:57 UTC-0700   DELETE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet1RouteTableAssociation    
14:58:57 UTC-0700   DELETE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet3RouteTableAssociation    
14:58:56 UTC-0700   DELETE_COMPLETE AWS::EC2::Route LoadBalancerRoute   
14:58:46 UTC-0700   DELETE_COMPLETE AWS::EC2::SecurityGroup CFRouterSecurityGroup   
14:58:46 UTC-0700   DELETE_COMPLETE AWS::EC2::SecurityGroup CFSSHProxySecurityGroup 
14:58:45 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterSecurityGroup   
14:58:45 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxySecurityGroup 
14:58:42 UTC-0700   DELETE_COMPLETE AWS::EC2::SecurityGroup CFRouterInternalSecurityGroup   
14:58:42 UTC-0700   DELETE_COMPLETE AWS::EC2::SecurityGroup CFSSHProxyInternalSecurityGroup 
14:58:41 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet4RouteTableAssociation    
14:58:41 UTC-0700   DELETE_COMPLETE AWS::ElasticLoadBalancing::LoadBalancer CFSSHProxyLoadBalancer  
14:58:41 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet2RouteTableAssociation    
14:58:41 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet1RouteTableAssociation    
14:58:41 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterInternalSecurityGroup   
14:58:41 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet3RouteTableAssociation    
14:58:41 UTC-0700   DELETE_COMPLETE AWS::ElasticLoadBalancing::LoadBalancer CFRouterLoadBalancer    
14:58:41 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::Route LoadBalancerRoute   
14:58:40 UTC-0700   DELETE_IN_PROGRESS  AWS::ElasticLoadBalancing::LoadBalancer CFSSHProxyLoadBalancer  
14:58:40 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxyInternalSecurityGroup 
14:58:38 UTC-0700   UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS    AWS::CloudFormation::Stack  bbl-aws-hR3w8   
14:58:22 UTC-0700   UPDATE_ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack  bbl-aws-hR3w8   The following resource(s) failed to create: [CFRouterLoadBalancer, CFSSHProxyInternalSecurityGroup, CFRouterInternalSecurityGroup, LoadBalancerSubnet3RouteTableAssociation, LoadBalancerSubnet1RouteTableAssociation, LoadBalancerSubnet2RouteTableAssociation, LoadBalancerSubnet4RouteTableAssociation].
14:58:20 UTC-0700   CREATE_FAILED   AWS::EC2::SecurityGroup CFRouterInternalSecurityGroup   Resource creation cancelled
14:58:16 UTC-0700   CREATE_FAILED   AWS::EC2::SecurityGroup CFSSHProxyInternalSecurityGroup Resource creation cancelled
14:58:09 UTC-0700   CREATE_FAILED   AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet1RouteTableAssociation    Resource creation cancelled
14:58:08 UTC-0700   CREATE_FAILED   AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet2RouteTableAssociation    Resource creation cancelled
14:58:08 UTC-0700   CREATE_FAILED   AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet3RouteTableAssociation    Resource creation cancelled
14:58:08 UTC-0700   CREATE_FAILED   AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet4RouteTableAssociation    Resource creation cancelled
14:58:08 UTC-0700   CREATE_COMPLETE AWS::ElasticLoadBalancing::LoadBalancer CFSSHProxyLoadBalancer  
14:58:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet1RouteTableAssociation    Resource creation Initiated
14:58:07 UTC-0700   CREATE_IN_PROGRESS  AWS::ElasticLoadBalancing::LoadBalancer CFSSHProxyLoadBalancer  Resource creation Initiated
14:58:07 UTC-0700   CREATE_FAILED   AWS::ElasticLoadBalancing::LoadBalancer CFRouterLoadBalancer    Server Certificate not found for the key: arn:aws:iam::126623727247:server-certificate/bbl-cert-dfb7b96b-111f-d16e-5854-9936e7153e8b
14:58:07 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet1RouteTableAssociation    
14:58:07 UTC-0700   CREATE_IN_PROGRESS  AWS::ElasticLoadBalancing::LoadBalancer CFRouterLoadBalancer    
14:58:06 UTC-0700   CREATE_IN_PROGRESS  AWS::ElasticLoadBalancing::LoadBalancer CFSSHProxyLoadBalancer  
14:58:05 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterInternalSecurityGroup   
14:58:02 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet1 
14:58:01 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet4RouteTableAssociation    Resource creation Initiated
14:58:01 UTC-0700   CREATE_COMPLETE AWS::EC2::Route LoadBalancerRoute   
14:58:00 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet3RouteTableAssociation    Resource creation Initiated
14:58:00 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet4RouteTableAssociation    
14:58:00 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet2RouteTableAssociation    Resource creation Initiated
14:58:00 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxyInternalSecurityGroup 
14:58:00 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup CFRouterSecurityGroup   
14:57:59 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet3RouteTableAssociation    
14:57:59 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet2RouteTableAssociation    
14:57:58 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterSecurityGroup   Resource creation Initiated
14:57:56 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup CFSSHProxySecurityGroup 
14:57:55 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet4 
14:57:55 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet3 
14:57:55 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet2 
14:57:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxySecurityGroup Resource creation Initiated
14:57:46 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet1 Resource creation Initiated
14:57:45 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet1 
14:57:45 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route LoadBalancerRoute   Resource creation Initiated
14:57:44 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route LoadBalancerRoute   
14:57:42 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterSecurityGroup   
14:57:40 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    LoadBalancerRouteTable  
14:57:39 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    LoadBalancerRouteTable  Resource creation Initiated
14:57:39 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet3 Resource creation Initiated
14:57:39 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    LoadBalancerRouteTable  
14:57:39 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet2 Resource creation Initiated
14:57:38 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet4 Resource creation Initiated
14:57:38 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet3 
14:57:38 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxySecurityGroup 
14:57:38 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet2 
14:57:38 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet4 
14:57:32 UTC-0700   UPDATE_IN_PROGRESS  AWS::CloudFormation::Stack  bbl-aws-hR3w8   User Initiated

Prior events:

14:44:29 UTC-0700   CREATE_COMPLETE AWS::CloudFormation::Stack  bbl-aws-hR3w8   
14:44:26 UTC-0700   CREATE_COMPLETE AWS::EC2::EIP   NATEIP  
14:44:11 UTC-0700   CREATE_COMPLETE AWS::EC2::Route InternalRoute   
14:43:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route InternalRoute   Resource creation Initiated
14:43:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   NATEIP  Resource creation Initiated
14:43:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   NATEIP  
14:43:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route InternalRoute   
14:43:51 UTC-0700   CREATE_COMPLETE AWS::EC2::Instance  NATInstance 
14:43:03 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Instance  NATInstance Resource creation Initiated
14:43:03 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH 
14:43:03 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH 
14:43:02 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Instance  NATInstance 
14:43:02 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH Resource creation Initiated
14:43:02 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH Resource creation Initiated
14:43:02 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH 
14:43:01 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH 
14:42:58 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    
14:42:58 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup BOSHSecurityGroup   
14:42:58 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup NATSecurityGroup    
14:42:57 UTC-0700   CREATE_COMPLETE AWS::EC2::EIP   BOSHEIP 
14:42:57 UTC-0700   CREATE_COMPLETE AWS::EC2::Route BOSHRoute   
14:42:56 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup NATSecurityGroup    Resource creation Initiated
14:42:56 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup BOSHSecurityGroup   Resource creation Initiated
14:42:55 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    
14:42:55 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    
14:42:55 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    
14:42:55 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation 
14:42:42 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf 
14:42:42 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   BOSHEIP Resource creation Initiated
14:42:42 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    Resource creation Initiated
14:42:41 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route BOSHRoute   Resource creation Initiated
14:42:41 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf 
14:42:41 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   BOSHEIP 
14:42:41 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    
14:42:41 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf Resource creation Initiated
14:42:41 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf 
14:42:40 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup NATSecurityGroup    
14:42:40 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route BOSHRoute   
14:42:40 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    Resource creation Initiated
14:42:40 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf Resource creation Initiated
14:42:40 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    Resource creation Initiated
14:42:40 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    Resource creation Initiated
14:42:40 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup BOSHSecurityGroup   
14:42:40 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation Resource creation Initiated
14:42:40 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf 
14:42:39 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    
14:42:39 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    
14:42:39 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    
14:42:39 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation 
14:42:38 UTC-0700   CREATE_COMPLETE AWS::IAM::AccessKey BOSHUserAccessKey   
14:42:37 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::AccessKey BOSHUserAccessKey   Resource creation Initiated
14:42:37 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::AccessKey BOSHUserAccessKey   
14:42:37 UTC-0700   CREATE_COMPLETE AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
14:42:36 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup InternalSecurityGroup   
14:42:36 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet3 
14:42:36 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet2 
14:42:36 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet4 
14:42:36 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet1 
14:42:35 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    BOSHSubnet  
14:42:35 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   Resource creation Initiated
14:42:34 UTC-0700   CREATE_COMPLETE AWS::IAM::User  BOSHUser    
14:42:21 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    Resource creation Initiated
14:42:21 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    InternalRouteTable  
14:42:20 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    BOSHRouteTable  
14:42:20 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 Resource creation Initiated
14:42:20 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet3 Resource creation Initiated
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 Resource creation Initiated
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  Resource creation Initiated
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 Resource creation Initiated
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  Resource creation Initiated
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  Resource creation Initiated
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet3 
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 
14:42:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  
14:42:15 UTC-0700   CREATE_COMPLETE AWS::EC2::VPC   VPC 
14:42:13 UTC-0700   CREATE_COMPLETE AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
14:41:58 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPC   VPC Resource creation Initiated
14:41:57 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   Resource creation Initiated
14:41:57 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::User  BOSHUser    Resource creation Initiated
14:41:57 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPC   VPC 
14:41:57 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
14:41:57 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::User  BOSHUser    
14:41:52 UTC-0700   CREATE_IN_PROGRESS  AWS::CloudFormation::Stack  bbl-aws-hR3w8   User Initiated

Looks like the problem was Server Certificate not found for the key: arn:aws:iam::126623727247:server-certificate/bbl-cert-dfb7b96b-111f-d16e-5854-9936e7153e8b

Thanks
Dennis && @jaresty

test

bbl should have command to display load balancer information

When we use bbl to create a load balancer, we need to setup DNS out of band to route to it. If bbl could provide some information about the LB, perhaps it only needs to be the host DNS address, it would greatly simplify automating of the out-of-band DNS configuration

bbl should have a command to obtain the director ip address alone

Currently to ssh we need to do the following to pass in the gateway host
bosh ssh --gateway_host $(bbl director-address | sed 's/https:\/\///' | sed 's/:25555//')

Possible bug in bbl (bosh-bootloader) created ELB when deploying CloudFoundry (on AWS)

Basically, all one needs to do with this amazing bbl tool is call the following two commands:

bbl unsupported-deploy-bosh-on-aws-for-concourse
bbl unsupported-create-lbs --type=cf

This will bootstrap the environment needed in AWS to use BOSH, for example to install CloudFoundry.
With this environment ready, you can try to install the OSS cf-release as described here.

However, this bootstrapped environment contains a minor bug that might cause you to to pull out your hairs. The router_z1 and router_z2 instances do get registered correctly to one of the two created ELB's (named CFRouter and CFSSHPro). One of them, CFRouter, seems to be used to route HTTP(S) traffic towards the routers, about the purpose of the other ELB I am unsure at this point.

However, the CFRouter will be in a Security Group called CFRouterSecurityGroup which does allow traffic on TCP ports 80, 443 and 4443 from everywhere.

However, the traffic needs to pass from the ELB in CFRouterSecurityGroup to the actual router instances, which are in a Security Group called InternalSecurityGroup. As you can see in the screenshot below, this security group does not allow traffic (from Security Group CFRouterSecurityGroup) on TCP ports 80, 443 and 4443, causing all the ELB EC2 instances to have Status OutOfService (because the Health Checks fail) and the ELB returning HTTP 503 on every request.

To get this setup working, one has to modify the InternalSecurityGroup to allow traffic on TCP ports 80, 443 and 4443 to be accepted when it originates from the CFRouterSecurityGroup as you can see in the screenshot below.

Changing this (manually, for now) is the way to go to get a healthy and responsive CFRouter ELB and a (presumably) working CloudFoundry installation (on AWS), at least as far as bosh deploy has finished successfully. We strongly believe that this should be working out of the box when one runs bbl unsupported-create-lbs --type=cf.

bbl generated ca-cert does not verify correctly with bosh director

Certs are hard. :(

It seems like it has something to do with the serial number maybe?

 bosh -n -d cf-2.0 -e https://52.207.147.110:25555 --user=user-hWlBQMg --password=REDACTED --ca-cert=/tmp/bbl-ca deploy manifest/cf-deployment.yml
Using environment 'https://52.207.147.110:25555' as user 'user-hWlBQMg'

Using deployment 'cf-2.0'

Diffing manifest:
  Fetching diff result:
    Performing request POST 'https://52.207.147.110:25555/deployments/cf-2.0/diff?redact=true':
      Performing POST request:
        Post https://user-hWlBQMg:[email protected]:25555/deployments/cf-2.0/diff?redact=true: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "serial:1234")

Exit code 1

Either way after banging away at this for a while we tried verifying the server.crt against the ca cert openssl verify -verbose -CAfile ca.crt director.pem which results in:

openssl verify -verbose -CAfile ca-cert/ca-cert /tmp/director.pem
/tmp/director.pem: C = USA, O = Cloud Foundry, CN = 52.207.147.110
error 7 at 0 depth lookup:certificate signature failure
139903998613152:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:
139903998613152:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:721:
139903998613152:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:233:

Not a very useful error, but an error anyway.

Also it might be useful to note that when comparing the decrypted CA cert file from following this there are some differences such as:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1234 (0x4d2)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=USA, O=Cloud Foundry, CN=BOSH Bootloader
...
X509v3 extensions:
            X509v3 Key Usage: critical
                Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Alternative Name:
                IP Address:52.207.147.110

^^ from bbl

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ae:6e:5e:e6:12:39:e0:21
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=AU, ST=Some-State, O=Internet Widgits Pty Ltd
...
X509v3 extensions:
            X509v3 Subject Key Identifier:
                79:E4:EF:B7:18:8E:1C:0B:08:52:64:CF:19:04:55:B7:23:BC:D8:BF
            X509v3 Authority Key Identifier:
                keyid:79:E4:EF:B7:18:8E:1C:0B:08:52:64:CF:19:04:55:B7:23:BC:D8:BF
                DirName:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
                serial:AE:6E:5E:E6:12:39:E0:21

            X509v3 Basic Constraints:
                CA:TRUE

^^ from docs

Thanks
@APShirley && Dennis
CF Release Integration Team

Tagging instances created by bosh, and bosh itself

When deploying to an AWS environment I want to be able to tag the bosh director, and have bosh tag instances it creates. This helps with cost allocation and organizing of systems.

bbl should optionally provide output in machine-parseable format

The human-readable output of commands like unsupported-lbs is good for humans, but brittle when it comes to machine parsing.

If bbl supported a --output-format=[human|json|yaml] flag, it would greatly improve the experience of scripting it.

Global flags must be provided before the command - they should be applied anywhere

Global flags should not have to be ordered i.e:

 2016-06-02 10:18:06 ☆ ruby 2.0.0p645 (system) rdimsdale in ~
○ → bbl unsupported-create-lbs --aws-access-key-id='foo' --aws-secret-access-key='bar' --aws-region=baz


--aws-access-key-id must be provided

But moving the command after returns more meaningful output:

 2016-06-02 10:19:29 ☆ ruby 2.0.0p645 (system) rdimsdale in ~
○ → bbl --aws-access-key-id='foo' --aws-secret-access-key='bar' --aws-region=baz unsupported-create-lbs


"" is not a valid lb type, valid lb types are: concourse and cf

Using t2.micro worker I ran out of space for Concourse deployment

I tried using t2.micro for concourse ci since still in POC phase ran into disk space issue. The fix was to change vm type to m3.large.

Done updating job web > web/0 (34b5bfc6-aeb5-4578-9322-0fe76632f6df) (canary) (00:00:16)
Failed updating job worker > worker/0 (e92fd18b-c11c-4990-a1c7-661acc870d9e) (canary): Action Failed get_task: Task 8de9bbb4-75da-442b-7fca-8bbf63042117 result: Applying: Applying job groundcrew: Applying package garden-linux for job groundcrew: Decompressing package files: Shelling out to tar: Running command: 'tar --no-same-owner -xzvf /var/vcap/data/tmp/bosh-blobstore-externalBlobstore-Get034396911 -C /var/vcap/data/tmp/bosh-agent-applier-packages-CompiledPackageApplier-Apply134349264', stdout: './
./bin/
./bin/garden-linux
./src/
...
...
./pkg/linux_amd64/golang.org/x/net/
./pkg/linux_amd64/golang.org/x/net/context.a
', stderr: 'tar: ./src/github.com/cloudfoundry-incubator/garden-linux/out/garden-linux: Wrote only 7680 of 10240 bytes
tar: ./src/github.com/cloudfoundry-incubator/garden-linux/container_daemon: Cannot mkdir: No space left on device
tar: ./src/github.com/cloudfoundry-incubator/garden-linux/container_daemon: Cannot mkdir: No space left on device
tar: ./src/github.com/cloudfoundry-incubator/garden-linux/container_daemon/fake_rlimits_env_encoder: Cannot mkdir: No such file or directory
tar: ./src/github.com/cloudfoundry-incubator/garden-linux/container_daemon: Cannot mkdir: No space left on device
tar: ./src/github.com/cloudfoundry-incubator/garden-linux/container_daemon/fake_rlimits_env_encoder/fake_rlimits_env_encoder.go: Cannot open: No such file or directory
tar: ./src/github.com/cloudfoundry-incubator/garden-linux/container_daemon: Cannot mkdir: No space left on device
tar: ./src/github.com/cloudfoundry-incubator/garden-linux/container_daemon/fake_cmdpreparer: Cannot mkdir: No such file or directory
tar: ./src/github.com/cloudfoundry-incubator/garden-linux/container_daemon: Cannot mkdir: No space left on device
tar: ./src/github.com/cloudfoundry-incubator/garden-linux/container_daemon/fake_cmdpreparer/fake_cmdpreparer.go: Cannot open: No such file or directory
tar: ./src/github.com/cloudfoundry-incubator/garden-linux/container_daemon: Cannot mkdir: No space left on device```

bbl dies when it asks for operations in some AZs

In this case, the CloudFormation template asked for a subnet in us-east-1c. c is either full or snobby, because it rejects this request (saying that AZs a, b, d and e are able to do it).

us-east-1c seems to be hard-coded. I tried running bbl twice and got the same result.

2016-08-20  Status  Type    Logical ID  Status reason
16:15:56 UTC-0400   ROLLBACK_COMPLETE   AWS::CloudFormation::Stack  stack-bbl-env-huron-2016-08-20T20-14Z   
16:15:55 UTC-0400   DELETE_COMPLETE AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
16:15:54 UTC-0400   DELETE_COMPLETE AWS::EC2::VPC   VPC 
16:15:39 UTC-0400   DELETE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
16:15:38 UTC-0400   DELETE_IN_PROGRESS  AWS::EC2::VPC   VPC 
16:15:36 UTC-0400   DELETE_COMPLETE AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
16:15:36 UTC-0400   DELETE_COMPLETE AWS::EC2::Subnet    InternalSubnet4 
16:15:36 UTC-0400   DELETE_COMPLETE AWS::EC2::Subnet    InternalSubnet1 
16:15:36 UTC-0400   DELETE_COMPLETE AWS::EC2::Subnet    InternalSubnet5 
16:15:36 UTC-0400   DELETE_COMPLETE AWS::EC2::Subnet    InternalSubnet2 
16:15:36 UTC-0400   DELETE_COMPLETE AWS::EC2::Subnet    BOSHSubnet  
16:15:23 UTC-0400   DELETE_COMPLETE AWS::IAM::User  BOSHUser    
16:15:22 UTC-0400   DELETE_COMPLETE AWS::EC2::SecurityGroup InternalSecurityGroup   
16:15:21 UTC-0400   DELETE_COMPLETE AWS::EC2::RouteTable    InternalRouteTable  
16:15:21 UTC-0400   DELETE_COMPLETE AWS::EC2::RouteTable    BOSHRouteTable  
16:15:20 UTC-0400   DELETE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 
16:15:20 UTC-0400   DELETE_IN_PROGRESS  AWS::IAM::User  BOSHUser    
16:15:20 UTC-0400   DELETE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   
16:15:20 UTC-0400   DELETE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 
16:15:20 UTC-0400   DELETE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
16:15:20 UTC-0400   DELETE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  
16:15:20 UTC-0400   DELETE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 
16:15:20 UTC-0400   DELETE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  
16:15:20 UTC-0400   DELETE_COMPLETE AWS::EC2::Subnet    InternalSubnet3 
16:15:20 UTC-0400   DELETE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet5 
16:15:20 UTC-0400   DELETE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  
16:14:49 UTC-0400   ROLLBACK_IN_PROGRESS    AWS::CloudFormation::Stack  stack-bbl-env-huron-2016-08-20T20-14Z   The following resource(s) failed to create: [InternalSubnet2, BOSHUser, InternalSubnet3, InternalSecurityGroup, InternalSubnet5, InternalSubnet4, InternalSubnet1, BOSHSubnet, VPCGatewayAttachment]. . Rollback requested by user.
16:14:48 UTC-0400   CREATE_FAILED   AWS::EC2::SecurityGroup InternalSecurityGroup   Resource creation cancelled
16:14:33 UTC-0400   CREATE_FAILED   AWS::EC2::Subnet    InternalSubnet2 Resource creation cancelled
16:14:33 UTC-0400   CREATE_FAILED   AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    Resource creation cancelled
16:14:33 UTC-0400   CREATE_FAILED   AWS::EC2::Subnet    InternalSubnet1 Resource creation cancelled
16:14:33 UTC-0400   CREATE_FAILED   AWS::EC2::Subnet    InternalSubnet5 Resource creation cancelled
16:14:33 UTC-0400   CREATE_FAILED   AWS::EC2::Subnet    InternalSubnet4 Resource creation cancelled
16:14:33 UTC-0400   CREATE_FAILED   AWS::IAM::User  BOSHUser    Resource creation cancelled
16:14:33 UTC-0400   CREATE_FAILED   AWS::EC2::Subnet    BOSHSubnet  Resource creation cancelled
16:14:33 UTC-0400   CREATE_COMPLETE AWS::EC2::RouteTable    BOSHRouteTable  
16:14:33 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    Resource creation Initiated
16:14:32 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   
16:14:32 UTC-0400   CREATE_COMPLETE AWS::EC2::RouteTable    InternalRouteTable  
16:14:32 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 Resource creation Initiated
16:14:32 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
16:14:32 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 Resource creation Initiated
16:14:32 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  Resource creation Initiated
16:14:32 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 Resource creation Initiated
16:14:32 UTC-0400   CREATE_FAILED   AWS::EC2::Subnet    InternalSubnet3 Value (us-east-1c) for parameter availabilityZone is invalid. Subnets can currently only be created in the following availability zones: us-east-1a, us-east-1d, us-east-1b, us-east-1e.
16:14:31 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet5 Resource creation Initiated
Physical ID:subnet-38aeb812
16:14:31 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 
16:14:31 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 
16:14:31 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  Resource creation Initiated
16:14:31 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  Resource creation Initiated
16:14:31 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  
16:14:31 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 
16:14:31 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet5 
16:14:31 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet3 
16:14:31 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  
16:14:31 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  
16:14:28 UTC-0400   CREATE_COMPLETE AWS::EC2::VPC   VPC 
16:14:27 UTC-0400   CREATE_COMPLETE AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
16:14:12 UTC-0400   CREATE_IN_PROGRESS  AWS::IAM::User  BOSHUser    Resource creation Initiated
16:14:11 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::VPC   VPC Resource creation Initiated
16:14:11 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   Resource creation Initiated
16:14:10 UTC-0400   CREATE_IN_PROGRESS  AWS::IAM::User  BOSHUser    
16:14:10 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::VPC   VPC 
16:14:10 UTC-0400   CREATE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
16:14:03 UTC-0400   CREATE_IN_PROGRESS  AWS::CloudFormation::Stack  stack-bbl-env-huron-2016-08-20T20-14Z

EBS Encryption

One requirement for using Public Cloud is encryption of at rest systems. So being able to enable encryption of EBS volumes would be required feature. Even better would be ability to use your own Encryption key.

director disk fills up

Opening this issue so we can discuss it later.

If people don't run bosh cleanup regularly, the director disk fills up. Right now bbl doesn't let you change the disk size, and any manual changes you make will conflict with the BOSH state that bbl stores. We could:

(a) make the disk size bigger
(b) document that people should be running cleanup regularly
(c) make the disk size configurable via a flag or command
(d) have a way to setup the IaaS but then just print out the director manifest and cloud config for users to edit and manage on their own, applying whatever post-processing they want
(e) several of the above

/cc @jtarchie @dsabeti

bbl destroy fails and is left in a state where subsequent bbl destroys also fail

We attempted a bbl destroy. However the destroy failed with the following error message: CloudFormation failure on stack 'bbl-aws-ICU0e'. Check the AWS console for error events related to this stack, and/or open a GitHub issue at https://github.com/pivotal-cf-experimental/bosh-bootloader/issues.

Looking at the cloudformation stack, the following snippet is the root cause of bbl being unable to destroy.

12:31:24 UTC-0700   DELETE_FAILED   AWS::CloudFormation::Stack  bbl-aws-ICU0e   The following resource(s) failed to delete: [LoadBalancerSubnet1, VPCGatewayInternetGateway, VPC, VPCGatewayAttachment].
12:31:22 UTC-0700   DELETE_FAILED   AWS::EC2::InternetGateway   VPCGatewayInternetGateway   The internetGateway 'igw-26f9c342' has dependencies and cannot be deleted.
11:51:42 UTC-0700   DELETE_FAILED   AWS::EC2::VPC   VPC The vpc 'vpc-e11d7f86' has dependencies and cannot be deleted.
11:50:14 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::VPC   VPC 
11:50:11 UTC-0700   DELETE_COMPLETE AWS::EC2::SecurityGroup CFRouterSecurityGroup   
11:48:48 UTC-0700   DELETE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet3 
11:48:26 UTC-0700   DELETE_FAILED   AWS::EC2::Subnet    LoadBalancerSubnet1 The subnet 'subnet-70252228' has dependencies and cannot be deleted.
11:48:25 UTC-0700   DELETE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
11:48:23 UTC-0700   DELETE_FAILED   AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    Network vpc-e11d7f86 has some mapped public address(es). Please unmap those public address(es) before detaching the gateway.

We ensured that there were no bosh deployments.
Could this be a race condition when deleting the director vm?

Regardless when we attempted to run bbl destroy again, it failed with the following error message:

Command 'delete' failed:
  Deleting deployment:
    Checking existance of vm for instance 'unknown/0':
      Checking existance of VM 'i-b26ef94a':
        CPI 'has_vm' method responded with error: CmdError{"type":"Unknown","message":"AWS was not able to validate the provided access credentials","ok_to_retry":false}

We think this has left us in a state where the only way to delete the stack is via AWS and not by bbl.

Thanks
@DennisDenuto && Chunyi

Rollback not deleting key pairs

We got some failures while trying to spin up a new environment and saw that the key pairs weren't deleted as part of the cloud formation roll back.

More disk sizes in cloud config

The cloud_config created by bbl contains one disk type:

disk_types:
- name: default
  disk_size: 1024
  cloud_properties:
    type: gp2
    encrypted: true

Many deployments (e.g. concourse) need control over the disk sizing. I think bbl needs to provide more disk types (e.g. 50GB, 100GB, 500GB) and/or support customizing the disk types.

Currently we have to edit the cloud config after bbl creates it in order to support our concourse workers; we would prefer that bbl was aware of our changes instead of us configuring things out of band.

bbl should fail fast with helpful error message if bosh-init is too old

We ran into an issue where our bosh-init was too old. bbl should fail fast with clear error message stating the actual version, and the minimum version it requires

gopkg.in fails on submodule init

On a fresh GOPATH we had the following error with gopkg.in.

go get github.com/pivotal-cf-experimental/bosh-bootloader/bbl

# cd /Users/pivotal/workspace/golang/src/github.com/pivotal-cf-experimental/bosh-bootloader; git submodule update --init --recursive
Submodule 'vendor/github.com/aws/aws-sdk-go' (https://github.com/aws/aws-sdk-go.git) registered for path 'vendor/github.com/aws/aws-sdk-go'
Submodule 'vendor/github.com/cloudfoundry/multierror' (https://github.com/cloudfoundry/multierror.git) registered for path 'vendor/github.com/cloudfoundry/multierror'
Submodule 'vendor/github.com/onsi/ginkgo' (https://github.com/onsi/ginkgo) registered for path 'vendor/github.com/onsi/ginkgo'
Submodule 'vendor/github.com/onsi/gomega' (https://github.com/onsi/gomega) registered for path 'vendor/github.com/onsi/gomega'
Submodule 'vendor/github.com/pivotal-cf-experimental/bosh-test' (https://github.com/pivotal-cf-experimental/bosh-test.git) registered for path 'vendor/github.com/pivotal-cf-experimental/bosh-test'
Submodule 'vendor/github.com/pivotal-cf-experimental/gomegamatchers' (https://github.com/pivotal-cf-experimental/gomegamatchers) registered for path 'vendor/github.com/pivotal-cf-experimental/gomegamatchers'
Submodule 'vendor/github.com/rosenhouse/awsfaker' (https://github.com/rosenhouse/awsfaker.git) registered for path 'vendor/github.com/rosenhouse/awsfaker'
Submodule 'vendor/github.com/square/certstrap' (https://github.com/square/certstrap) registered for path 'vendor/github.com/square/certstrap'
Submodule 'vendor/golang.org/x/crypto' (https://github.com/golang/crypto.git) registered for path 'vendor/golang.org/x/crypto'
Submodule 'vendor/gopkg.in/yaml.v2' (https://gopkg.in/yaml.v2) registered for path 'vendor/gopkg.in/yaml.v2'
Cloning into '/Users/pivotal/workspace/golang/src/github.com/pivotal-cf-experimental/bosh-bootloader/vendor/github.com/aws/aws-sdk-go'...
Cloning into '/Users/pivotal/workspace/golang/src/github.com/pivotal-cf-experimental/bosh-bootloader/vendor/github.com/cloudfoundry/multierror'...
Cloning into '/Users/pivotal/workspace/golang/src/github.com/pivotal-cf-experimental/bosh-bootloader/vendor/github.com/onsi/ginkgo'...
Cloning into '/Users/pivotal/workspace/golang/src/github.com/pivotal-cf-experimental/bosh-bootloader/vendor/github.com/onsi/gomega'...
Cloning into '/Users/pivotal/workspace/golang/src/github.com/pivotal-cf-experimental/bosh-bootloader/vendor/github.com/pivotal-cf-experimental/bosh-test'...
Cloning into '/Users/pivotal/workspace/golang/src/github.com/pivotal-cf-experimental/bosh-bootloader/vendor/github.com/pivotal-cf-experimental/gomegamatchers'...
Cloning into '/Users/pivotal/workspace/golang/src/github.com/pivotal-cf-experimental/bosh-bootloader/vendor/github.com/rosenhouse/awsfaker'...
Cloning into '/Users/pivotal/workspace/golang/src/github.com/pivotal-cf-experimental/bosh-bootloader/vendor/github.com/square/certstrap'...
Cloning into '/Users/pivotal/workspace/golang/src/github.com/pivotal-cf-experimental/bosh-bootloader/vendor/golang.org/x/crypto'...
Cloning into '/Users/pivotal/workspace/golang/src/github.com/pivotal-cf-experimental/bosh-bootloader/vendor/gopkg.in/yaml.v2'...
fatal: unable to access 'https://gopkg.in/yaml.v2/': Could not resolve host: gopkg.in
fatal: clone of 'https://gopkg.in/yaml.v2' into submodule path '/Users/pivotal/workspace/golang/src/github.com/pivotal-cf-experimental/bosh-bootloader/vendor/gopkg.in/yaml.v2' failed
package github.com/pivotal-cf-experimental/bosh-bootloader/bbl: exit status 1

It appears that the submodule cannot be checked out.

Setup-infrastructure failed to authenticate when updating the stack, leaving behind a VM in status `CREATE_COMPLETE`

> bbl --version
bbl 0.0.1

git commit SHA 33a7fa5

These commands succeeded:

>  bbl \
    --state-dir bbl-infrastructure \
    --aws-access-key-id "$AWS_ACCESS_KEY_ID" \
    --aws-secret-access-key "$AWS_SECRET_ACCESS_KEY" \
    --aws-region us-east-1 \
    unsupported-deploy-bosh-on-aws-for-concourse

>  echo "$BBL_LB_CERT" > /tmp/bbl-cert
>  echo "$BBL_LB_KEY" > /tmp/bbl-key

>  bbl \
    --state-dir bbl-infrastructure \
    unsupported-create-lbs \
    --type=cf \
    --cert=/tmp/bbl-cert \
    --key=/tmp/bbl-key \
    --skip-if-exists

And then this failed to authenticate

>  bbl \
    --state-dir bbl-infrastructure \
    unsupported-update-lbs \
    --cert=/tmp/bbl-cert \
    --key=/tmp/bbl-key

step: creating keypair
step: generating cloudformation template
step: creating cloudformation stack
•••••••••••••••••••••
step: finished applying cloudformation template
step: generating bosh-init manifest
step: deploying bosh director
Deployment manifest: '/tmp/bosh-init975318574/bosh.yml'
Deployment state: '/tmp/bosh-init975318574/bosh-state.json'

Started validating
  Downloading release 'bosh'... Finished (00:00:03)
  Validating release 'bosh'... Finished (00:00:27)
  Downloading release 'bosh-aws-cpi'... Finished (00:00:00)
  Validating release 'bosh-aws-cpi'... Finished (00:00:00)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Downloading stemcell... Finished (00:00:00)
  Validating stemcell... Finished (00:00:00)
Finished validating (00:00:32)

Started installing CPI
  Compiling package 'ruby_aws_cpi/5e8696452d4676dd97010e91475e86b23b7e2042'... Finished (00:02:51)
  Compiling package 'bosh_aws_cpi/480f4b5c61db777243d8df4ca813ea78e4170336'... Finished (00:01:15)
  Installing packages... Finished (00:00:00)
  Rendering job templates... Finished (00:00:00)
  Installing job 'aws_cpi'... Finished (00:00:00)
Finished installing CPI (00:04:07)

Starting registry... Finished (00:00:00)
Uploading stemcell 'bosh-aws-xen-hvm-ubuntu-trusty-go_agent/3262'... Failed (00:00:05)
Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Command 'deploy' failed:
  creating stemcell (bosh-aws-xen-hvm-ubuntu-trusty-go_agent 3262):
    CPI 'create_stemcell' method responded with error: CmdError{"type":"Unknown","message":"AWS was not able to validate the provided access credentials","ok_to_retry":false}


exit status 1
/tmp/build/be72c34d/env-repo /tmp/build/be72c34d/env-repo /tmp/build/be72c34d
/tmp/build/be72c34d/env-repo /tmp/build/be72c34d
/tmp/build/be72c34d /tmp/build/be72c34d/env-repo /tmp/build/be72c34d
/tmp/build/be72c34d/env-repo /tmp/build/be72c34d

AWS Cloudformation Logs

2016-08-22  Status  Type    Logical ID  Status reason
15:15:01 UTC-0700   CREATE_COMPLETE AWS::CloudFormation::Stack  bbl-aws-1yP8d   
15:14:55 UTC-0700   CREATE_COMPLETE AWS::IAM::AccessKey BOSHUserAccessKey   
15:14:54 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::AccessKey BOSHUserAccessKey   Resource creation Initiated
15:13:42 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::AccessKey BOSHUserAccessKey   
15:13:38 UTC-0700   CREATE_COMPLETE AWS::IAM::User  BOSHUser    
15:12:36 UTC-0700   CREATE_COMPLETE AWS::EC2::EIP   NATEIP  
15:12:21 UTC-0700   CREATE_COMPLETE AWS::EC2::Route InternalRoute   
15:12:13 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::User  BOSHUser    Resource creation Initiated
15:12:06 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route InternalRoute   Resource creation Initiated
15:12:05 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   NATEIP  Resource creation Initiated
15:12:04 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route InternalRoute   
15:12:04 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   NATEIP  
15:12:00 UTC-0700   CREATE_COMPLETE AWS::EC2::Instance  NATInstance 
15:11:29 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH 
15:11:27 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH Resource creation Initiated
15:11:27 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH 
15:11:14 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH 
15:11:14 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH Resource creation Initiated
15:11:13 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH 
15:11:13 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Instance  NATInstance Resource creation Initiated
15:11:12 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Instance  NATInstance 
15:11:09 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    
15:11:07 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup BOSHSecurityGroup   
15:11:07 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup NATSecurityGroup    
15:11:05 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup NATSecurityGroup    Resource creation Initiated
15:11:05 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup BOSHSecurityGroup   Resource creation Initiated
15:11:03 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    
15:11:03 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    
15:11:03 UTC-0700   CREATE_COMPLETE AWS::EC2::Route BOSHRoute   
15:11:03 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation 
15:11:02 UTC-0700   CREATE_COMPLETE AWS::EC2::EIP   BOSHEIP 
15:11:02 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    
15:10:53 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    Resource creation Initiated
15:10:52 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    
15:10:51 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf 
15:10:50 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf 
15:10:49 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf Resource creation Initiated
15:10:49 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf 
15:10:49 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf Resource creation Initiated
15:10:49 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup BOSHSecurityGroup   
15:10:49 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf 
15:10:48 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup NATSecurityGroup    
15:10:48 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    Resource creation Initiated
15:10:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    Resource creation Initiated
15:10:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation Resource creation Initiated
15:10:47 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet1 
15:10:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    
15:10:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   BOSHEIP Resource creation Initiated
15:10:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route BOSHRoute   Resource creation Initiated
15:10:46 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation 
15:10:46 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    
15:10:46 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route BOSHRoute   
15:10:46 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   BOSHEIP 
15:10:45 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    Resource creation Initiated
15:10:45 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup InternalSecurityGroup   
15:10:44 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    
15:10:42 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   Resource creation Initiated
15:10:42 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    BOSHSubnet  
15:10:42 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet4 
15:10:41 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet3 
15:10:41 UTC-0700   CREATE_COMPLETE AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
15:10:41 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet2 
15:10:30 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 Resource creation Initiated
15:10:30 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 
15:10:27 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    InternalRouteTable  
15:10:26 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    BOSHRouteTable  
15:10:26 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 Resource creation Initiated
15:10:26 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  Resource creation Initiated
15:10:25 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    Resource creation Initiated
15:10:25 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  Resource creation Initiated
15:10:25 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  
15:10:25 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
15:10:25 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 
15:10:25 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet3 Resource creation Initiated
15:10:25 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   
15:10:25 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  
15:10:24 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 Resource creation Initiated
15:10:24 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  Resource creation Initiated
15:10:24 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet3 
15:10:24 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 
15:10:24 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  
15:10:21 UTC-0700   CREATE_COMPLETE AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
15:10:21 UTC-0700   CREATE_COMPLETE AWS::EC2::VPC   VPC 
15:10:05 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::User  BOSHUser    
15:10:05 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   Resource creation Initiated
15:10:03 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPC   VPC Resource creation Initiated
15:10:03 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
15:10:03 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPC   VPC 
15:09:57 UTC-0700   CREATE_IN_PROGRESS  AWS::CloudFormation::Stack  bbl-aws-1yP8d   User Initiated

@ChunyiLyu && jaresty

nokogiri failure [bosh-init step]

I can figure this out, but I wouldn't expect a new user to be able to, or to want to:

puffin:bin pivotal$ mkdir hi
puffin:bin pivotal$ ./bbl --aws-access-key-id "REDACTED" --aws-secret-access-key "REDACTED" --aws-region "us-east-1" --state-dir hi unsupported-deploy-bosh-on-aws-for-concourse
step: creating keypair
step: generating cloudformation template
step: creating cloudformation stack
•••••••••••••
step: finished applying cloudformation template
step: generating bosh-init manifest
step: deploying bosh director
Deployment manifest: '/private/var/folders/44/zr3txl1n45b23kd2kgx1xqq00000gn/T/bosh-init972161108/bosh.yml'
Deployment state: '/private/var/folders/44/zr3txl1n45b23kd2kgx1xqq00000gn/T/bosh-init972161108/bosh-state.json'

Started validating
  Downloading release 'bosh'... Finished (00:00:27)
  Validating release 'bosh'... Finished (00:00:00)
  Downloading release 'bosh-aws-cpi'... Finished (00:00:09)
  Validating release 'bosh-aws-cpi'... Finished (00:00:00)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Downloading stemcell... Finished (00:00:00)
  Validating stemcell... Finished (00:00:00)
Finished validating (00:00:38)

Started installing CPI
  Compiling package 'ruby_aws_cpi/a5b66d011ce1b31642ff148ea2c9097af65ff78c'... Finished (00:02:20)
  Compiling package 'bosh_aws_cpi/d7ffe4e7cd4cc233372185d8fd9374b737c3320a'... Failed (00:00:04)
Failed installing CPI (00:02:24)

Command 'deploy' failed:
  Installing CPI:
    Compiling job package dependencies for installation:
      Compiling job package dependencies:
        Compiling package:
          Running command: 'bash -x packaging', stdout: 'Installing rake 10.3.2
Installing CFPropertyList 2.3.1
Installing addressable 2.3.8
Installing json 1.8.3
Installing mini_portile 0.6.2

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

    /Users/pivotal/.bosh_init/installations/8038c0b8-35ba-4661-6336-b185771deb2d/packages/ruby_aws_cpi/bin/ruby extconf.rb
checking if the C compiler accepts ... yes
checking if the C compiler accepts -Wno-error=unused-command-line-argument-hard-error-in-future... no
Building nokogiri using packaged libraries.
-----
The file "/usr/include/iconv.h" is missing in your build environment,
which means you haven't installed Xcode Command Line Tools properly.

To install Command Line Tools, try running `xcode-select --install` on
terminal and follow the instructions.  If it fails, open Xcode.app,
select from the menu "Xcode" - "Open Developer Tool" - "More Developer
Tools" to open the developer site, download the installer for your OS
version and run it.
-----
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/Users/pivotal/.bosh_init/installations/8038c0b8-35ba-4661-6336-b185771deb2d/packages/ruby_aws_cpi/bin/ruby
    --help
    --clean
    --use-system-libraries
    --enable-static
    --disable-static
    --with-zlib-dir
    --without-zlib-dir
    --with-zlib-include
    --without-zlib-include=${zlib-dir}/include
    --with-zlib-lib
    --without-zlib-lib=${zlib-dir}/lib
    --enable-cross-build
    --disable-cross-build


Gem files will remain installed in /Users/pivotal/.bosh_init/installations/8038c0b8-35ba-4661-6336-b185771deb2d/packages/bosh_aws_cpi/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2 for inspection.
Results logged to /Users/pivotal/.bosh_init/installations/8038c0b8-35ba-4661-6336-b185771deb2d/packages/bosh_aws_cpi/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/ext/nokogiri/gem_make.out
An error occurred while installing nokogiri (1.6.6.2), and Bundler cannot
continue.
Make sure that `gem install nokogiri -v '1.6.6.2'` succeeds before bundling.
', stderr: '+ set -e -x
+ BOSH_PACKAGES_DIR=/Users/pivotal/.bosh_init/installations/8038c0b8-35ba-4661-6336-b185771deb2d/packages
+ cp -a bosh_aws_cpi/Gemfile bosh_aws_cpi/Gemfile.lock bosh_aws_cpi/bin bosh_aws_cpi/bosh_aws_cpi.gemspec bosh_aws_cpi/lib bosh_aws_cpi/scripts bosh_aws_cpi/spec bosh_aws_cpi/vendor /Users/pivotal/.bosh_init/installations/8038c0b8-35ba-4661-6336-b185771deb2d/packages/bosh_aws_cpi
+ bundle_cmd=/Users/pivotal/.bosh_init/installations/8038c0b8-35ba-4661-6336-b185771deb2d/packages/ruby_aws_cpi/bin/bundle
+ cd /Users/pivotal/.bosh_init/installations/8038c0b8-35ba-4661-6336-b185771deb2d/packages/bosh_aws_cpi
+ /Users/pivotal/.bosh_init/installations/8038c0b8-35ba-4661-6336-b185771deb2d/packages/ruby_aws_cpi/bin/bundle install --local --no-prune --deployment
':
            exit status 5

Maybe we could use something like https://github.com/phusion/traveling-ruby for the cpis?

bbl up fails when NAT box picks the same IP address as the BOSH director

Currently cloudformation will create a NAT box with a random IP address, which often works, but sometimes AWS will pick 10.0.0.6, which is the same IP address that BOSH wants. This causes a failure during up and a cryptic error message.

bbl version

step: checking if keypair "keypair-bbl-env-urmia-2016-10-13T19:17Z" exists
step: creating keypair
step: generating cloudformation template
step: checking if cloudformation stack "stack-bbl-env-urmia-2016-10-13T19-17Z" exists
step: creating cloudformation stack
•••••••••••
step: finished applying cloudformation template
step: generating bosh-init manifest
step: deploying bosh director
Deployment manifest: '/tmp/bosh-init273840685/bosh.yml'
Deployment state: '/tmp/bosh-init273840685/bosh-state.json'

Started validating
  Downloading release 'bosh'... Finished (00:00:11)
  Validating release 'bosh'... Finished (00:00:03)
  Downloading release 'bosh-aws-cpi'... Finished (00:00:01)
  Validating release 'bosh-aws-cpi'... Finished (00:00:00)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Downloading stemcell... Finished (00:00:13)
  Validating stemcell... Finished (00:00:00)
Finished validating (00:00:30)

Started installing CPI
  Compiling package 'ruby_aws_cpi/5e8696452d4676dd97010e91475e86b23b7e2042'... Finished (00:01:37)
  Compiling package 'bosh_aws_cpi/c2563067cbc20e3406ea36ca2984df2f0eaebcb0'... Finished (00:00:41)
  Installing packages... Finished (00:00:00)
  Rendering job templates... Finished (00:00:00)
  Installing job 'aws_cpi'... Finished (00:00:00)
Finished installing CPI (00:02:19)

Starting registry... Finished (00:00:00)
Uploading stemcell 'bosh-aws-xen-hvm-ubuntu-trusty-go_agent/3263.5'... Finished (00:00:05)

Started deploying
  Creating VM for instance 'bosh/0' from stemcell 'ami-bc0948ab light'... Failed (00:09:54)
Failed deploying (00:09:54)

Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Command 'deploy' failed:
  Deploying:
    Creating instance 'bosh/0':
      Creating VM:
        Creating vm with stemcell cid 'ami-bc0948ab light':
          CPI 'create_vm' method responded with error: CmdError{"type":"Unknown","message":"Address 10.0.0.6 is in use.","ok_to_retry":false}

This causes a situation which you may never recover from. The bosh-init manifest is hard coded to expect 10.0.0.6 for the BOSH director, and your NAT box is squatting on that address. The only way to fix this is to destroy your partially bbl'd up environment and start over. We'll need to hardcode the NAT box to use an address other than 10.0.0.6.

cc @zankich

concourse deployment should support ELB

Concourse web VMs should have an ELB in front of them, but currently there is no ELB information in the cloud config, meaning I have to download the cloud config, manually add ELB, and re-upload it, which means I'm working outside of the bbl tool, and I imagine that my config will be removed next time I run bbl (I haven't confirmed this though).

Also, we will need to configure cert information for the ELB somehow via bbl, otherwise we have to upload the certs out of band.

feature request: use SSL cert defined in Certificate Manager

AWS provides a mechanism to provide SSL certs that are hosted through them. This includes them being a trusted source, so you don't have to skip SSL validation.

It appears this could be done, by providing the name of the cert. Just needs to be provided as a command line argument.

bbl cloud-config does not provide all instance types in vm_types

E.g. c4.2xlarge was missing.

/cc @ematpl @nkaviani

Issue deleting cloudformation stack

Output

bbl --state-dir . --aws-access-key-id REDACTED --aws-secret-access-key REDACTED --aws-region us-east-1 destroy
Are you sure you want to delete your infrastructure? This operation cannot be undone! (y/N): y
step: destroying BOSH director and AWS stack
step: generating bosh-init manifest
step: destroying bosh director
Deployment manifest: '/private/var/folders/yq/cqd4_t1n4r1fvn82chvygmh00000gn/T/bosh-init152152389/bosh.yml'
Deployment state: '/private/var/folders/yq/cqd4_t1n4r1fvn82chvygmh00000gn/T/bosh-init152152389/bosh-state.json'

Started validating
  Downloading release 'bosh-aws-cpi'... Skipped [Found in local cache] (00:00:00)
  Validating release 'bosh-aws-cpi'... Finished (00:00:00)
  Validating cpi release... Finished (00:00:00)
Finished validating (00:00:00)

Started installing CPI
  Compiling package 'ruby_aws_cpi/c14af73d0f99b7571c4208b146f098a7af6ae7ab'... Finished (00:00:00)
  Compiling package 'bosh_aws_cpi/d30fd4f6cb22a5ad904dfe91d7499e061f1a2f54'... Finished (00:00:00)
  Installing packages... Finished (00:00:01)
  Rendering job templates... Finished (00:00:00)
  Installing job 'aws_cpi'... Finished (00:00:00)
Finished installing CPI (00:00:01)

Starting registry... Finished (00:00:00)

Started deleting deployment
  Waiting for the agent on VM 'i-3237f4a8'... Finished (00:00:00)
  Stopping jobs on instance 'unknown/0'... Finished (00:00:00)
  Unmounting disk 'vol-1e03c6bb'... Finished (00:00:08)
  Deleting VM 'i-3237f4a8'... Finished (00:01:08)
  Deleting disk 'vol-1e03c6bb'... Finished (00:00:08)
  Deleting stemcell 'ami-b23f3ed8 light'... Finished (00:00:06)
Finished deleting deployment (00:01:37)

Uninstalling local artifacts for CPI and deployment... Finished (00:00:00)
Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)
step: deleting cloudformation stack
••••••••••••••••••••••••••••••••••••••••••••••

CloudFormation failure on stack 'bbl-aws-oABBL'.
Check the AWS console for error events related to this stack,
and/or open a GitHub issue at https://github.com/pivotal-cf-experimental/bosh-bootloader/issues.

When we looked at AWS console we found that it failed to delete a vpc and subnet due to dependencies. Manually deleting the stack via AWS console worked fine.

Thanks,

Dennis & @APShirley

Readme is out of date

The readme instructs the user to use the commands

unsupported-create-lbs --type=<concourse,cf> --cert=<path> --key=<path> [--chain=<path>] [--skip-if-exists] "attaches a load balancer with the supplied certificate, key, and optional chain"
  unsupported-update-lbs --cert=<path> --key=<path> [--chain=<path>] [--skip-if-missing]                      "updates a load balancer with the supplied certificate, key, and optional chain"
  unsupported-delete-lbs [--skip-if-missing]

It appears that these commands are now subcommands to unsupported-deploy-bosh-on-aws-for-concourse I may be wrong on some syntax here, but the readme certainly does not reflect the current command state

bbl should have a teardown command

It should be called bbl burst or bbl pop.

Unhelpful error message when attempting to re-run failed create

± |master S:3 ✗| → bbl --aws-access-key-id=$(cat ~/workspace/deployments-rmd/bbl-concourse/aws_access_key_id) --aws-secret-access-key=$(cat ~/workspace/deployments-rmd/bbl-concourse/aws_secret_access_key) --state-dir="~/workspace/deployments-rmd/bbl-concourse/state" --aws-region=us-east-1 unsupported-deploy-bosh-on-aws-for-concourse
step: creating keypair
step: generating cloudformation template
step: creating cloudformation stack
•

aws cloudformation failed: ROLLBACK_COMPLETE

What do I do with this error? I tried to destroy but that didn't work:

± |master S:3 ✗| → bbl --aws-access-key-id=$(cat ~/workspace/deployments-rmd/bbl-concourse/aws_access_key_id) --aws-secret-access-key=$(cat ~/workspace/deployments-rmd/bbl-concourse/aws_secret_access_key) --state-dir="~/workspace/deployments-rmd/bbl-concourse/state" --aws-region=us-east-1 destroy
Are you sure you want to delete your infrastructure? This operation cannot be undone! (y/N): y
step: destroying infrastructure


stack not found

I deleted the Cloud Formation stack manually in the UI and this seems to allow me to progress.

Check that AWS calls will succeed before attempting the application of CloudFormation

Currently the application of Cloud Formation could fail at any point for reasons outside of bbl's control (such as not having enough VPCs available, not having enough elastic IPs available). Upon encountering these things, bbl will correctly fail and the CloudFormation stack will roll back, as desired.

However, it can take a while until it fails - up to 15 minutes - so in the interests of speeding up the feedback loop, it would be a better user experience if bbl could validate that it will not fail before starting to apply cloudformation. I understand that validating up-front does not guarantee that later cloud formation calls will succeed, but it will likely catch most of the situations where bbl fails part-way through the CF stack.

VPN Blocks 6868 and 25555

Although this is not a bbl issue wanted to call it out. Right now having to do bosh on AWS work outside of VPN, and when that is done, I can connect back to VPN. I have made the request to get those open, but is a challenge because of the wide range of IP's for AWS.

I may have started command while off vpn and then jumped onto VPN.

Cleaning up rendered CPI jobs... Finished (00:00:00)

Command 'deploy' failed:
Deploying:
Creating instance 'bosh/0':
Waiting until instance is ready:
Sending ping to the agent:
Performing request to agent endpoint 'https://mbus-user-WZAZmbBXHfqLb9h:[email protected]:6868/agent':
Performing POST request:
Post https://mbus-user-WZAZmbBXHfqLb9h:[email protected]:6868/agent: dial tcp XXX.XXX.XXX.XXX:6868: getsockopt: connection refused

unavailable availability zone

bbl assumes that the account being used has access to specific availability zones -- us-east-1b. My account is actually not able to produce things on that AZ.

The Cloud Formation template errors:

Value (us-east-1b) for parameter availabilityZone is invalid. Subnets can currently only be created in the following availability zones: us-east-1a, us-east-1c, us-east-1d, us-east-1e.

This happened while trying to create the subnets across AZs.

unsupported-create-lbs is not idempotent

It would be useful if unsupported-create-lbs did not fail if a lb has been previously created. (unsupported-deploy-bosh-on-aws-for-concourse performs a no-op. This functionality would be ideal for unsupported-create-lbs)

Compilation VM number is too small for timely CF deployment

At cloud ops we favor a pretty robust compilation VM size and number. If you do a deployment during the work day, programmer time costs alot more than VM time.

3 VM takes a long time to deploy CF. We set our cloud config to a more liberal 6.

bbl destroy should be idempotent even if stack is missing

see e.g. https://runtime.ci.cf-app.com/pipelines/cf-bosh-2-0/jobs/destroy-infrastructure/builds/2

I don't quite know what state the system was in before this build was run, but I would expect it to be idempotent and remove everything successfully.

bbl-deployed BOSH has insufficient permissions to upload stemcell

Feedback from @hhercules

bosh upload stemcell https://d26ekeud912fhb.cloudfront.net/bosh-stemcell/aws/bosh-stemcell-3215.4-aws-xen-ubuntu-trusty-go_agent.tgz --skip-if-exists
Acting as user 'user-UUzILhU' on 'my-bosh'

Using remote stemcell `https://d26ekeud912fhb.cloudfront.net/bosh-stemcell/aws/bosh-stemcell-3215.4-aws-xen-ubuntu-trusty-go_agent.tgz'

Director task 1
  Started update stemcell
  Started update stemcell > Downloading remote stemcell. Done (00:00:14)
  Started update stemcell > Extracting stemcell archive. Done (00:00:04)
  Started update stemcell > Verifying stemcell manifest. Done (00:00:00)
  Started update stemcell > Checking if this stemcell already exists. Done (00:00:00)
  Started update stemcell > Uploading stemcell bosh-aws-xen-ubuntu-trusty-go_agent/3215.4 to the cloud. Failed: Unknown CPI error 'Unknown' with message 'You are not authorized to perform this operation.' (00:08:48)

Error 100: Unknown CPI error 'Unknown' with message 'You are not authorized to perform this operation.'

Task 1 error

For a more detailed error report, run: bosh task 1 —debug

After updating policy attached to Bosh IAM user:

bosh upload stemcell https://d26ekeud912fhb.cloudfront.net/bosh-stemcell/aws/bosh-stemcell-3215.4-aws-xen-ubuntu-trusty-go_agent.tgz --skip-if-exists
Acting as user 'user-UUzILhU' on 'my-bosh'

Using remote stemcell `https://d26ekeud912fhb.cloudfront.net/bosh-stemcell/aws/bosh-stemcell-3215.4-aws-xen-ubuntu-trusty-go_agent.tgz'

Director task 4
  Started update stemcell
  Started update stemcell > Downloading remote stemcell. Done (00:00:08)
  Started update stemcell > Extracting stemcell archive. Done (00:00:04)
  Started update stemcell > Verifying stemcell manifest. Done (00:00:00)
  Started update stemcell > Checking if this stemcell already exists. Done (00:00:00)
  Started update stemcell > Uploading stemcell bosh-aws-xen-ubuntu-trusty-go_agent/3215.4 to the cloud. Done (00:08:48)
  Started update stemcell > Save stemcell bosh-aws-xen-ubuntu-trusty-go_agent/3215.4 (ami-116d9c71). Done (00:00:00)
     Done update stemcell (00:09:00)

Task 4 done

Started 2016-04-24 06:20:15 UTC
Finished 2016-04-24 06:29:15 UTC
Duration 00:09:00

Stemcell uploaded and created.

bbl should (optionally) write it's output to a S3 bucket to allow for bootstrapping from a cf pushed docker image

Customizable vm_types in cloud config

Similar to #28 but for vm_types; we need a way to be able to specific my how ephemeral disk a vm_type has - currently they all have 1 GB. Our concourse workers typically need about 100GB, and currently we have to edit the generated cloud_config and operate outside of bbl's visibility. We'd like a way to be able to choose the vm_types properties in such a way that bbl knows about them.

Workflow description to get started with CloudFoundry on AWS

As a follow-up on this issue: cloudfoundry/bosh#1298 (comment)

We would like to give this experimental new tooling a spin and see if we can get it to work, and provide you with feedback so we can get to a place where we can replace these docs with something that works sooner.

Could you describe a workflow that should get us started with CloudFoundry on AWS?

Cannot get help of an individual function with --help/help

I want to get help on a specific command, so I run:

○ → bbl unsupported-create-lbs --help


--aws-access-key-id must be provided

So I add dummy values and try again:

○ → bbl --aws-access-key-id='foo' --aws-secret-access-key='bar' --aws-region=baz unsupported-create-lbs --help


flag: help requested

and then finally I try removing -- from --help and see:

 2016-06-02 10:17:08 ☆ ruby 2.0.0p645 (system) rdimsdale in ~
○ → bbl --aws-access-key-id='foo' --aws-secret-access-key='bar' --aws-region=baz unsupported-create-lbs help


"" is not a valid lb type, valid lb types are: concourse and cf

User to use for ssh would be good to document

Since for on premises installs some users change to ubuntu I was unsure which user to use to login. I would be good to document that. New users and "seasoned" users like myself would find it helpful I think.

For AWS users to use when I looked it up found this, but I realize any could be used:
https://alestic.com/2014/01/ec2-ssh-username/

This made reference to use ubuntu or root for Ubuntu, but did not think to use vcap although had used that previously.

bbl does not update internally stored bosh-init state when the bosh director gets updated

If there is a reason for bosh-init to update the director (ie. the version of the director CPI changes), bosh-init will redeploy a new VM with that update. This produces an updated bosh-state.json file as an output of running bosh-init deploy.

Upon inspection of the code here, it looks like bbl will not update it's internal storage of this new state unless this state was previously empty.

In this case, the state is not empty, but was modified and should still be persisted. It seems like the fix would be to remove the above mentioned conditional so that the bosh state field always gets persisted.

Optionally tag infrastructure components

When using bbl to create multiple VPCs etc on the same account, it is hard to differentiate which components belong to which state.json.

For example, all the VPCs are named bbl and the subnets are Internal1, LoadBalancer2 etc.

If bbl supported an optional --tag we could prepend/append this tag to aid discovery, e.g. VPC-my-tag or IAAS-3-Loadbalancer2.

I haven't made an exhaustive list of the components this tag should be added to, but I imagine this would apply to more than just VPC and subnets.

Thanks,
Rob

manual cloud-config changes destroyed by `bbl unsupported-deploy-bosh-on-aws-for-concourse`

It would be nice if there were some way make manual updates to cloudconfig persist.

Currently my best bet is to update with bbl, fetch the cloud config, spruce merge in my additions (networks), then bosh update cloud-config1

Which isn't so bad; the worst part is that spruce messes up the nicely ordered hashes. It's so much easier to skim when the name keys appear first :)