Check that AWS calls will succeed before attempting the application of CloudFormation about bosh-bootloader HOT 3 CLOSED

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/119121759

The labels on this github issue will be updated when the story is started.

from bosh-bootloader.

Hi @robdimsdale,

Would you mind telling us exactly what failed and how long it took to fail? Our expectation is that cloudformation should take less than 5 minutes, so we would be interested to hear if your experience is different than that.

Unfortunately, it is pretty hard for us to fail fast if the account is reaching its limits. AWS has default limits for most of the resource types that bosh-bootloader creates but all of these limits can be changed with a request to AWS.

Only two of the relevant limits are queryable by the API (using aws ec2 describe-account-attributes). Specifically, we are able to query the maximum number of elastic ips you can allocate for use in vpcs and the maximum number of on-demand instances that you can run.

Importantly, we cannot query the maximum number of VPCs or the maximum number of instances you are allowed to run of a given instance type (eg. m4.large for NAT and m3.xlarge for the BOSH director).

Most of the aws ec2 commands have --dry-run options, but they only fail if the given user does not have permissions. They do not fail if the instance limit has been reached.

All in all, we don't have too much power to fast fail but we would still like to know the details of your experience.

Thanks!
George and @kkallday

from bosh-bootloader.

That completely makes sense, and I understand that it may be quite challenging (or even impossible) to determine whether the execution will succeed before actually trying. I can't remember how long it took to fail - perhaps it was less than the five minutes you describe?

The types of failures I know I've seen before:

• Not enough VPCs
• Not enough elastic IPs (EIPs)
• Credentials have insufficient authorization for operations

Types of failures I might have hit before but I'm not sure:

• not enough VMs of a particular type

Thanks for the clarification, and if I run into any more failure modes I'll let you know.
Rob

from bosh-bootloader.