Support new YAML template features about stackup HOT 13 CLOSED

I'm kind of pissed at the way AWS have chosen to use (abuse?) YAML tags, in their adoption of YAML, e.g.

Outputs:
  LoadBalancerAddress:
    Value: !GetAtt
    - LoadBalancer
    - DNSName

as a shorthand for

Outputs:
  LoadBalancerAddress:
    Value: 
      Fn::GetAtt:
      - LoadBalancer
      - DNSName

As I understand it, tags are intended to denote types. The really annoying thing is that the first example above doesn't even round-trip via YAML!

irb> puts YAML.dump(YAML.load(raw))
---
Outputs:
  LoadBalancerAddress:
    Value:
    - LoadBalancer
    - DNSName

FFS! I wish they'd just treated YAML as an alternative serialisation format.

Anyway, If we want Stackup to support the new syntax, we have two options:

A. retain input template format/body (awscli-compatible)
B. parse-time conversion to data (backward-compatible)

Option A: retain input template format

To be fully compatible with what the AWS CLI (now) does, we'd have to retain the original YAML template text, and send it intact up to CloudFormation. As a result:

• AWS would take take of the funky tag handling
• we wouldn't lose tags used in the input YAML
• Stackup's behaviour would change (where we now send JSON, we'd be sending YAML)
• Stackup internals would have to change (because we currently treat templates as Ruby data)
• we would no longer be "normalising" the template at load time, so "diffs" would get funky, if either the format (JSON/YAML) for formatting (e.g. amount of whitespace) changed

Option B: parse-time conversion

A simpler change is to stick with our current approach - converting the input JSON/YAML to data within Stackup - and just add some magic to convert the tags to their equivalent JSON-compatible forms, e.g. !GetAtt to Fn::GetAtt. And then:

• most of Stackup would be unchanged
• diffs would continue to work
• bit it's a bit lossy, as tags are converted to their more verbose form
• going forward, we would have to match any other "YAML magic" AWS chose to implement

from stackup.

Actually, stackup has always supported templates (and parameters) in YAML format. Internally, it's just data, which we then serialise to JSON when invoking the CloudFormation API.

I see no advantage in uploading the original YAML to CloudFormation, do you?

from stackup.

Perfect, thanks Mike.

from stackup.

I've reopened this because, while we already support YAML, we don't support the new "abbreviated syntax for tags", e.g. !GetAtt vs Fn::GetAtt.

from stackup.

I'm not sure how much of an advantage it is but if a template is uploaded in YAML, get-template gives it back to you in YAML.

from stackup.

@lukeck: Damn. If that's true it means we probably can't away with converting the template into a canonical form (i.e. data) within Stackup; instead, we'll have to retain the original template text.

from stackup.

This thread is relevant to my interests.

from stackup.

I'm not in love with option A but option B looks like it's a recipe for an ongoing headache.

from stackup.

I'm not sure it's that much of an ongoing headache, @lukeck. The logic I've implemented in #32 is basically:

• if you see !Ref, substitute Ref
• if you see !SomethingElse, substitute Fn::SomethingElse

See

from stackup.

Closed with #32.

from stackup.

Stackup's implementation doesn't seem to support YAML AWS pseudo-parameter substitution via !Sub .

  SubscribeLambda:
    Type: "AWS::SNS::Subscription"
    Properties:
      Endpoint: !Sub |
        arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:addBuyProfile
      Protocol: lambda
      TopicArn: !Sub |
        arn:aws:sns:${AWS::Region}:${AWS::AccountId}:cbr-profiler

I concur with @lukeck assessment that option (b) will be a headache, due to this sort of thing

from stackup.

Additionally, there seems to be a trailing newlines that gets inserted into ARN strings in YAML, making them fail AWS validation

Template:

Resources:
  SnsPermission:
    Type: "AWS::Lambda::Permission"
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !Sub |
        arn:aws:lambda:ap-southeast-2:123456789012:function:myLambda
      Principal: sns.amazonaws.com
      SourceArn: !Sub |
        arn:aws:sns:ap-southeast-2:123456789012:*

Result (note the newline that has been retained at end of ARN is whats failing the regex validation)

INFO: [15:35:50] SnsPermission - CREATE_FAILED - 2 validation errors detected: Value 'arn:aws:sns:ap-southeast-2:123456789012:*
' at 'sourceArn' failed to satisfy constraint: Member must satisfy regular expression pattern: arn:aws:([a-zA-Z0-9\-])+:([a-z]{2}-[a-z]+-\d{1})?:(\d{12})?:(.*)

from stackup.

It should support !Sub, @benhutchison. I think the problem is that by using |, you've got newlines in the source YAML. Try without them, e.g.

 SubscribeLambda:
    Type: "AWS::SNS::Subscription"
    Properties:
      Endpoint: !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:addBuyProfile
      Protocol: lambda
      TopicArn: !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:cbr-profiler

If that still doesn't work, please raise a separate issue.

from stackup.