Git Product home page Git Product logo

ufo's Introduction

UFO: ECS Deploy Tool

Join the chat at https://gitter.im/tongueroo/ufo Support

BoltOps Badge

Please watch/star this repo to help grow and support the project.

UFO provides convenient tooling to make it easier and more fun to work with ECS. UFO tries to bring a heroku-like experience to ECS.

See ufoships.com for full documentation.

Important

If you are upgrading, please refer to the Upgrading docs

Installation

gem install ufo

Full installation instructions are at Install Ufo.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/tongueroo/ufo/issues.

QA Checklist

QA Checklist is a good list of things to check.

ufo's People

Contributors

aaronlippold avatar axel avatar breezeight avatar carlallen avatar epoberezhny avatar everplays avatar gurpreetatwal avatar hnatt avatar jdguzman avatar jlchenwenbo avatar tongueroo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ufo's Issues

Allow referencing existing Docker image from ECR

To create truly reliable deployments it should be possible to deploy a service and reference a Docker image from ECR instead of rebuilding/pushing it each time.

e.g. it could be done via --tag flag so that I can deploy my own image-name:deployed tag.

This will also allow UFO to run in Docker because it won't require a docker daemon running since it won't reference any docker commands.

catching migration fails

I've been playing with the ufo task with the wait flag to see if it will respond differently when the migration fails. From my tests, I don't see any difference. In a CI/CD setup, I need to know if the migrations fail and stop the deployment. Is there something I'm missing with regards to detecting a failed migration?

add `--no-verify-ssl` flag for ufo commands

Would it be possible to add a --no-verify-ssl flag to the cli so that use of ufo behind ssl proxies in corporate environments would be much more useful.

This is the error that I am getting when I have the proxy turned.

➜  variables git:(ufo-deploy) ✗ ufo init --launch-type ec2 --image=mitre/heimdall
Traceback (most recent call last):
	60: from /Users/aaronl/.rbenv/versions/2.6.3/bin/ufo:23:in `<main>'
	59: from /Users/aaronl/.rbenv/versions/2.6.3/bin/ufo:23:in `load'
	58: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/ufo-4.5.7/exe/ufo:14:in `<top (required)>'
	57: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
	56: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/ufo-4.5.7/lib/ufo/command.rb:43:in `dispatch'
	55: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	54: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	53: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	52: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor.rb:40:in `block in register'
	51: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:115:in `invoke'
	50: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/group.rb:232:in `dispatch'
	49: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:133:in `invoke_all'
	48: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:133:in `map'
	47: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:133:in `each'
	46: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:133:in `block in invoke_all'
	45: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	44: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	43: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/ufo-4.5.7/lib/ufo/init.rb:52:in `set_network_options'
	42: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/ufo-4.5.7/lib/ufo/network/helper.rb:10:in `configure_network_settings'
	41: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/memoist-0.16.0/lib/memoist.rb:170:in `vpc_id'
	40: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/ufo-4.5.7/lib/ufo/network/fetch.rb:15:in `vpc_id'
	39: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-ec2-1.102.0/lib/aws-sdk-ec2/client.rb:21765:in `describe_vpcs'
	38: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/seahorse/client/request.rb:70:in `send_request'
	37: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/seahorse/client/plugins/response_target.rb:23:in `call'
	36: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
	35: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
	34: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
	33: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
	32: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
	31: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/param_validator.rb:24:in `call'
	30: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/seahorse/client/plugins/endpoint.rb:45:in `call'
	29: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/endpoint_discovery.rb:78:in `call'
	28: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/endpoint_pattern.rb:28:in `call'
	27: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
	26: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/query/handler.rb:28:in `call'
	25: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/retry_errors.rb:178:in `call'
	24: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/retry_errors.rb:190:in `retry_if_possible'
	23: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/retry_errors.rb:207:in `retry_request'
	22: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/retry_errors.rb:178:in `call'
	21: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/retry_errors.rb:190:in `retry_if_possible'
	20: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/retry_errors.rb:207:in `retry_request'
	19: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/retry_errors.rb:178:in `call'
	18: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/retry_errors.rb:190:in `retry_if_possible'
	17: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/retry_errors.rb:207:in `retry_request'
	16: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/retry_errors.rb:176:in `call'
	15: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
	14: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in `call'
	13: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/plugins/signature_v4.rb:66:in `call'
	12: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'
	11: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/seahorse/client/plugins/content_length.rb:17:in `call'
	10: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/seahorse/client/net_http/handler.rb:47:in `call'
	 9: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/seahorse/client/net_http/handler.rb:73:in `transmit'
	 8: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/seahorse/client/net_http/handler.rb:121:in `session'
	 7: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/seahorse/client/net_http/connection_pool.rb:96:in `session_for'
	 6: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.61.2/lib/seahorse/client/net_http/connection_pool.rb:297:in `start_session'
	 5: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/2.6.0/delegate.rb:83:in `method_missing'
	 4: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/2.6.0/net/http.rb:925:in `start'
	 3: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/2.6.0/net/http.rb:930:in `do_start'
	 2: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/2.6.0/net/http.rb:996:in `connect'
	 1: from /Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/2.6.0/net/protocol.rb:44:in `ssl_socket_connect'
/Users/aaronl/.rbenv/versions/2.6.3/lib/ruby/2.6.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) (Seahorse::Client::NetworkingError)

upgrade to V5

I don't see any upgrade instructions for version 4.5 to 5.0. Just wanted to check if there is anything I should do for the upgrade?

Docker image fails to push

I am trying to build and push the demo, but am getting this error:

Building docker image with:
  docker build -t tongueroo/demo-ufo:ufo-2018-10-19T16-35-07-eb0789a -f Dockerfile .
Sending build context to Docker daemon    105kB
Step 1/10 : FROM ruby:2.5.1
 ---> 3c8181e703d2
Step 2/10 : WORKDIR /app
 ---> Using cache
 ---> 6f32cccb1763
Step 3/10 : ADD Gemfile /app/Gemfile
 ---> Using cache
 ---> 834f52b91aa7
Step 4/10 : ADD Gemfile.lock /app/Gemfile.lock
 ---> Using cache
 ---> 47da938fec85
Step 5/10 : RUN bundle install --system
 ---> Using cache
 ---> 85b22cc531ff
Step 6/10 : ADD . /app
 ---> Using cache
 ---> e3e5473cafda
Step 7/10 : RUN bundle install --system
 ---> Using cache
 ---> 876dd45bc20b
Step 8/10 : EXPOSE 4567
 ---> Using cache
 ---> e9f4e3a77519
Step 9/10 : RUN chmod a+x bin/*
 ---> Using cache
 ---> ea6323a7c4e4
Step 10/10 : CMD ["bin/web"]
 ---> Using cache
 ---> 0ad5ec7460fe
Successfully built 0ad5ec7460fe
Successfully tagged tongueroo/demo-ufo:ufo-2018-10-19T16-35-07-eb0789a
Docker image tongueroo/demo-ufo:ufo-2018-10-19T16-35-07-eb0789a built.  
Docker build took 0s.
=> docker push tongueroo/demo-ufo:ufo-2018-10-19T16-35-07-eb0789a
The push refers to repository [docker.io/tongueroo/demo-ufo]
0eba6c28d7bc: Preparing 
4f8e6bd38f44: Preparing 
b4446fb7ed07: Preparing 
01c08c0fd4ad: Preparing 
6da5e8658278: Preparing 
0f8c9434318c: Waiting 
a4172301657f: Waiting 
3696bfafb5ed: Waiting 
dda5ada3fa53: Waiting 
f5e615773459: Waiting 
9978d084fd77: Waiting 
1191b3f5862a: Waiting 
08a01612ffca: Waiting 
8bb25f9cdc41: Waiting 
f715ed19c28b: Waiting 
denied: requested access to the resource is denied
ERROR: The docker image fail to push.

I'm admittedly pretty new to Docker, but I believe the issue is that this process does not have permission to push to the Docker repository. However, I'm not understanding how to configure the repository. Is this something that needs to be done within the UFO framework config? How do you configure what repository to push to?

Health Check Grace Period

I had a look around and could not find any references to this setting for a service in a cluster. Is it possible to set this with UFO?

Variables are not available in params.yml

I would like to have different subnets and security groups for production and staging clusters and am looking for the ways to do this. Subnets and security groups are set in params.yml, but this file does not seem to have a notion of environments.

The docs on params.yml say:

ERB and variables are available in the params file

But I have a different experience. instance_variables in the context of params.yml ERB is empty and self seems to be an instance of Object without any additional methods.

So how do I access variables from the params files?

@environment layers is not operating as expected - values are getting over written

When I shift the call to the helper.env_file and the helper.env_vars together, the last one called overwrites the previous values in the array.

If I switch the call order I only get the last one.

Is this a bug or am I using the Layering incorrectly?

Allowing me to use them together would allow me to better separate out sensitive data.

Thanks


@image = helper.full_image_name # includes the git sha tongueroo/demo-ufo:ufo-[sha].
@cpu = 128
@memory_reservation = 256
@environment = helper.env_file(".env")
When ufo ship is ran with UFO_ENV=production the variables/production.rb will be evaluated and layered on top of the variables defined in base.rb:

.ufo/variables/production.rb:

@environment = helper.env_vars(%Q[
  RAILS_ENV=production
  SECRET_KEY_BASE=secret
])
When ufo ship is ran with UFO_ENV=development the variables/development.rb will be evaluated and layered on top of the variables defined in base.rb```

stuck at cloudformation

First of all, thanks for the ufo. It makes dealing with fargate a lot easier.

I have tried to deploy my own project to fargate using ufo. However, it got stuck on AWS::ECS::Service Ecs Resource creation Initiated of building the stack using cloudformation. So I thought maybe I am doing something wrong. However, I faced the same issue by following the quick start example (http://ufoships.com/quick-start):

(both cases using ufo v4.1.7)

Building Task Definitions...
Generating Task Definitions:
  .ufo/output/demo-web.json
  .ufo/output/demo-worker.json
  .ufo/output/demo-clock.json
Task Definitions built in .ufo/output
Equivalent aws cli command:
  aws ecs register-task-definition --cli-input-json file://.ufo/output/demo-web.json
demo-web task definition registered.
Deploying demo-web...
Ensuring log group for demo-web task definition exists
Log group name: ecs/demo-web
development cluster created.
Creating stack development-demo-web...
Generated template saved at: /tmp/ufo/development-demo-web/stack.yml
Generated parameters saved at: /tmp/ufo/development-demo-web/parameters.yml
08:38:34PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack development-demo-web User Initiated
08:38:38PM CREATE_IN_PROGRESS AWS::EC2::SecurityGroup EcsSecurityGroup
08:38:38PM CREATE_IN_PROGRESS AWS::EC2::SecurityGroup ElbSecurityGroup
08:38:38PM CREATE_IN_PROGRESS AWS::ElasticLoadBalancingV2::TargetGroup TargetGroup
08:38:38PM CREATE_IN_PROGRESS AWS::ElasticLoadBalancingV2::TargetGroup TargetGroup Resource creation Initiated
08:38:39PM CREATE_COMPLETE AWS::ElasticLoadBalancingV2::TargetGroup TargetGroup
08:38:42PM CREATE_IN_PROGRESS AWS::EC2::SecurityGroup EcsSecurityGroup Resource creation Initiated
08:38:42PM CREATE_IN_PROGRESS AWS::EC2::SecurityGroup ElbSecurityGroup Resource creation Initiated
08:38:44PM CREATE_COMPLETE AWS::EC2::SecurityGroup EcsSecurityGroup
08:38:44PM CREATE_COMPLETE AWS::EC2::SecurityGroup ElbSecurityGroup
08:38:46PM CREATE_IN_PROGRESS AWS::ElasticLoadBalancingV2::LoadBalancer Elb
08:38:47PM CREATE_IN_PROGRESS AWS::EC2::SecurityGroupIngress EcsSecurityGroupRule
08:38:47PM CREATE_IN_PROGRESS AWS::EC2::SecurityGroupIngress EcsSecurityGroupRule Resource creation Initiated
08:38:47PM CREATE_IN_PROGRESS AWS::ElasticLoadBalancingV2::LoadBalancer Elb Resource creation Initiated
08:38:48PM CREATE_COMPLETE AWS::EC2::SecurityGroupIngress EcsSecurityGroupRule
08:40:48PM CREATE_COMPLETE AWS::ElasticLoadBalancingV2::LoadBalancer Elb
08:40:51PM CREATE_IN_PROGRESS AWS::ElasticLoadBalancingV2::Listener Listener
08:40:51PM CREATE_IN_PROGRESS AWS::ElasticLoadBalancingV2::Listener Listener Resource creation Initiated
08:40:51PM CREATE_COMPLETE AWS::ElasticLoadBalancingV2::Listener Listener
08:40:54PM CREATE_IN_PROGRESS AWS::ECS::Service Ecs
08:40:54PM CREATE_IN_PROGRESS AWS::ECS::Service Ecs Resource creation Initiated

It never goes farther than the last line ^.

I thought maybe the generated stack.yml in /tmp can help to pinpoint the issue:

Description: "Ufo ECS stack demo-web"
Parameters:
  # required
  Vpc:
    Description: Existing vpc id
    Type: AWS::EC2::VPC::Id
  ElbSubnets:
    Description: Existing subnet ids for ELB
    Type: List<AWS::EC2::Subnet::Id>
  EcsSubnets:
    Description: Existing subnet ids for ECS
    Type: List<AWS::EC2::Subnet::Id>
  EcsSecurityGroups:
    Description: Existing ecs security group ids
    Type: String
    Default: ''
  ElbSecurityGroups:
    Description: Existing elb security group ids. List with commas.
    Type: String
    Default: ''

  ElbTargetGroup:
    Description: Existing target group
    Type: String
    Default: '' # when blank the automatically created TargetGroup is used
  CreateElb:
    Description: Create elb
    Type: String
    Default: true
  EcsDesiredCount:
    Description: Ecs desired count
    Type: String
    Default: 1
  EcsTaskDefinition:
    Description: Ecs task definition arn
    Type: String

  # Using to keep state
  ElbEipIds:
    Description: ELB EIP Allocation ids to use for network load balancer
    Type: String
    Default: ''
  EcsSchedulingStrategy:
    Description: The scheduling strategy to use for the service
    Type: String
    Default: 'REPLICA'
Conditions:
  CreateElbIsTrue: !Equals [ !Ref CreateElb, true ]
  ElbTargetGroupIsBlank: !Equals [ !Ref ElbTargetGroup, '' ]
  CreateTargetGroupIsTrue: !And
  - !Condition CreateElbIsTrue
  - !Condition ElbTargetGroupIsBlank
  ElbSecurityGroupsIsBlank: !Equals [ !Ref ElbSecurityGroups, '' ]
  EcsSecurityGroupsIsBlank: !Equals [ !Ref EcsSecurityGroups, '' ]
  EcsDesiredCountIsBlank: !Equals [ !Ref EcsDesiredCount, '' ]
Resources:
  Elb:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Condition: CreateElbIsTrue
    Properties:
      Type: application
      Tags:
      - Key: Name
        Value: demo-web
      # Add additional extra security groups if parameters set
      SecurityGroups: !Split
        - ','
        - !If
          - ElbSecurityGroupsIsBlank
          - !Ref ElbSecurityGroup
          - !Join [',', [!Ref ElbSecurityGroups, !Ref ElbSecurityGroup]]
      Subnets: !Ref ElbSubnets
      Scheme: internet-facing


  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Condition: CreateTargetGroupIsTrue
    Properties:
      VpcId: !Ref Vpc
      Tags:
      - Key: Name
        Value: demo-web
      Protocol: HTTP
      Port: 80
      TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
        Value: 10


  Listener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Condition: CreateElbIsTrue
    Properties:
      DefaultActions:
      - Type: forward
        TargetGroupArn:
          !If [ElbTargetGroupIsBlank, !Ref TargetGroup, !Ref ElbTargetGroup]
      LoadBalancerArn: !Ref Elb
      Protocol: HTTP
      Port: 80



  ElbSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Condition: CreateElbIsTrue
    Properties:
      GroupDescription: Allow http to client host
      VpcId: !Ref Vpc
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: '80'
        ToPort: '80'
        CidrIp: 0.0.0.0/0
      SecurityGroupEgress:
      - IpProtocol: tcp
        FromPort: '0'
        ToPort: '65535'
        CidrIp: 0.0.0.0/0
      Tags:
      - Key: Name
        Value: demo-web-elb


  Ecs:
    Type: AWS::ECS::Service
    DependsOn: Listener
    Properties:
      Cluster: development
      DesiredCount: !If
      - EcsDesiredCountIsBlank
      - !Ref AWS::NoValue
      - !Ref EcsDesiredCount
      TaskDefinition: !Ref EcsTaskDefinition
      # Default to port 80 to get template to validate.  For worker processes
      # there is no actual port used.
      LoadBalancers: !If
      - CreateTargetGroupIsTrue
      - - ContainerName: web
          ContainerPort: 4567
          TargetGroupArn: !Ref TargetGroup
      - !If
        - ElbTargetGroupIsBlank
        - []
        - - ContainerName: web
            ContainerPort: 4567
            TargetGroupArn: !Ref ElbTargetGroup
      SchedulingStrategy: !Ref EcsSchedulingStrategy


  EcsSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow http to client host
      VpcId: !Ref Vpc
      # Outbound access: instance needs access to internet to pull down image
      # or else get CannotPullContainerError
      SecurityGroupEgress:
      - IpProtocol: tcp
        FromPort: '0'
        ToPort: '65535'
        CidrIp: 0.0.0.0/0
        Description: outbound traffic
      Tags:
      - Key: Name
        Value: demo-web


  # Allow all traffic from ELB SG to ECS SG
  EcsSecurityGroupRule:
    Type: AWS::EC2::SecurityGroupIngress
    Condition: CreateElbIsTrue
    Properties:
      IpProtocol: tcp
      FromPort: '0'
      ToPort: '65535'
      SourceSecurityGroupId: !GetAtt ElbSecurityGroup.GroupId
      GroupId: !GetAtt EcsSecurityGroup.GroupId
      Description: application elb access to ecs


Outputs:
  ElbDns:
    Description: Elb Dns
    Condition: CreateElbIsTrue
    Value: !GetAtt Elb.DNSName

Not sure if it's relevant but I'm trying this on eu-west-1 (Ireland) region.

v4.4 release confused our deploy?

I'm still a bit confused by what happened exactly, but it seems related to the 4.4 upgrade. Hopefully you can provide some illumination...

Specifically, we have our CI system build and deploy our containers with UFO. This has been working flawlessly up until this morning. On our CI, we always just run gem install ufo (that's our bad for not version-locking it) before we run the ufo commands. It picked up 4.4 this morning, threw a bunch of warnings about the upgrade switching around how it names the stack, and then everything went nuts...

We're using custom cluster names:

staging:
  cluster: fancyapp-ufo-stag

production:
  cluster: fancyapp-ufo-prod

We did this to make it easier to read our stack and other resource names and disambiguate since we have multiple apps with different names. So we have stacks like fancyapp-ufo-stag-fancyapp-web and fancyapp-ufo-prod-fancyapp-web

We were deploying both our prod and stag servers, and all sorts of timeout errors occurred on ECS/Fargate, and then some stacks seemed to deploy and were updated, but were running the WRONG ENVIRONMENTS (e.g. our 'prod' stack was running 'RAILS_ENV staging` somehow)? I'm super-confused how, exactly, it ended up like this.

Adjusting our CI to always do gem install ufo -v '~> 4.3.0' fixed everything and it deployed as normal.

What exactly happened? And if we're using custom cluster names like above, how should we upgrade properly to 4.4 to avoid any issues?

Ufo vs ecs-deploy?

I'm currently using https://github.com/silinternational/ecs-deploy and recently discovered ufo.

Could you compare and contrast ufo ship and ecs-deploy? It's great that ufo has increased scope (build, task, more control over definitions, etc), but I'm curious how the core deployment workflow differs?

Production - Staging Environments

Hi, I'm new to ECS and I have an app running on it right now. I used this tutorial to get it working https://www.ybrikman.com/writing/2015/11/11/running-docker-aws-ground-up/. It works but every deployment is a very manual process. So I ran into this tool. It looks like exactly what I was looking for. My idea is to have CircleCi run ufo so my deployments are not manual anymore. Currently I only have CircleCi create and deploy the docker image to ECS. In looking at ufoships.com/docs, I have a question in how can I create a production and staging environment? Do they go in the same cluster? How does ufo handle them? Any help would be much appreciated.

Thanks!

High Availability with Spot Instances

I was wondering if UFO can be used to create a service that uses spot instances and falls back to on-demand instances by use of CloudFormation templates.

Has this been given any thought? Any work done on this?

Secrets in Container Definitions

This is a similar question to this: #63, however with some differences.

  1. I see that aws now has support for specifying a secrets object in the container definitions: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html

The questions I have are:

  1. is it possible to create custom helpers that can actually take a secrets environment variable and inject it into the container definitions, similar to the full_image_name and env_vars helpers. So I would do something like the below in base.rb or another variables file:
@secrets = helper.secrets(%Q[
    DB_PASSWORD=/my_app/db_password
    APP_KEY=/my_app/app_key
])

which would convert it to something like this:

[{
    "name": "DB_PASSWORD",
    "valueFrom": "/my_app/db_password"
},
{
    "name": "APP_KEY",
    "valueFrom": "/my_app/app_key"
}]

Which i would then add to the container definitions in main.json.erb like below:

<% if @secrets %>
      "secrets": <%= @secrets.to_json %>,
<% end %>

This would allow me to inject secrets into my containers using aws cloudformation's new way of injecting secrets.

  1. Is it possible to interpolate environment variables into our variable declaration, for instance in the example above:
@secrets = helper.secrets(%Q[
    DB_PASSWORD=/my_app/db_password
    APP_KEY=/my_app/app_key
])

changing that to be:

@secrets = helper.secrets(%Q[
    DB_PASSWORD=/my_app-{environment}/db_password
    APP_KEY=/my_app-{environment}/app_key
])

UFO not using image for environment if you switch AWS profiles

I'm trying to use UFO to deploy to different AWS accounts depending on the environment (e.g. staging / production). I'm also using ECR, so that means that the image field needs to be configurable because each AWS account has its own ECR registry. However, when setting the AWS profile, it doesn't seem to be using the correct variable.

Here is an example settings.yml file

base:
   image: default/image

staging:
   image: new/image
   aws_profiles:
      - staging

If I set the AWS profile to staging (by setting AWS_PROFILE environment variable) and then call ufo ship, it still uses the base image, instead of the correct one for the staging environment.

Explore more verbose logging on issues

When using both the EC2 and Fargate method, we should try to provide more detailed feedback on success and failure. If ECS is having issues rather than just seeming to halt, we should try to report more clearly so the user can self service.

#57 is a great example of where - I hope - we could have provided more guidance via the CLI.

Suggestions or thoughts on how we could improve the reporting?

new physical resource

I've been deploying to my Fargate stack without any issues, but starting last week, I get the message, Requested update requires the creation of a new physical resource; hence creating one. and then 3 hours later the message UPDATE FAILED. ECS did not stabilize.

This happened with both v 4.6.2 and v5.0.0. I have yet to find a solution, so I was wondering if you've ever run into this?

Edit default init files

Hi!

Is it somehow possible to change the templates for the files, that the ufo init-command generates?

Most of my services follow the same overall structure, which means I'll have to perform the same changes every time I initialize a new UFO setup.

Are the "default setup files" stored somewhere accessible, where they're possible to change?

I'm on a Mac, by the way :-)

gem install ufo misses 'render_me_pretty'

$sudo gem install ufo
Successfully installed ufo-3.0.0
Parsing documentation for ufo-3.0.0
Done installing documentation for ufo after 0 seconds
1 gem installed

$ ufo init
/Users/niels/.rbenv/versions/2.4.1/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:120:in require: cannot load such file -- render_me_pretty (LoadError)
from /Users/niels/.rbenv/versions/2.4.1/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:120:in require
from /Users/niels/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/ufo-3.0.0/lib/ufo.rb:8:in <top (required)>
from /Users/niels/.rbenv/versions/2.4.1/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:68:in require
from /Users/niels/.rbenv/versions/2.4.1/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:68:in require
from /Users/niels/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/ufo-3.0.0/exe/ufo:11:in <top (required)>
from /Users/niels/.rbenv/versions/2.4.1/bin/ufo:22:in load
from /Users/niels/.rbenv/versions/2.4.1/bin/ufo:22:in

After manually installing render_me_pretty everything was fine. Just to let you know.

ELB/Service and other naming conventions

Currently some of there resources get their names randomly suffixed which makes it hard to use and generates problematic domain names for ELBs.

screen shot 2019-02-15 at 2 44 11 pm
screen shot 2019-02-15 at 2 44 20 pm

  • A similar hack for docker image tagging ${UFO_DOCKER_BUILD_OPTIONS/:*/real_tag_that_i_want_instead_of_what_ufo_generates} is already a lot of work

Request

  • It would be nice to customize/template these and other generated names in a more human-readable form

deploying production error: did not find expected key while parsing a block mapping

When I deploy the default development cluster it works fine but when I set UFO_ENV=production I get the following error:

Creating stack production-logistics-app-web...
Traceback (most recent call last):
	28: from /Users/egjoka/.rbenv/versions/2.5.1/bin/ufo:23:in `<main>'
	27: from /Users/egjoka/.rbenv/versions/2.5.1/bin/ufo:23:in `load'
	26: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/exe/ufo:14:in `<top (required)>'
	25: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
	24: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/command.rb:43:in `dispatch'
	23: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	22: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	21: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	20: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/cli.rb:94:in `ships'
	19: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/cli.rb:94:in `each_with_index'
	18: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/cli.rb:94:in `each'
	17: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/cli.rb:99:in `block in ships'
	16: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/ship.rb:28:in `deploy'
	15: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/ship.rb:65:in `deploy_stack'
	14: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/stack.rb:52:in `deploy'
	13: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/stack.rb:67:in `perform'
	12: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/stack.rb:85:in `stack_options'
	11: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/stack.rb:170:in `save_template'
	10: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/stack.rb:81:in `template_body'
	 9: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/memoist-0.11.0/lib/memoist.rb:130:in `scope'
	 8: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/stack/context.rb:31:in `scope'
	 7: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/memoist-0.11.0/lib/memoist.rb:130:in `cfn'
	 6: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/stack/context.rb:250:in `cfn'
	 5: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/memoist-0.11.0/lib/memoist.rb:130:in `data'
	 4: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ufo-4.1.10/lib/ufo/setting/profile.rb:20:in `data'
	 3: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/2.5.0/psych.rb:263:in `load'
	 2: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/2.5.0/psych.rb:350:in `parse'
	 1: from /Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/2.5.0/psych.rb:402:in `parse_stream'
/Users/egjoka/.rbenv/versions/2.5.1/lib/ruby/2.5.0/psych.rb:402:in `parse': (<unknown>): did not find expected key while parsing a block mapping at line 6 column 1 (Psych::SyntaxError)

Service auto scaling properties

I would like to setup autoscaling policies for the cluster/service in ecs during the ufo deploy.

Example:

Scale up tasks and or EC2 instances if CPUUtilization exceeds 75%

Cloudformation: ELB & DNS are not created when using multiple containers in task definition template

First and foremost, thanks for this project, it's really amazing.

I'm having an issue when shipping my project with 2 containers in the web task definition, something similar to what is described in #56

I have a Nginx in front of my app container, here is the task definition template:

.ufo/templates/fargate.json.erb:

{
  "family": "<%= @family %>",
  "requiresCompatibilities": ["FARGATE"],
  "networkMode": "awsvpc",
  "cpu": "<%= @cpu %>",
  "memory": "<%= @memory %>",
  "containerDefinitions": [
    {
      "name": "nginx",
      "image": "<%= @nginx_image %>",
      "essential": true,
      "portMappings": [
        {
          "containerPort": "<%= @nginx_port %>",
          "protocol": "tcp"
        }
      ]
    },
    {
      "name": "<%= @name %>",
      "image": "<%= @image %>",
      <% if @container_port %>
      "portMappings": [
        {
          "containerPort": <%= @container_port %>,
          "protocol": "tcp"
        }
      ],
      <% end %>
      "command": <%= @command.to_json %>,
      <% if @environment %>
      "environment": <%= @environment.to_json %>,
      <% end %>
      <% if @secrets %>
      "secrets": <%= @secrets.to_json %>,
      <% end %>
      <% if @awslogs_group %>
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "<%= @awslogs_group %>",
          "awslogs-region": "<%= @awslogs_region || 'us-east-1' %>",
          "awslogs-stream-prefix": "<%= @awslogs_stream_prefix %>"
        }
      },
      <% end %>
      "essential": true
    }
  ]
}

I've also enabled ELB and DNS support for Cloudformation:

---
# Options allow you to customize any resources that ufo creates with
# CloudFormation. These options are inserting into the generated template.
# More info: https://ufoships.com/docs/customize-cloudformation

Elb:
  Scheme: internet-facing

# https://docs.aws.amazon.com/fr_fr/elasticloadbalancing/latest/APIReference/API_CreateTargetGroup.html
#
# When using SSL with network elb, the target group protocol is usually http still
# unless you also handle SSL termination at the app level.
TargetGroup:
  Port: 80 # only used with ECS if awsvpc mode
  # Protocol: TCP # valid values - application elb: HTTP HTTPS, network elb: TCP
                  # ufo sets defaults in cloudformation template
                  #   application elb: HTTP
                  #   network elb: TCP
                  # so we can keep this commented out, unless we need HTTPS at the app level
  # Health check settings are supported by application load balancer only:
  HealthCheckPath: /healthcheck # health check
  HealthCheckIntervalSeconds: 30 # default: 30. Network ELB can only take 10 or 30
  HealthyThresholdCount: 2
  UnhealthyThresholdCount: 5 # default: 10
  # HealthCheckProtocol: HTTP # HTTP or HTTPS
  # HealthCheckPort: traffic-port
  TargetGroupAttributes:
  - Key: deregistration_delay.timeout_seconds
    Value: 10

# https://docs.aws.amazon.com/fr_fr/elasticloadbalancing/latest/APIReference/API_CreateListener.html
#
# This is the default listener and normally should listen to port 80.
Listener:
  Port: 80
  # For Application Load Balancers, the supported protocols are HTTP and HTTPS. For Network Load Balancers, the supported protocol is TCP.
  # Protocol: TCP # valid values - application elb: HTTP HTTPS, network elb: TCP, TLS
                  # ufo sets these defaults:
                  #   application elb: HTTP  # unless port is 443
                  #   application elb: HTTPS # if port is 443
                  #   network elb: TCP  # unless port is 443
                  #   network elb: TLS  # if port is 443
                  # Can keep protocol commented out,
                  # unless need to override the defaults.
  # If using the listener to handle SSL
  # Certificates:
  # - CertificateArn: arn:aws:acm:us-east-1:111111111111:certificate/11111111-2222-3333-4444-555555555555

# An optional second listener can be created.
# If HTTPS and SSL is required then the listener_ssl config is what you should use.
#   Application ELBs support SSL termination.
#   Network load balancers do not and must pass the request through to the app
#   to handle SSL termination.
#
# ufo creates an ssl listener when listener_ssl is set.
ListenerSsl:
  Port: 443
  Certificates:
  - CertificateArn: arn:aws:acm:us-east-1:111111111111:certificate/11111111-2222-3333-4444-555555555555
  # Protocol: TCP # valid values - application elb: HTTP HTTPS, network elb: TCP, TLS
                  # ufo handles setting the defaults:
                  #   application elb: HTTPS
                  #   network elb: TLS

# Configure dns to automatically be associated with the ELB dns name.
# Note, the route53 record set for the domain name must already exist.
# The {stack_name} variable gets replaced with the name of the CloudFormation stack name.
# Example: {stack_name} => demo-web
Dns:
  Name: "sub.domain.com."
  HostedZoneName: domain.com. # dont forget the trailing period
  TTL: '60' # ttl has special upcase casing

This configuration works fine when creating ECS services and tasks, but it does not create any ELB and subdomain when doing ufo ship.

I've tested removing the ngnix container definition from .ufo/templates/fargate.json.erb, so having only my app container definition, and the ELB and DNS resources are created in Cloudformation. Am I missing something in the configuration? Thank you!

Command to compile Dockerfile.erb without build

I'm using docker-compose and from time to time I need to just compile the Dockerfile.erb so that I can use it with docker-compose without actually building the container. Is this possible with ufo?

Task definitions per environment

I don't like the idea of environments being handled through different versions of the same task definition name.

I'd rather have my-app-dev, my-app-prod, etc.

I see no way to customize the task definition name in ufo. Am I missing something or is this a new feature to be added?

Is NatGateway supported?

Is it possible to configure a nat gateway with UFO and launch ecs services in private subnets and route traffic through that nat gateway to public internet ?

Service Discovery using CloudMap

AWS CloudMap must be created at or before the ECS service. It would be a great feature if "ufo" could register the task/container with CloudMap during "ship".

ELB creation fails for tasks that do not include a container named "web"

Hello 👋

I just started investigating this project and I like the look of it so far, but I have encountered one issue when trying to run ufo deploy --elb=true for a service that does not include a web container.

I'm trying to deploy a Fargate service with a container named api like so:

deply ship --elb=true

During deployment, it encounters the following error and a rollback is triggered:

CREATE_FAILED AWS::ECS::Service Ecs The container web does not exist in the task definition. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: ...)

In the CFN stack generated on AWS I see a web container referenced in the LoadBalancer config, which in my case should be api:

Resources:
  Ecs:
    Type: AWS::ECS::Service
    Properties:
      ...
      LoadBalancers:
        Fn::If:
        - CreateTargetGroupIsTrue
        - - ContainerName: web    # incorrect, should be api
            ContainerPort: 80
            TargetGroupArn:
              Ref: TargetGroup
        - Fn::If:
          - ElbTargetGroupIsBlank
          - []
          - - ContainerName: web    # incorrect, should be api
              ContainerPort: 80
              TargetGroupArn:
                Ref: ElbTargetGroup
      ...
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      ...
      ContainerDefinitions:
      - Name: api                   # correct

I believe the problem is caused by the following lines being hardcoded, but I'm not sure if the dynamic container name is available in this template: https://github.com/tongueroo/ufo/blob/389d2f02370ff6556c7420fb4e775a39c6915884/lib/ufo/stack/builder/resources/ecs.rb#L35 https://github.com/tongueroo/ufo/blob/389d2f02370ff6556c7420fb4e775a39c6915884/lib/ufo/stack/builder/resources/ecs.rb#L46

Passing additional options to task command

My task definition uses FARGATE as a launch type, which I believe requires the awsvpc networking mode. That means in order to run my task, I need to be able to pass additional options to the run-task command, eg:

--network-configuration "awsvpcConfiguration={assignPublicIp=ENABLED,subnets=[subnet-aaa],securityGroups=[sg-bbb]}"

Would you accept a pull request for this, and if so, would you want additional options to run-task simply appended to the ufo run task command like the cluster option is?

Asset building/handling not covered in documentation

Hello,

Neat tool! I'm trying it out now and after reading the docs, I have an unanswered question that I both seek an answer to and simultaneously recommend you cover in your FAQ. Specifically, I'm trying to figure out how you handle asset building when using ufo.

In my current Dockerfile, I set RAILS_ENV using a build arg. A docker build command for me looks something like: docker build --build-arg RAILS_ENV=production --tag analytics-web:production-2018-05-12-XXXX . The assets then get built with the configuration of that environment, and RAILS_ENV gets set for the image by the Dockerfile rather than being set at runtime.

This presents two questions:

  • Since UFO doesn't seem to have a way to build environment-specific images, how do you recommend assets be built? Do you build them at deploy-time? Or do you build them into the image itself?
  • Is there a way to control how ufo generates the tags to, e.g., tag staging/development/production builds distinctly?

Thank you for sharing your hard work automating ECS deploys. I appreciate any answers you can share and also recommend incorporating them into the FAQ.

Undefined method error in generating Task Definition

Hi!

My setup might be a bit atypical, so I'm sorry if it's "out of scope", but after updating to the latest version, I'm no longer able to ship my services. I'm getting the following error:

Building Task Definitions...
Generating Task Definitions:
Traceback (most recent call last):
	17: from /usr/local/var/rbenv/versions/2.5.0/bin/ufo:23:in `<main>'
	16: from /usr/local/var/rbenv/versions/2.5.0/bin/ufo:23:in `load'
	15: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/ufo-3.5.1/exe/ufo:14:in `<top (required)>'
	14: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
	13: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/ufo-3.5.1/lib/ufo/command.rb:43:in `dispatch'
	12: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
	11: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
	10: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
	 9: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/ufo-3.5.1/lib/ufo/cli.rb:53:in `ship'
	 8: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/ufo-3.5.1/lib/ufo/tasks/builder.rb:8:in `ship'
	 7: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/ufo-3.5.1/lib/ufo/tasks/builder.rb:20:in `build'
	 6: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/ufo-3.5.1/lib/ufo/dsl.rb:18:in `run'
	 5: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/ufo-3.5.1/lib/ufo/dsl.rb:67:in `build_task_definitions'
	 4: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/ufo-3.5.1/lib/ufo/dsl.rb:67:in `each'
	 3: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/ufo-3.5.1/lib/ufo/dsl.rb:68:in `block in build_task_definitions'
	 2: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/ufo-3.5.1/lib/ufo/dsl/task_definition.rb:25:in `build'
	 1: from /usr/local/var/rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/ufo-3.5.1/lib/ufo/dsl/task_definition.rb:25:in `instance_eval'
./.ufo/task_definitions.rb:20:in `block in evaluate_template_definitions': undefined method `+' for nil:NilClass (NoMethodError)

My ufo files are like the following:

task_definitions.rb

task_definition "services-static-images-tsen" do
  source "main" # will use ufo/templates/main.json.erb
  variables(
    family: task_definition_name,
    name: task_definition_name,
    container_port: helper.dockerfile_port,
    environment: @environment + @tsen_environment,
    domain: @tsen_domain,
    awslogs_group: "ecs/tsen/services/static/images",
    awslogs_stream_prefix: @environment_name,
    awslogs_region: helper.current_region,
  )
end

base.rb

@image = helper.full_image_name # includes the git sha tongueroo/hi:ufo-[sha].
@cpu = 0
@memory_reservation = 128
@environment = [
	{name: "AWS_ACCESS_KEY", value: "XXX"},
	{name: "AWS_REGION", value: "XXX"},
	{name: "AWS_SECRET_KEY", value: "XXX"},
  ]

development.rb

# Example ufo/variables/development.rb
# More info on how variables work: http://ufoships.com/docs/variables/
@environment = @environment + helper.env_vars(%Q[
  ASPNETCORE_ENVIRONMENT=Development
])

@tsen_environment = [
  {name: "AWS_S3_BUCKET_NAME", value: "XXX"},
  {name: "SENTRY_DSN", value: "XXX"}
]

@tsen_domain = "api.example.com"
@environment_name = "staging"

My bet would be, that it has something to do with the environment concatenations. I'll try to narrow it down to figure out, where the error happens.

helper.env_file removes hash (#) at the end of an environment variable value in task definition

Hi there. First of all, thanks for this awesome gem.

One of my environment variables looks like this: SOME_KEY=WYNL3g!EvjSLG&W#

The helper method will always remove the trailing #, so in the resulting task definition file, the variable always looks like this:

{
  "name": "SOME_KEY",
  "value": "WYNL3g!EvjSLG&W"
 }

Adding double or single quotes around the value doesn't help. Any suggestions?

Bolts - ufo version

I'm using a mac and installed ufo via homebrew brew cask install boltopslabs/software/bolts. However I noticed that the ufo version it comes with is 2.0.0 and the latest release is 3.0.1. Been trying to find out how to update my ufo but can't find how. Any pointers would be much appreciated.

Parameterized Image Name by Environment

Is it possible to parameterize the image name in settings.yml so it can be changed by environment? my use case is deploying the same app to multiple environments hosted in different AWS accounts, so the ECR image name is different for each environment.

Automatic daemon service creation

Hi!
I understand that ufo will automatically create a new service if none exists yet -- is it possible for me to pre-define the scheduling strategy i.e. REPLICA vs DAEMON anywhere in the ufo configuration?

Thanks.

proxy task's exit code for one off tasks

I have integrated UFO into our continues deployment and so far, everything is working well. However, I can think of cases that the deployment needs to be stopped specially when dealing with migrations.

Currently, in the deployment script, I do something like ufo task web-worker -c bundle exec rake db:migrate before ufo ships web-{web,worker}. Even though it hasn't happened yet, I can think of scenarios where migration fails and I want to prevent the ufo ships command to happen. This can be implemented if ufo task returns the exit code of the execution on the server.

What I have in mind:

Adding a flag to ufo task to make it wait for the execution to finish with a desired timeout:

ufo task web-worker -t 300 -p -c bundle exec rake db:migrate
# -t for timeout, and -p for proxy unless you have a better suggestion.

Implementation:

It will be fairly simple:

  • use aws ecs wait tasks-stopped with the provided timeout.
  • use aws ecs describe-tasks to extract the exit code and exit with it.

I would like to implement this if you agree with the idea.

Issues after upgrading to version 3

Hey,

Thanks for the tool :-)

$ ufo version
3.0.1

I attempted upgrading my project through $ ufo upgrade3:

$ ufo upgrade3
Upgrading structure of your current project to the new ufo version 3 project structure
Upgraded settings: ufo/settings.yml
mv ufo .ufo
Upgrade complete.

Running $ ufo ship <service name> results in a seemingly required gem not being installed:

$ ufo ship commentr
/Users/martin/.rbenv/versions/2.4.2/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- tilt/erb (LoadError)
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/vendor/render_me_pretty/lib/render_me_pretty/erb.rb:42:in `<top (required)>'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/vendor/render_me_pretty/lib/render_me_pretty.rb:9:in `result'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/setting.rb:34:in `load_file'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/setting.rb:21:in `data'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/docker/builder.rb:54:in `image_name'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/docker/builder.rb:81:in `generate_name'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/docker/builder.rb:76:in `store_full_image_name'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/docker/builder.rb:24:in `build'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/cli.rb:127:in `build_docker'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/cli.rb:50:in `ship'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/command.rb:43:in `dispatch'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/exe/ufo:14:in `<top (required)>'
	from /Users/martin/.rbenv/versions/2.4.2/bin/ufo:23:in `load'
	from /Users/martin/.rbenv/versions/2.4.2/bin/ufo:23:in `<main>'

Running $ gem install tilt we get a bit onwards, but still no avail:

$ ufo ship commentr
/Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/docker/builder.rb:54:in `image_name': undefined method `[]' for nil:NilClass (NoMethodError)
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/docker/builder.rb:81:in `generate_name'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/docker/builder.rb:76:in `store_full_image_name'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/docker/builder.rb:24:in `build'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/cli.rb:127:in `build_docker'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/cli.rb:50:in `ship'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/lib/ufo/command.rb:43:in `dispatch'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
	from /Users/martin/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/ufo-3.0.1/exe/ufo:14:in `<top (required)>'
	from /Users/martin/.rbenv/versions/2.4.2/bin/ufo:23:in `load'
	from /Users/martin/.rbenv/versions/2.4.2/bin/ufo:23:in `<main>'

Not sure why setting.data is nil.

$ cat ufo/settings.yml:

# More info: http://ufoships.com/docs/ufo-settings/
image: <redacted>.dkr.ecr.eu-west-1.amazonaws.com/commentr
# clean_keep: 30
# ecr_keep: 30

aws_profile_ufo_env_map:
  default: stag

ufo_env_cluster_map:
  default: stag-web
  stag: stag-web
  prod: prod-web

Was converted into:

$ cat .ufo/settings.yml:

---
stag:
  aws_profiles:
  - default
  cluster: stag-web
default:
  cluster: stag-web
prod:
  cluster: prod-web
base:
  image: <redacted>.dkr.ecr.eu-west-1.amazonaws.com/commentr

Demo steps lead to stack trace - undefined method vpc_id

Following the demo here: http://ufoships.com/docs/tutorial-ufo-init/

I get this on the first ufo init command:

$ ufo init --app=demo --image=tongueroo/demo-ufo
Traceback (most recent call last):
        20: from /usr/local/bin/ufo:23:in `<main>'
        19: from /usr/local/bin/ufo:23:in `load'
        18: from /var/lib/gems/2.5.0/gems/ufo-4.1.4/exe/ufo:14:in `<top (required)>'
        17: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
        16: from /var/lib/gems/2.5.0/gems/ufo-4.1.4/lib/ufo/command.rb:43:in `dispatch'
        15: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
        14: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
        13: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
        12: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor.rb:40:in `block in register'
        11: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:115:in `invoke'
        10: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor/group.rb:232:in `dispatch'
         9: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:133:in `invoke_all'
         8: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:133:in `map'
         7: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:133:in `each'
         6: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:133:in `block in invoke_all'
         5: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
         4: from /var/lib/gems/2.5.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
         3: from /var/lib/gems/2.5.0/gems/ufo-4.1.4/lib/ufo/init.rb:52:in `set_network_options'
         2: from /var/lib/gems/2.5.0/gems/ufo-4.1.4/lib/ufo/network/helper.rb:10:in `configure_network_settings'
         1: from /var/lib/gems/2.5.0/gems/memoist-0.16.0/lib/memoist.rb:170:in `vpc_id'
/var/lib/gems/2.5.0/gems/ufo-4.1.4/lib/ufo/network/fetch.rb:18:in `vpc_id': undefined method `vpc_id' for nil:NilClass (NoMethodError)

NameError: uninitialized constant Module::ActiveSupport

Greetings.

After upgrading to 5.0.2 from 4.5.11, when trying to run ufo or bundle exec ufo I got this error:

bundler: failed to load command: ufo (/var/app/current/vendor/bundle/ruby/2.6.0/bin/ufo)
NameError: uninitialized constant Module::ActiveSupport
  /var/app/current/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/core_ext/module/deprecation.rb:21:in `deprecate'
  /var/app/current/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/core_ext/class/delegating_attributes.rb:26:in `<class:Class>'
  /var/app/current/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/core_ext/class/delegating_attributes.rb:6:in `<top (required)>'
  /var/app/current/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/core_ext/class.rb:2:in `require'
  /var/app/current/vendor/bundle/ruby/2.6.0/gems/activesupport-4.2.11.3/lib/active_support/core_ext/class.rb:2:in `<top (required)>'
  /var/app/current/vendor/bundle/ruby/2.6.0/gems/ufo-5.0.2/lib/ufo.rb:4:in `require'
  /var/app/current/vendor/bundle/ruby/2.6.0/gems/ufo-5.0.2/lib/ufo.rb:4:in `<top (required)>'
  /var/app/current/vendor/bundle/ruby/2.6.0/gems/ufo-5.0.2/exe/ufo:11:in `require'
  /var/app/current/vendor/bundle/ruby/2.6.0/gems/ufo-5.0.2/exe/ufo:11:in `<top (required)>'
  /var/app/current/vendor/bundle/ruby/2.6.0/bin/ufo:23:in `load'
  /var/app/current/vendor/bundle/ruby/2.6.0/bin/ufo:23:in `<top (required)>'

Modifying ufo-5.0.2/lib/ufo.rb as follows seems to have solved the issue for me:

$:.unshift(File.expand_path('../', __FILE__))
require 'active_support' #Added
require 'active_support/core_ext/class'

This is with Ruby 2.6.6, Rails 4.2.11 (ancient, yes :D) on Alpine/Debian.

single task security group

I'm would like to run a single task to perform a db migration, but the migration fails with a message about not being able to connect to the db. I'm going to guess that it is a security group issue. Any help you can provide on how to debug this would greatly appreciated.

The only difference I can see between the single task and the regular task, is that the single task has a group of family:stage-worker while the task has a group of service:stage-worker-stage-Ecs-.

Additional container in task definition

Hi there,

Congrats for the project. Quite impressive.

It's quite common to have a sidecar container where the image is built on every deploy. Is there a way to build/deploy 2 containers on a single task definition ? I am looking for way to deploy nginx + app containers but couldn't find a way to specify the second Dockerfile.

Thanks in advance.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.