Git Product home page Git Product logo

citadel's Introduction

Poise

Build Status Gem Version Cookbook Version Coverage Gemnasium License

What is Poise?

The poise cookbook is a set of libraries for writing reusable cookbooks. It provides helpers for common patterns and a standard structure to make it easier to create flexible cookbooks.

Writing your first resource

Rather than LWRPs, Poise promotes the idea of using normal, or "heavy weight" resources, while including helpers to reduce much of boilerplate needed for this. Each resource goes in its own file under libraries/ named to match the resource, which is in turn based on the class name. This means that the file libraries/my_app.rb would contain Chef::Resource::MyApp which maps to the resource my_app.

An example of a simple shell to start from:

require 'poise'
require 'chef/resource'
require 'chef/provider'

module MyApp
  class Resource < Chef::Resource
    include Poise
    provides(:my_app)
    actions(:enable)

    attribute(:path, kind_of: String)
    # Other attribute definitions.
  end

  class Provider < Chef::Provider
    include Poise
    provides(:my_app)

    def action_enable
      notifying_block do
        ... # Normal Chef recipe code goes here
      end
    end
  end
end

Starting from the top, first we require the libraries we will be using. Then we create a module to hold our resource and provider. If your cookbook declares multiple resources and/or providers, you might want additional nesting here. Then we declare the resource class, which inherits from Chef::Resource. This is similar to the resources/ file in an LWRP, and a similar DSL can be used. We then include the Poise mixin to load our helpers, and then call provides(:my_app) to tell Chef this class will implement the my_app resource. Then we use the familiar DSL, though with a few additions we'll cover later.

Then we declare the provider class, again similar to the providers/ file in an LWRP. We include the Poise mixin again to get access to all the helpers and call provides() to tell Chef what provider this is. Rather than use the action :enable do ... end DSL from LWRPs, we just define the action method directly. The implementation of action comes from a block of recipe code wrapped with notifying_block to capture changes in much the same way as use_inline_resources, see below for more information about all the features of notifying_block.

We can then use this resource like any other Chef resource:

my_app 'one' do
  path '/tmp'
end

Helpers

While not exposed as a specific method, Poise will automatically set the resource_name based on the class name.

Notifying Block

As mentioned above, notifying_block is similar to use_inline_resources in LWRPs. Any Chef resource created inside the block will be converged in a sub-context and if any have updated it will trigger notifications on the current resource. Unlike use_inline_resources, resources inside the sub-context can still see resources outside of it, with lookups propagating up sub-contexts until a match is found. Also any delayed notifications are scheduled to run at the end of the main converge cycle, instead of the end of this inner converge.

This can be used to write action methods using the normal Chef recipe DSL, while still offering more flexibility through subclassing and other forms of code reuse.

Include Recipe

In keeping with notifying_block to implement action methods using the Chef DSL, Poise adds an include_recipe helper to match the method of the same name in recipes. This will load and converge the requested recipe.

Resource DSL

To make writing resource classes easier, Poise exposes a DSL similar to LWRPs for defining actions and attributes. Both actions and default_action are just like in LWRPs, though default_action is rarely needed as the first action becomes the default. attribute is also available just like in LWRPs, but with some enhancements noted below.

One notable difference over the standard DSL method is that Poise attributes can take a block argument.

Template Content

A common pattern with resources is to allow passing either a template filename or raw file content to be used in a configuration file. Poise exposes a new attribute flag to help with this behavior:

attribute(:name, template: true)

This creates four methods on the class, name_source, name_cookbook, name_content, and name_options. If the name is set to '', no prefix is applied to the function names. The content method can be set directly, but if not set and source is set, then it will render the template and return it as a string. Default values can also be set for any of these:

attribute(:name, template: true, default_source: 'app.cfg.erb',
          default_options: {host: 'localhost'})

As an example, you can replace this:

if new_resource.source
  template new_resource.path do
    source new_resource.source
    owner 'app'
    group 'app'
    variables new_resource.options
  end
else
  file new_resource.path do
    content new_resource.content
    owner 'app'
    group 'app'
  end
end

with simply:

file new_resource.path do
  content new_resource.content
  owner 'app'
  group 'app'
end

As the content method returns the rendered template as a string, this can also be useful within other templates to build from partials.

Lazy Initializers

One issue with Poise-style resources is that when the class definition is executed, Chef hasn't loaded very far so things like the node object are not yet available. This means setting defaults based on node attributes does not work directly:

attribute(:path, default: node['myapp']['path'])
...
NameError: undefined local variable or method 'node'

To work around this, Poise extends the idea of lazy initializers from Chef recipes to work with resource definitions as well:

attribute(:path, default: lazy { node['myapp']['path'] })

These initializers are run in the context of the resource object, allowing complex default logic to be moved to a method if desired:

attribute(:path, default: lazy { my_default_path })

def my_default_path
  ...
end

Option Collector

Another common pattern with resources is to need a set of key/value pairs for configuration data or options. This can done with a simple Hash, but an option collector attribute can offer a nicer syntax:

attribute(:mydata, option_collector: true)
...

my_app 'name' do
  mydata do
    key1 'value1'
    key2 'value2'
  end
end

This will be converted to {key1: 'value1', key2: 'value2'}. You can also pass a Hash to an option collector attribute just as you would with a normal attribute.

Debugging Poise

Poise has its own extra-verbose level of debug logging that can be enabled in three different ways. You can either set the environment variable $POISE_DEBUG, set a node attribute node['POISE_DEBUG'], or touch the file /POISE_DEBUG. You will see a log message Extra verbose logging enabled at the start of the run to confirm Poise debugging has been enabled. Make sure you also set Chef's log level to debug, usually via -l debug on the command line.

Upgrading from Poise 1.x

The biggest change when upgrading from Poise 1.0 is that the mixin is no longer loaded automatically. You must add require 'poise' to your code is you want to load it, as you would with normal Ruby code outside of Chef. It is also highly recommended to add provides(:name) calls to your resources and providers, this will be required in Chef 13 and will display a deprecation warning if you do not. This also means you can move your code out of the Chef module namespace and instead declare it in your own namespace. An example of this is shown above.

Sponsors

The Poise test server infrastructure is generously sponsored by Rackspace. Thanks Rackspace!

License

Copyright 2013-2016, Noah Kantrowitz

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

citadel's People

Contributors

coderanger avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

citadel's Issues

Recommended approach for provisioning during image creation (packer etc)

Hey there, we're not quite sure of the cleanest way to proceed with configuring Packer for running cookbooks containing citadel calls. We either have to have a dev bucket with dev secrets that we pass to packer (this will work, as we already have this for our dev environments, but it feels sloppy to bake these into an image), or exclude certain recipes/resources based on an attribute - lots of conditionals all over the place. Have you run into this issue, and how did you work around it?

One idea I had was adding a global "killswitch" attribute that could be set that would turn citadel(..) into a no-op. I've seen this pattern in some other cookbooks (Pagerduty'd Sumologic CB, for example). If you're fine with this approach, I'll happily submit a PR. Thanks!

newline in key value due to the citadel method

hey guys,

thanks for writing this, its a great alternative to data-bags!

I'm seeing some wierdness at converge time though with the citadel method. It seems to be appending a newline to the value in my keyfile. in my test, instead of seeing the string supersecretsomestring I'm getting supersecret\nsomestring. Thanks!

example recipe.

puts "#{citadel["#{cookbook_name}/secret.pem"]}" + "somestring"

This is what my converge looks like in test-kitchen:

-----> Starting Kitchen (v1.3.1)
-----> Converging <default-base>...
       Preparing files for transfer
       Preparing dna.json
       Resolving cookbook dependencies with Berkshelf 3.2.3...
       Removing non-cookbook files before transfer
       Preparing validation.pem
       Preparing client.rb
-----> Chef Omnibus installation detected (install only if missing)
       Transferring files to <default-base>
       Starting Chef Client, version 12.4.1
       [2015-09-09T11:50:00+09:00] WARN: Child with name 'dna.json' found in multiple directories: /tmp/kitchen/dna.json and /tmp/kitchen/dna.json
       resolving cookbooks for run list: ["common::_test_citadel"]
       Synchronizing Cookbooks:
         - common
         - chef-client
         - apt
         - ntp
         - resolver
         - citadel
         - cron
         - logrotate
         - windows
         - chef_handler
       Compiling Cookbooks...
       Digest::Digest is deprecated; use Digest
       supersecret
       somestring
       Converging 0 resources

       Running handlers:
       Running handlers complete
       Chef Client finished, 0/0 resources updated in 2.46349934 seconds
       Finished converging <default-base> (0m11.37s).

citadel not able to download the object from Frankfurt region

We are facing an issue when trying to get the objects/secrets from eu-central-1 AWS region but it works fine in AWS us-west-2.

Relevant File Content:

/var/chef/cache/cookbooks/citadel/files/halite_gem/citadel/s3.rb:

78: "s3-#{region}.amazonaws.com"
79: end
80: Chef::Log.warn("citadel: path #{hostname}")
81: Chef::Log.warn("citadel: hostname #{hostname}")
82: begin
83: Chef::HTTP.new("https://#{hostname}").get("#{bucket}/#{path}", headers)
84: rescue Net::HTTPServerException => e
85>> raise CitadelError.new("Unable to download #{path}: #{e}")
86: end
87: end
88:
89: end
90: end
91:

[2018-05-15T06:17:03+00:00] ERROR: Unable to download keys/keys.json: 400 "Bad Request"

S3 best practice documentation

It's clear after a little research that the limitation of a single IAM role per instance is a key factor in the design of how to use citadel effectively.

It would be very helpful if the README gave an example of a recommended S3 structure. This seems like something that any existing user has had to think about and/or fight with, and passing that knowledge on to new adopters like myself would help a lot.

I'm happy to do the documentation itself if that helps.

Create ChefSpec

I ran into some problems when I create some testing for my recipe with file resource and citadel. I was able to set some fake credentials in my spec file, like this:

let(:chef_run) do
  runner = ChefSpec::ServerRunner.new do |node, server|
    node.set['citadel']['access_key_id'] = 'unittest'
    node.set['citadel']['secret_access_key'] = 'unittest'

    server.update_node(node)
  end
end

it 'download from S3' do
  expect(chef_run).to create_file('/tmp/s3_file.txt')
end
...

But citadel still tries to reach out to S3 and the test fails and any subsequent tests.

It would be great if you could provide a ChefSpec, so unit testing would work.

Thx

Chef::Exceptions::CookbookNotFound

I'm trying to include this cookbook include_recipe 'citadel' and getting Chef::Exceptions::CookbookNotFound.

I already included it in metadata.rb and Berksfile, and installed it with berks.

I'd really appreciate some guidance here as the documentation is not clear to me. Thanks.

NoMethodError - undefined method `[]' for nil:NilClass

I am building an ec2 instance which connects to a chef server on first boot. The run list includes a recipe which utilizes citadel to join a Windows active directory. This works as expected. But at the end of the chef run, an exception is raised. Any ideas what is causing this?

Issue appears to be Chef cannot define AWS credentials in line 67 but in my case, it successfully uses the IAM role

        access_key_id: role_creds['AccessKeyId'],
        secret_access_key: role_creds['SecretAccessKey'],
        token: role_creds['Token'],

Here is the log file

[2017-01-09T16:35:55+00:00] INFO: Chef Run complete in 315.272277643 seconds
[2017-01-09T16:35:55+00:00] INFO: Running report handlers
[2017-01-09T16:35:55+00:00] INFO: Report handlers complete
[2017-01-09T16:35:55+00:00] DEBUG: Re-raising exception: NoMethodError - undefined method `[]' for nil:NilClass
/var/chef/cache/cookbooks/citadel/files/halite_gem/citadel.rb:67:in `find_credentials'
  /var/chef/cache/cookbooks/citadel/files/halite_gem/citadel.rb:42:in `initialize'
  /var/chef/cache/cookbooks/citadel/files/halite_gem/citadel/chef_dsl.rb:26:in `new'
  /var/chef/cache/cookbooks/citadel/files/halite_gem/citadel/chef_dsl.rb:26:in `citadel'
  /var/chef/cache/cookbooks/users/recipes/join_domain.rb:99:in `block (2 levels) in from_file'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/property.rb:640:in `instance_exec'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/property.rb:640:in `exec_in_resource'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/property.rb:658:in `stored_value_to_output'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/property.rb:332:in `get'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/mixin/params_validate.rb:470:in `get'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/mixin/params_validate.rb:481:in `call'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/mixin/params_validate.rb:122:in `set_or_return'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/resource/execute.rb:64:in `command'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/resource.rb:547:in `block in identity'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/resource.rb:546:in `each'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/resource.rb:546:in `identity'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/resource_reporter.rb:64:in `for_json'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/resource_reporter.rb:270:in `block in prepare_run_data'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/resource_reporter.rb:269:in `map'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/resource_reporter.rb:269:in `prepare_run_data'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/resource_reporter.rb:227:in `post_reporting_data'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/resource_reporter.rb:208:in `run_completed'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/event_dispatch/dispatcher.rb:43:in `call'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/event_dispatch/dispatcher.rb:43:in `block in call_subscribers'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/event_dispatch/dispatcher.rb:34:in `each'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/event_dispatch/dispatcher.rb:34:in `call_subscribers'
  (eval):2:in `run_completed'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/client.rb:300:in `run'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/application.rb:294:in `block in fork_chef_client'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/application.rb:282:in `fork'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/application.rb:282:in `fork_chef_client'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/application.rb:247:in `block in run_chef_client'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/local_mode.rb:44:in `with_server_connectivity'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/application.rb:235:in `run_chef_client'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/application/client.rb:464:in `sleep_then_run_chef_client'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/application/client.rb:451:in `block in interval_run_chef_client'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/application/client.rb:450:in `loop'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/application/client.rb:450:in `interval_run_chef_client'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/application/client.rb:434:in `run_application'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/lib/chef/application.rb:59:in `run'
  /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.17.44/bin/chef-client:26:in `<top (required)>'
  /usr/bin/chef-client:57:in `load'
  /usr/bin/chef-client:57:in `<main>'
[2017-01-09T16:35:55+00:00] ERROR: Running exception handlers
[2017-01-09T16:35:55+00:00] ERROR: Exception handlers complete
[2017-01-09T16:35:55+00:00] ERROR: undefined method `[]' for nil:NilClass
[2017-01-09T16:35:55+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Doesn't work with AWS4-HMAC-SHA256

Apparently the cookbook can't be used with eu-central-1 region as it only supports v4 auth. As a result, this error is raised:

==> default: <Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message><RequestId>D030719A68950988</RequestId><HostId>FNWf9KANCf2qmcUDGP4ddOKWohUzz5XV4iatRJjn2QGyHegQOOpvd6E6mTtN4eIVsO9UgXKkdhc=</HostId></Error>

Initial implementation

Braindump from tonight:

  1. First step is to get the IAM Role name. This should be overridable by a node attr (since it is truly node-global) but you can find a sane default via http://169.254.169.254/latest/meta-data/iam/info via the InstanceProfileArn:

    "arn:aws:iam::\d+:instance-profile/<rolename>"
    
  2. Then grab http://169.254.169.254/latest/meta-data/iam/security-credentials/<rolename> and store them in the in the run state.

  3. Use these creds until they expire, check the expire datetime and refresh as needed (since a rotation cycle could intersect a chef run).

  4. These creds can now be used via the AWS gem to make S3 GetObject calls.

  5. Sketch of the recipe API:

    ci_deploy_key 'whatever' do
      key citadel['deploykey']['key.pem']
    end

    So the secret store is exposed as a method just like #node that supports #[] with the arguments being path sections to S3 files. Would citadel['deploykey/key.pem'] be clearer?

  6. For development in Vagrant check some node attributes for manual AWS credentials which can be set via the Vagrantfile (which in turn should come from env vars so the Vagrantfile can be checked in).

Issues:

This 100% only works on EC2 other than the insecure stuff for Vagrant.

IAM Roles have to be managed manually and an Instance can only have one IAM Role, this complicates stuff like some secrets being available to all machines, and some secrets mapping to chef roles.

newline in key value due to the citadel method ref issue #20

@coderanger I saw the issue #20 is closed because isn't possible to reproduce the problem, that the citadel adding a breakline when we create a template using them, but I can reproduce the behavior:

try to create a template like this:

template "/tmp/test" do
source "test.erb"
owner "root"
group "root"
mode 0660
variables({
:password => node['cookbook_teste]['password']
})
end

will generate this:

cat /tmp/test
password=P455word

if I create a file in my s3 bucket with this content:
P455word

and modify my recipe to use citadel like this:

template "/tmp/test_citadel" do
source "test.erb"
owner "root"
group "root"
mode 0660
variables({
:password => citadel['keys_secrets/password']
})
end

The result will be a /tmp/test_citadel with this content, including a breakline
P455word

diff between files /tmp/test /tmp/test_citadel

12:04:13 ip-10-0-54-20:~# diff /tmp/teste /tmp/teste_citadel
2a3

OHAI ec2 plugin

Great tool - thanks!

Running on EC2 in a VPC in ap-southeast-2 it took me some time to work out why I was receiving runtime errors on convergence.

My issue was the Ohai ec2 metadata plugin wasn't running as it uses mac address to determine if an node is in EC2 before running.

An ohai hint can be used to tell the plugin to run anyway, below is a quick snippet to do this:

include_recipe 'chef-sugar::default'

compile_time do
  directory '/etc/chef/ohai/hints' do
    recursive true
  end

  file '/etc/chef/ohai/hints/ec2.json'

  ohai 'reload_ec2' do
    action :reload
    plugin 'ec2'
  end
end

I hope this helps someone in future.

Cookbook Version 1.1.0 Silently Fails To Render S3 Content

Running into issue with version 1.1.0 of the cookbook. Using a standard file resource similar to the example. The cookbook will converge on my Test Kitchen node, but the file is empty. I set the constraint to pull 1.0.2 instead of the latest (1.1.0) and everything again worked as expected.

I also tested this with a JSON.parse to see if I could glean more information this way and indeed I received an error, instead of the silent fail/converge.

       JSON::ParserError
       -----------------
       A JSON text must at least contain two octets!

Something appears broken, or at least different, with cookbook version 1.1.0. Happy to provide more information if it helps. Cheers.

bucket name, https, static website hosting?

I had to do some arguably strange things to get this to (sorta) work:

I have node['citadel']['bucket'] set to "com.example.vcat.secrets"

I have this in the recipe:

file '/etc/rootonly/file-we-want' do
  owner 'root'
  group 'root'
  mode '600'
  content citadel['file-we-want']
end

I have the policy on the bucket set to this:

{
    "Version": "2008-10-17",
    "Id": "vcat",
    "Statement": [
        {
            "Sid": "vcat-secrets",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::0000000000:role/vcat"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::com.example.vcat.secrets/*"
        }
    ]
}

Citadel generates a url "https://com.example.vcat.secrets.s3.amazonaws.com/file-we-want" but the chef-client run fails because the hostname does not match the certificate. The README states

While citadel uses HTTPS, Chef does not verify certificates by default. You can enable verification by adding ssl_verify_mode :verify_peer to your client.rb.

I don't believe that is the case, anymore. I changed libraries/s3_file.rb to use http as a test, but that only works if I enable "static website hosting" on the bucket in question and is lame anyway. What am I missing here? Thank you in advance.

Chef: 12.2.1
Ubuntu: 14.04
Enterprise Hosted Chef
Citadel version '1.0.2'

EC2 instance can only have one IAM role

While the internals of the API seem to to suggest that at one time Amazon planned to support multiple roles per server, that isn't currently the case. This complicates managing access to overlapping sets of secrets (example, deploy keys vs. DB passwords) as you need to make composite roles, which is an O(N^2) problem in the end.

OpsWorks compatability

Would you be open to a pull request to make this cookbook usable on OpsWorks?

There are two issues that I see:

  1. OpsWorks is still running Chef 11.4.x on Ruby 1.8.7 - so use of Ruby 1.9 hash syntax needs to be changed
  2. I'm still troubleshooting this, but it appears that the EC2 metadata is either not properly parsed by Ohai or Opsworks stores somewhere other than under node['ec2']. A workaround could be to get it "manually" from the ec2 metadata via net/http or open-uri and json. I have this working in my environment and would be happy to include it in a PR.

Getting Net::HTTPServerException when running on an instance with no IAM role

We wrap citadel in a method that catches errors and uses fallback values (for builds and test kitchen). The new version of Citadel is throwing an error that seems incorrect. I included the output below but essentially it looks to me like metadata_service.get('latest/meta-data/iam/security-credentials/') is expected to return nil w/o an IAM role and instead 404s and returns Net::HTTPServerException:

    amazon-ebs: Net::HTTPServerException
    amazon-ebs: ------------------------
    amazon-ebs: 404 "Not Found"
    amazon-ebs:
    amazon-ebs: Cookbook Trace:
    amazon-ebs: ---------------
    amazon-ebs: /var/chef/cache/cookbooks/citadel/files/halite_gem/citadel.rb:58:in `find_credentials'
    amazon-ebs: /var/chef/cache/cookbooks/citadel/files/halite_gem/citadel.rb:42:in `initialize'
    amazon-ebs: /var/chef/cache/cookbooks/citadel/files/halite_gem/citadel/chef_dsl.rb:26:in `new'
    amazon-ebs: /var/chef/cache/cookbooks/citadel/files/halite_gem/citadel/chef_dsl.rb:26:in `citadel'
    amazon-ebs: /var/chef/cache/cookbooks/chefopslibs/libraries/secrets.rb:12:in `get_secret'
    amazon-ebs: /var/chef/cache/cookbooks/oc-sumologic/recipes/default.rb:10:in `from_file'
    amazon-ebs:
    amazon-ebs: Relevant File Content:
    amazon-ebs: ----------------------
    amazon-ebs: /var/chef/cache/cookbooks/citadel/files/halite_gem/citadel.rb:
    amazon-ebs:
    amazon-ebs: 51:        }
    amazon-ebs: 52:      elsif @node['ec2']
    amazon-ebs: 53:        role_creds = if @node['ec2']['iam'] && @node['ec2']['iam']['security-credentials']
    amazon-ebs: 54:          # Creds loaded from Ohai.
    amazon-ebs: 55:          @node['ec2']['iam']['security-credentials'].values.first
    amazon-ebs: 56:        else
    amazon-ebs: 57:          metadata_service = Chef::HTTP.new('http://169.254.169.254')
    amazon-ebs: 58>>         iam_role = metadata_service.get('latest/meta-data/iam/security-credentials/')
    amazon-ebs: 59:          if iam_role.nil? || iam_role.empty?
    amazon-ebs: 60:            raise 'Unable to find IAM role for node from EC2 metadata'
    amazon-ebs: 61:          else
    amazon-ebs: 62:            creds_json = metadata_service.get("latest/meta-data/iam/security-credentials/#{iam_role}")
    amazon-ebs: 63:            Chef::JSONCompat.parse(creds_json)
    amazon-ebs: 64:          end
    amazon-ebs: 65:        end
    amazon-ebs: 66:        {
    amazon-ebs: 67:          access_key_id: role_creds['AccessKeyId'],
    amazon-ebs:
    amazon-ebs: Platform:
    amazon-ebs: ---------
    amazon-ebs: x86_64-linux
    amazon-ebs:
    amazon-ebs: [2016-06-03T16:30:26+00:00] ERROR: Running exception handlers
    amazon-ebs: [2016-06-03T16:30:26+00:00] ERROR: Exception handlers complete
    amazon-ebs: [2016-06-03T16:30:26+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
    amazon-ebs: [2016-06-03T16:30:26+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
    amazon-ebs: [2016-06-03T16:30:26+00:00] ERROR: 404 "Not Found"

Issues with Citadel::S3.get

I've been debugging why the Citadel::S3.get method returns nil for me despite have all of the same inputs as another version of citadel (ignore which version, it's a fork we created to speed up development internally).

I did some hacking around in chef-shell against 1.1.0 in the chef supermarket and found that if I change the way the request is constructed the requests work as before (in our forked implementation):

Working implementation:

Chef::HTTP.new("https://#{bucket}.s3.amazonaws.com").get(path, headers)

Broken implementation

Chef::HTTP.new("https://#{hostname}").get("#{bucket}/#{path}", headers)

With the broken implementation I get:

chef:recipe (12.19.36)> Citadel::S3.get(bucket: @bucket, path: key, region: @region, **@credentials)
 => nil

When I monkey patch the Citadel::S3.get method with the correct implementation I get:

chef:recipe (12.19.36)> Citadel::S3.get(bucket: @bucket, path: key, region: @region, **@credentials)
 => "{}\n"

See:
https://github.com/poise/citadel/blob/master/lib/citadel/s3.rb#L77

Ideally, we'd move to the working implementation and bump the patch version of this cookbook so that everybody can use it.

Should be a relatively small amount of work to update the specs and change the line.

Handle key expiration

Right now the cookbook is just using the temporary credentials retrieved at the start of the Chef run. As the credentials just rotate every few hours irrespective of when they were retrieved, long Chef runs (or even short ones if you are unlucky) could see them expire during the run. The code should check the expiration before making a request, and get new credentials if needed.

Use STS to handle overlapping roles?

Because of the single role limitation of instance profiles mentioned in #3 I tend to have restricted roles that have the ability to assume the privileged roles with STS.

Providing assume role capabilities may be a convenient way to write recipes requiring overlapping permissions between roles. However, it still doesn't solve the complexities of configuring the roles for creating the instance profile role.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.