chef / cheffish Goto Github PK
View Code? Open in Web Editor NEWResources and tools for testing and interacting with Chef and Chef Server.
License: Apache License 2.0
Resources and tools for testing and interacting with Chef and Chef Server.
License: Apache License 2.0
As a user, I'd like to be able to specify analytics rules and notifications using cheffish opposed to creating them through the UI and/or a knife command.
I was thinking this might look something like this
e.g.
chef_analytics_rule 'watch-admins-group' do
priority 0
message :run_converge
rule <<-EOH
when
#blah
then
#blah
EOH
notify 'blah'
end
I used chef-client 12.4.1 which has cheffish version 1.2. I hit issue #50 when trying to create organization with members.
I then used chef_gem at compile time to install version 1.5.0 (1.6.0 conflicts with chef-client 12.4.1) however I can not activate 1.5.0 since chef-client seems to have already activated 1.2. Only workaround (which kind of stinks) I have is to stop chef-client after 1.5.0 is installed and then trigger a new chef-client run and it seems to pick up latest version. Same result even if you have a chef_gem resource removing 1.2 (doesn't matter as its already activated) before installing 1.5.0. Is there anyway to get chef-client to deactivate cheffish 1.2 during chef-client run?
[poliva-chef-server-be-1a] Starting Chef Client, version 12.4.1
resolving cookbooks for run list: ["bb_chef_server_cluster::backend", "bb_chef_server_cluster::manage_rbac"]
Synchronizing Cookbooks:
- bb_apt
- bb_chef_server_cluster
- bb_lvm
- bb_nsupdate
- apt
- lvm
- chef-ingredient
- apt-chef
- yum-chef
- yum
Compiling Cookbooks...
Recipe: bb_chef_server_cluster::default
* chef_gem[install cheffish version 1.5.0] action install
- install version 1.5.0 of package cheffish
["1.5.0", "1.2"]
================================================================================
Recipe Compile Error in /var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb
================================================================================
Gem::LoadError
--------------
can't activate cheffish-1.5.0, already activated cheffish-1.2
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb:19:in `from_file'
Relevant File Content:
----------------------
/var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb:
12: output = `#{File.join(RbConfig::CONFIG['bindir'], 'gem')} list | grep cheffish`.slice(/\(.*\)/)
13: versions = output[1..-2].split(',').collect{|x| x.strip} unless output.nil?
14: versions ||= []
15: p versions
16:
17: require 'English'
18: #require 'cheffish'
19>> gem 'cheffish' , '=1.5.0'
20: require 'cheffish'
21:
22: ruby_block "only modify rbac content on master backend" do
23: block do
24: end
25: only_if { `/usr/bin/chef-server-ctl ha-status` && $CHILD_STATUS.to_i.eql?(0) }
26: only_if { `/usr/bin/chef-server-ctl ha-status | /bin/grep 'cluster status = master'` && $CHILD_STATUS.to_i.eql?(0) }
27: end
28:
Via IRC, it looks like the minimum version is 11.12 since thats when chef/http/remote_request_id
was added.
I'm not sure which gem's fault this is but story is I'm setting up cluster with chef-provisioning, my knife.rb is:
drivers({
'fog:DigitalOcean:staging' => {
:driver_options => {
:compute_options => {
:digitalocean_api_key => '<hash>',
:digitalocean_client_id => '<hash>'
}
}
}
})
I've also proper profile defined I'm running chef-client and I'm receiving '401 Unauthorized' from DO. chef-provisioning-fog is using Cheffish to get result config, extracts client_id and api_key.
result = Cheffish::MergedConfig.new(new_config, config, new_defaults)
...
id = result[:driver_options][:compute_options][:digitalocean_client_id]
When I'm running inspect on result[:driver_options]
I'm getting:
#<Cheffish::MergedConfig:0x007fe81fc63bf8 @configs=[{:compute_options=>{:provider=>"DigitalOcean", :digitalocean_client_id=>"staging"}}, {:compute_options=>{:digitalocean_api_key=>"<hash>", :digitalocean_client_id=>"<hash>"}}, {:compute_options=>{}}], @merge_arrays={}>
my driver id is used as 'digitalocean_client_id'. It should be digital_client_id hash.
Chef 12.1 was a while ago now, so perhaps the solution here is to end support for it in Cheffish. (Or explore earlier versions of chef-zero
, but that seems like a less good idea.)
Bundler could not find compatible versions for gem "ffi-yajl":
In Gemfile:
chef-zero (>= 0) ruby depends on
ffi-yajl (~> 2.2) ruby
chef (>= 0) ruby depends on
ffi-yajl (~> 1.2) ruby
chef-zero (>= 0) ruby depends on
ffi-yajl (~> 2.2) ruby
It's a little bit confusing that
attribute 'x', 'y'
does not modify other attributes,
but attributes 'x' => 'y'
removes all other attributes.
See #57
We pinned to v0 and we should update everything to use v1
Is there any way to create the node with default permissions, similar to how it's created via bootstrap?
Should probably live with chef/chef or follow the same release coordination as chef/chef and ohai will going forwards.
Having the rspec helpers in cheffish is weird as hell. Even the README.md makes it clear that its weird "here's a pile of resources... oh BTW here some 'bonus' rspec helpers...." WTF?
Now cheffish is pulled into chef/chef because of the dep that chef/chef's integration tests have on cheffish's rspec helpers which is another tangled nightmare of tightly-coupled dependencies.
And most likely since there's apparently not a lot of unit testing around the rspec helpers (at least I was able to break the tests that use them in chef/chef without breaking any spec tests in this repo) that suggests that the compatibility between the rspec helpers and old versions of chef is most likely a lie -- certain its untested and at any point may become a lie.
chef_group will create new group in your organization but the users do not appear to be added to group until 2nd chef-client run.
recipe snippet
url_with_org = {
:chef_server_url => URI.join(chef_server_url, "/organizations/blackberry").to_s,
:options => {
:client_name => "pivotal",
:signing_key_filename => "/etc/opscode/pivotal.pem" } }
chef_group 'ops' do
users ["ops-blackberry", "anbrown"]
complete true
chef_server url_with_org
action :create
end
1st chef-client run
* chef_group[ops] action create
- create group ops at https://127.0.0.1/organizations/blackberry
- update users from [] to ["ops-blackberry", "anbrown"]
knife group show ops -c ~/.chef/knife-chef12.rb (via knife-acl)
actors:
annotated_groups:
clients:
groupname: ops
groups:
name: ops
orgname: blackberry
users:
2nd chef-client run
* chef_group[ops] action create
- update group ops at https://127.0.0.1/organizations/blackberry
- update users from [] to ["ops-blackberry", "anbrown"]
- remove orgname
knife group show ops -c ~/.chef/knife-chef12.rb (via knife-acl)
actors:
anbrown
ops-blackberry
annotated_groups:
clients:
groupname: ops
groups:
name: ops
orgname: blackberry
users:
anbrown
ops-blackberry
The documented workaround in #37 doesn't seem to get passed through the HTTP client.
I posted a wrapper cookbook against the chef-server cookbook to bootstrap users (and organizations eventually) in https://gist.github.com/aespinosa/62bf4801ac570ab76f47a644b793b194
I have done various ways of adding ssl_verify_mode and none seem to work. So far I have tried the following
Chef::Config.ssl_verify_mode
inside a ruby_block
chef_server
parameter inside a lazy { Chef::Config.ssl_verify_mode ... }
blockIt would be nice to add following attributes to chef_organization as currently you can't set private key a new organization uses when creating its related validator.pem.
new attributes
attribute :validator_client_key # String or OpenSSL::PKey::*
attribute :validator_client_key_path , :kind_of => String
attribute :validator_client_key_pass_phrase
example usage
chef_organization orgname do
validator_client_key_path
end
It would be good to see chef_environment append to an environment so it is not just overwritten completely.
as in one recipe could in theory just add a specific context into the environment (for example in normal attributes)
then in another recipe to the same environment append a different context instead of overwriting all of normal attributes with what it decided.
Not sure if even feasible but never hurts to ask
Discovered that when you try to update an email of a existing user the chef_user blows up with 400 "Bad Request" error. Tried other fields like display_name and hit similar results.
Discovered using combination of cheffish 2.0.2
and chef-client 12.5.1
.
* chef_user[admin-helm] action create
["update user admin-helm at https://127.0.0.1/users", " update email from \"[email protected]\" to \"[email protected]\""]
{"name"=>"admin-helm", "username"=>"admin-helm", "display_name"=>"Admin Shared User for HELM Org", "admin"=>false, "json_class"=>"Chef::WebUIUser", "chef_type"=>"webui_user", "salt"=>nil, "password"=>nil, "openid"=>nil, "email"=>"[email protected]", "public_key"=>"-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqNKOarIC4dUS2SUdOuOZ\n0ACVepwMeoHuzQYdUtwVA1+fDXjLqfZy63b4KObCIPx8m4bdrWXw15Fr9eMrjw36\nBf4tPHxU4Jq+CivD1pW3QVAV4Z2cZ89mruOD4xwLfjycQ7/LWydhmGs59hC7l6Fk\nxxmDSoWVs3pmNs2Ug9ZRmN6OsAltb0JbMpLqFrleg2j4RalfLls24rOQqkKC5n8K\nAmef0zt3F5HaufxXbuBWBG1wmNCxAzQswTzpFUd1ZSSdXxPg3KOkNiZdY6GEt1AY\nJW4ReFWo5MISnv7JyNSEZWr0eGOJqzTEURGPrbJUWS6UbkioFdwLtBC+AsLk3Cvo\nBQIDAQAB\n-----END PUBLIC KEY-----\n"}
================================================================================
Error executing action `create` on resource 'chef_user[admin-helm]'
================================================================================
Net::HTTPServerException
------------------------
400 "Bad Request"
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb
92: chef_user user do
93: display_name values['display_name']
94: email values['email']
95: password lazy { determine_user_password(chef_server_url, user, values['password']) }
96: source_key values['source_key']
97: chef_server url_root
98: action :create
99: not_if { values['source_key'].nil? }
100: only_if { node.run_state['check_ha_status'] }
101: end
102: end
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb:92:in `block in from_file'
chef_user("admin-helm") do
action [:create]
retries 0
retry_delay 2
default_guard_interpreter :default
chef_server {:chef_server_url=>"https://127.0.0.1:443", :options=>{:client_name=>"pivotal", :signing_key_filename=>"/etc/opscode/pivotal.pem", :api_version=>"0"}}
declared_type :chef_user
cookbook_name "bb_chef_server_cluster"
recipe_name "manage_rbac"
display_name "Admin Shared User for HELM Org"
email "[email protected]"
password #<ChefCompat::CopiedFromChef::Chef::DelayedEvaluator:0x0000000584e770@/var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb:95>
source_key "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCo0o5qsgLh1RLZJR0645nQAJV6nAx6ge7NBh1S3BUDX58NeMup9nLrdvgo5sIg/Hybht2tZfDXkWv14yuPDfoF/i08fFTgmr4KK8PWlbdBUBXhnZxnz2au44PjHAt+PJxDv8tbJ2GYazn2ELuXoWTHGYNKhZWzemY2zZSD1lGY3o6wCW1vQlsykuoWuV6DaPhFqV8uWzbis5CqQoLmfwoCZ5/TO3cXkdq5/Fdu4FYEbXCY0LEDNCzBPOkVR3VlJJ1fE+Dco6Q2Jl1joYS3UBglbhF4VajkwhKe/snI1IRlavR4Y4mrNMRREY+tslRZLpRuSKgV3Au0EL4CwuTcK+gF"
not_if { #code block }
only_if { #code block }
end
Here is the complete stack trace:
Generated at 2016-05-12 14:01:32 +0000
Net::HTTPServerException: chef_user[admin-helm] (bb_chef_server_cluster::manage_rbac line 92) had an error: Net::HTTPServerException: 400 "Bad Request"
/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in `error!'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/http.rb:145:in `request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/http.rb:118:in `put'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/cheffish-2.0.2/lib/cheffish/chef_actor_base.rb:21:in `block in create_actor'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/mixin/why_run.rb:52:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/mixin/why_run.rb:52:in `add_action'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:175:in `converge_by'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/cheffish-2.0.2/lib/cheffish/chef_actor_base.rb:20:in `create_actor'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/cheffish-2.0.2/lib/chef/resource/chef_user.rb:42:in `block in <class:ChefUser>'
(eval):2:in `block in action_create'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/compat_resource-12.9.1/files/lib/chef_compat/copied_from_chef/chef/provider.rb:122:in `instance_eval'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/compat_resource-12.9.1/files/lib/chef_compat/copied_from_chef/chef/provider.rb:122:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:144:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource.rb:585:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:49:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in `block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in `each'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/resource_list.rb:83:in `block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/resource_list.rb:81:in `execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:80:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:653:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:648:in `catch'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:648:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:687:in `converge_and_save'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:269:in `run'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:270:in `block in fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:258:in `fork'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:258:in `fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:224:in `block in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:212:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:408:in `block in interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:398:in `loop'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:398:in `interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:388:in `run_application'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:60:in `run'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/bin/chef-client:26:in `<top (required)>'
/usr/bin/chef-client:54:in `load'
/usr/bin/chef-client:54:in `<main>'
All of our chef runs are updating ACLs every single time because the acl values are arrays and they are compared without sorting
Example chef run output -
update acl environments/environment at environments/environment
15:08:34.045 create: update groups from ["admins", "prod", "clients", "our"] to ["our", "admins", "prod", "clients"]�
15:08:34.045 �read: update groups from ["clients", "admins", "users", "our"] to ["our", "clients", "admins", "users"]�
15:08:34.046 �update, delete, grant: update groups from ["admins", "our"] to ["our", "admins"]�
Solution: Make the comparision insensitive to order
Similar problem to #83 but this time with chef_organization lwrp.
On subsequent runs it keeps reporting its removing guid for this organization.
* chef_organization[blackberry] action create
- update organization blackberry at https://127.0.0.1:443
- remove guid
Our spec runs are full of Net::HTTPServerException deprecation warnings. Now that we only support Ruby 2.6+ we should udpate all these exceptions to avoid warnings.
Dependabot couldn't parse the Gemfile found at /Gemfile
.
The error Dependabot encountered was:
Dependabot only supports uninterpolated string arguments to eval_gemfile. Got `__FILE__ + ".local"`
The raw_json
property of the chef_data_bag_item
resource is not used/respected when creating a data bag item
v16.11.7
Ubuntu-20.04
The following recipe can be used to replicate the issue.
my_data_bag_item = {}
my_data_bag_item['id'] = 'adam'
my_data_bag_item['real_name'] = 'Adam Brent Jacob'
chef_data_bag_item 'my_data_bag/adam' do
raw_json my_data_bag_item
action :create
compile_time true
end
example = data_bag_item('my_data_bag', 'adam')
log example.inspect
As you can see, the data bag only contains an id
which itself is computed from the name of the resource. Nothing provided via raw_json
gets added to the data bag.
-----> Starting Test Kitchen (v2.10.0)
-----> Converging <default-ubuntu-2004>...
Preparing files for transfer
Installing cookbooks for Policyfile /Users/gschofield/Workspace/progress/web_server/Policyfile.rb using `chef install`
Installing cookbooks from lock
Installing web_server 0.1.3
Preparing dna.json
Exporting cookbook dependencies from Policyfile /var/folders/s5/1w2wn2cs3v12x1pjv09gky8h0000gn/T/default-ubuntu-2004-sandbox-20210406-59850-mjt40t...
Exported policy 'web_server' to /var/folders/s5/1w2wn2cs3v12x1pjv09gky8h0000gn/T/default-ubuntu-2004-sandbox-20210406-59850-mjt40t
To converge this system with the exported policy, run:
cd /var/folders/s5/1w2wn2cs3v12x1pjv09gky8h0000gn/T/default-ubuntu-2004-sandbox-20210406-59850-mjt40t
chef-client -z
Removing non-cookbook files before transfer
Preparing data_bags
Preparing validation.pem
Preparing client.rb
chef installation detected
install_strategy set to 'once'
Nothing to install
Transferring files to <default-ubuntu-2004>
Starting Chef Infra Client, version 16.11.7
Patents: https://www.chef.io/patents
Using policy 'web_server' at revision 'c66cf3e426227ae1bb20fef2f4dba2ab656d52aa2f0fa8d992c688120dc2df2c'
resolving cookbooks for run list: ["web_server::[email protected] (58cfa44)"]
Synchronizing Cookbooks:
- web_server (0.1.3)
Installing Cookbook Gems:
Compiling Cookbooks...
Recipe: web_server::default
* chef_data_bag_item[my_data_bag/adam] action create
- create data bag item adam at chefzero://localhost:1
Converging 2 resources
* chef_data_bag_item[my_data_bag/adam] action nothing (skipped due to action :nothing)
* log[data_bag_item["my_data_bag", "adam", {"id"=>"adam"}]] action write
Running handlers:
Running handlers complete
Chef Infra Client finished, 1/3 resources updated in 01 seconds
Downloading files from <default-ubuntu-2004>
Finished converging <default-ubuntu-2004> (0m5.76s).
From our chef/chef spes:
/opt/chef/embedded/lib/ruby/gems/2.7.0/gems/cheffish-16.0.8/lib/cheffish/chef_run.rb:42: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
If you attempt to use an existing encrypted ssh_key but do not provide a pass_phrase, the keyformatter can't read key and is 'rescued' by returning nil.
The private_key class receives a 'nil' key and generates then overwrites the existing key.
This change causes a few issues:
1 - chef_server_api is expecting a MergedConfig object. When a MergedConfig object gets passed in, it dies on the merge! method because MergedConfig is not a hash. Like so:
NoMethodError
-------------
machine_batch[default] (/tmp/kitchen/cache/cookbooks/provisioner/providers/cluster.rb line 52) had an error:
NoMethodError: chef_client[QA1DLABTESTUBU01] (basic_chef_client::block line 132) had an error:
NoMethodError: Unexpected method merge! for MergedConfig with arguments [{:api_version=>"0"}]
2 - chef_server[:options] ||= {}
is null coalescing to a hash, when it should be null coalescing to an empty MergedConfig.
As a result, since chef-provisioning has a dependency on Cheffish of cheffish ~> 1.1
, this breaks a number of resources in the live version of chef-provisioning.
The expect_recipe
matcher currently converges the recipe. The matchers inside the AWS integration tests run converge themselves. We need to expose a new matcher that basically calls expect( recipe {...})
without converging the recipe.
The be_idempotent
matcher should check to see if the recipe that just ran had any failures. If it did, we shouldn't try to run it again - this will just cause the recipe to fail again and in the AWS case it will spin up more AWS resource. Inside be_idempotent
it looks like we can check chef_run#converge_failed?
Finally, when running the AWS tests (even without expect_recipe
) something is trying to converge a failing recipe an additional time. Adding a breakpoint here and you can see it getting called twice (if the recipe being converged raises an exception). A fix might be as simple as checking converge_failed?
and not re-converge, but I would like to understand why the converge is being called a second time on a failure. Is it called a second time on success? Is it called a second time if using expect_recipe
instead of expect(recipe {
?
When creating groups the server api endpoint used is https://localhost/groups/admins
but I keep getting back 404 errors.
[2015-02-23T17:18:30+00:00] DEBUG: Re-raising exception: Net::HTTPServerException - chef_group[admins](bb_chef12::_manage_rbac line 65) had an error: Net::HTTPServerException: 404 "Not Found"
I think the organization endpoints used is wrong according to the docs.
https://docs.chef.io/api_chef_server.html#groups
(Request) GET /organizations/NAME/groups
https://docs.chef.io/api_chef_server.html#groups-group-name
(Request) GET /organizations/NAME/groups/GROUP_NAME
I think the LWRP needs to take in an organization name to figure out the endpoint correctly.
chef-boneyard/chef-provisioning-aws#250 and #52 indicate that the behavior of expect_recipe
Since with_chef_server
is a helper as opposed to resource, arguments passed to can only be evaluated in the compile phase. There are situations where the Chef Server ip/hostname won't be known until halfway through the CCR...for example you might be provisioning the new Chef Server at the beginning of the CCR. By adding a Chef resource that updates the Chef server config like with_chef_server
we will get things like lazy attributes out of the box!
Support for policies or cookbooks is necessary to have a delightful cluster creation experience.
Idempotency of machine
resource in chef-provisioning is broken when working against Hosted Chef due to the fact that v1 API now does not return public_key
but the state comparison in cheffish checks for it.
Repro Steps:
machine
resourceSince Server API v1, all keys must be updated via the keys endpoint.
Details:
The logic here will break when running against v1 API because (pry debug values from that point):
[2] pry(#<Chef::Provider::ChefClient>)> a = new_json
=> {"name"=>"chef-server-backend-1",
"clientname"=>"chef-server-backend-1",
"admin"=>false,
"validator"=>false,
"chef_type"=>"client",
"orgname"=>"chef-infra",
"public_key"=>
"-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA00gA2sz6/Ki3geC99K06\nAoHMThQTlIoop5TJAnKdG9SUzl2ENT8ZD5cV85hiVNjTLQxcBDPBLxn9rMKjSXKF\nhuOkTvoy/PLTKcGKtfmspEv9VsqKqMlrj0TOeYGL02HsaqJ4hTrjpC7AMT7GWPhK\n0JbL8U3gYU3UltZ+gIdIrB3qzvJLXdWqcRR2nb2tEHnOooGXNmVUNr4OTmhsZ+Pk\nL1LqUkHtwh2k2aqRyccjcZkALTXitINLxixEHQzkDIzr7goADrK/sxCJG+s84Dv0\ngBMhF9eBIGacL8Y/Swsgysc7EGBoK1Bxl5BRRGGZcEh5jpag1HUSCd8vUz8bPjIT\nZwIDAQAB\n-----END PUBLIC KEY-----\n"}
[4] pry(#<Chef::Provider::ChefClient>)> b = current_json
=> {"name"=>"chef-server-backend-1",
"clientname"=>"chef-server-backend-1",
"admin"=>false,
"validator"=>false,
"chef_type"=>"client",
"orgname"=>"chef-infra"}
[5] pry(#<Chef::Provider::ChefClient>)>
Possible Fix:
Talking to @stevendanna we need to either pin to v0 or update it's calls to be compatible with v1.
Update would probably be to check if the new_json
has any public_key
and if it does call the /keys
endpoint to obtain the current keys and compare them with the ones in current_json
and if there are any differences call the /keys
endpoint to update them.
Since this is somewhat complicated pinning to v0 might be a temporary fix until we fix this the right hard way.
QUESTION - Is there a way to use cheffish to create a node attribute for the chef_orgnization? I would like to query a nodes chef_organization in a multi-tenancy environment.
Deprecated features used!
encrypt nil currently does not overwrite the value of encrypt. This will change in Chef 13, and the value will be set to nil instead. Please change your code to explicitly accept nil using "property :encrypt, [MyType, nil]", or stop setting this value to nil. at 1 location:
- /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.5.0/lib/chef/provider/chef_data_bag_item.rb:115:in `load_current_resource'
encryption_version nil currently does not overwrite the value of encryption_version. This will change in Chef 13, and the value will be set to nil instead. Please change your code to explicitly accept nil using "property :encryption_version, [MyType, nil]", or stop setting this value to nil. at 1 location:
- /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.5.0/lib/chef/provider/chef_data_bag_item.rb:116:in `load_current_resource'
Just documenting a dep warning I'm seeing.
Versions can be seen in the output.
If you have a private key with period characters in the name, you get down around here:
Line 103 in 087f683
Probably an elsif File.exists? key
block down at the end would fix it.
This differs from #12 in that these are primary objects (chef_acl '...' do)
Trying out cheffish 0.8, this bit of code is breaking in ec-metal: https://github.com/opscode/ec-metal/blob/master/cookbooks/ec-harness/recipes/ec2.rb#L22-L23
with_chef_local_server :chef_repo_path => repo_path,
:cookbook_path => [ File.join(harness_dir, 'cookbooks'),
File.join(repo_path, 'cookbooks'),
File.join(repo_path, 'vendor', 'cookbooks') ],
namely that clients cannot find any cookbooks from the second cookbook_path directory.
old behavior (0.7.x):
[2014-09-09T07:54:02-07:00] INFO: Starting chef-zero on port 9010 with repository at repository at /oc/src/ec-metal-aws-ubuntu
One version per cookbook
clients at /oc/src/ec-metal-aws-ubuntu/chef-repo/clients
cookbooks at /oc/src/ec-metal-aws-ubuntu/cookbooks, /oc/src/ec-metal-aws-ubuntu/chef-repo/cookbooks, /oc/src/ec-metal-aws-ubuntu/chef-repo/vendor/cookbooks
data_bags at /oc/src/ec-metal-aws-ubuntu/chef-repo/data_bags
environments at /oc/src/ec-metal-aws-ubuntu/chef-repo/environments
nodes at /oc/src/ec-metal-aws-ubuntu/chef-repo/nodes
roles at /oc/src/ec-metal-aws-ubuntu/chef-repo/roles
new behavior (0.8.x):
Compiling Cookbooks...
[2014-09-08T17:00:47-07:00] INFO: Starting chef-zero on port 9010 with repository at repository at /oc/src/ec-metal-aws-ubuntu/chef-repo
One version per cookbook
I found the documentation was unclear on how to set users key on :create action. The use in conjunction
phrase is very misleading. After reviewing the code it seems you either specify source_key
or source_key_pass_phrase
and source_key_path
. You can not use all three in combination as the documentation implies.
https://docs.chef.io/resource_chef_user.html
source_key
Use to copy a public or private key, but apply a different format and password. Use in conjunction with source_key_pass_phrase` and source_key_path.
source_key_pass_phrase
The pass phrase for the public key. Use in conjunction with source_key` and source_key_path.
source_key_path
The path to the public key. Use in conjunction with source_key` and source_key_pass_phrase.
Related code links:
https://github.com/chef/cheffish/blob/master/lib/cheffish/actor_provider_base.rb#L25
https://github.com/chef/cheffish/blob/master/lib/cheffish/actor_provider_base.rb#L73
I have no idea what this really does. Please add a more descriptive README with examples and whatever else documentation might be helpful :)
Thanks!
In commit ca752a7 the required version of ruby was bumped to 2.3.3. Chef-zero bumped its ruby requirement to 2.3.1
in commit chef/chef-zero@1f3952e, I've opened chef/chef-zero#268 to try to get that reverted, but 2.3.3
isn't even in chefdk! I can (somewhat) understand requiring 2.3.1
since that's what's shipped with chefdk, but 2.3.3
doesn't make any sense. Can this please be reverted or at least adjusted? Thanks!
Similar problem to #83 but this time with chef_group lwrp.
On subsequent runs it keeps reporting its removing orgname for this group.
* chef_group[noc] action create
- update group noc at https://127.0.0.1/organizations/blackberry
- remove orgname
Debugging a little bit I have discovered that the new_resource.data_bag
has a wrong value. I am calling the resource:
45: chef_data_bag_item 'keys/delivery_builder_keys' do
46: raw_data { DATA }
47: action :create
48: end
And the result is this:
[5] pry(#<Chef::Resource::ChefDataBagItem action provider>)> Chef::Cheffish::VERSION
(pry):5: warning: toplevel constant Cheffish referenced by Chef::Cheffish
=> "2.0.1"
[6] pry(#<Chef::Resource::ChefDataBagItem action provider>)> new_resource.name
=> "keys/delivery_builder_keys"
[7] pry(#<Chef::Resource::ChefDataBagItem action provider>)> new_resource.data_bag
=> "k"
[8] pry(#<Chef::Resource::ChefDataBagItem action provider>)> new_resource.id
=> "delivery_builder_keys"
using with with chef12 in most cases runs into problems with ssl verification, is there anything I can pass into the opts in Cheffish::ServerApi that would mimic adding ssl_verify_mode :verify_none
to my client.rb?
I dont think I fully understand how this gem fits in. Are there other chef technologies other than chef provisioning that use it? For example, I recently (and its not the first time) uncovered a snafu of my usage of the complete
attribute on the machine
resource. This is not an attribute that hangs directly off of machine
and appears to be defined here.
Usually I look to the chef gem for core chef workings but this gem seems to sit in the middle and it would help to understand the intent behind this gem to better attack provisioning troubleshooting.
Thanks!
The chef_user resource always fails when run against an Enterprise Chef server because it does not provide the display_name field in the API call, resulting in a 400 Bad Request from the server.
https://gist.github.com/charlesjohnson/38b20a18251b25a783b1
Display_name should be added to the resource and the provider.
It appears that its possible create a group and add other groups to this group. However I am confused about the acl implications. In spec example below does any user in group 'g' have all acl permissions set for group 'g' plus those also set for group 'x'? Which acl permissions would take precedence if group 'g' and 'x' rules are in conflict? Is it some kind of union? Sorry this probably isn't a cheffish related issue but I couldn't find these answers on chef docs (https://docs.chef.io/server_orgs.html) and thought someone here might know.
related spec: https://github.com/chef/cheffish/blob/master/spec/integration/chef_group_spec.rb#L109
context 'and has a group named x' do
group 'g', {}
group 'g2', {}
group 'g3', {}
group 'g4', {}
user 'u', {}
user 'u2', {}
user 'u3', {}
user 'u4', {}
client 'c', {}
client 'c2', {}
client 'c3', {}
client 'c4', {}
group 'x', {
'users' => %w(u u2),
'clients' => %w(c c2),
'groups' => %w(g g2)
}
it 'Converging chef_group "x" adds new users' do
expect_recipe {
chef_group 'x' do
users 'u3'
clients 'c3'
groups 'g3'
end
}.to have_updated('chef_group[x]', :create)
expect(get('groups/x')).to eq({
'name' => 'x',
'groupname' => 'x',
'orgname' => 'foo',
'actors' => %w(c c2 c3 u u2 u3),
'groups' => %w(g g2 g3),
'users' => %w(u u2 u3),
'clients' => %w(c c2 c3)
})
end
The chef_user will not work unless I specify :password
attribute. I'll get 400 bad request error if I try to comment attribute out.
Net::HTTPServerException
------------------------
400 "Bad Request"
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb
63: chef_user user do
64: display_name user
65: email "#{user}@#{org}.com"
66: #password "pass0wrd"
67: source_key_path key_path
68: chef_server url_root
69: action :nothing
70: subscribes :create, 'ruby_block[only modify rbac content on master backend]', :immediately
71: end
However even though the user is already created it keeps trying to update password in sequential chef-client runs. I can not get this resource to skip and report (update to date).
* chef_user[poliva] action create
- update user poliva at https://127.0.0.1/users
- update password from nil to "passw0rd"
* chef_user[anbrown] action create
- update user anbrown at https://127.0.0.1/users
- update password from nil to "passw0rd"
* chef_user[bbuczynski] action create
- update user bbuczynski at https://127.0.0.1/users
- update password from nil to "passw0rd"
* chef_user[eyurchenko] action create
- update user eyurchenko at https://127.0.0.1/users
- update password from nil to "passw0rd"
I noticed related comment here, so maybe known already, but didn't see any issue reported on this.
https://github.com/chef/cheffish/blob/master/lib/chef/resource/chef_user.rb#L25
On chef-client re-runs I noticed sometimes chef-acl would report (up to date)
but then other times it reported its updating even though no acls need to be changed. Seems like the order it finds out about current acls actually matters as the from
and to
lists below have same values but from
list is unsorted whereas to
list is sorted alphabetically.
* chef_acl[containers/sandboxes] action create
- update acl containers/sandboxes at containers/sandboxes
- create, update, delete: update groups from ["dev", "admins"] to ["admins", "dev"]
- read: update groups from ["clients", "users", "dev", "admins", "ops", "noc"] to ["admins", "clients", "dev", "noc", "ops", "users"]
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.