The ping sketch would be much more useful if it returned the ping status. This will implement and test #114

The temporary directories that are created are left behind, its not a big deal, they are empty, but it would be nice if they were removed when you are done with them.

This is what I got when I ran the vcs_mirror sketch:

Q: "...sr/bin/git clon": fatal: could not create leading directories of '/mirrors/design-center': Permission denied
I: Last 1 quoted lines were generated by promiser "/usr/bin/git clone -b master https://github.com/cfengine/design-center.git /mirrors/design-center"

cf-sketch> list -v

The following sketches are installed:

1. CFEngine::stdlib (library)
2. VCS::vcs_mirror (configured)
   Instance #1: (Activated on 'any')
   branch: master
   nowipe: !any
   origin: https://github.com/cfengine/design-center.git
   path: /mirrors/design-center
   runas: cfengine
   umask: 022
   vcs: /usr/bin/git

cf-sketch>

I was expecting it to create /mirrors/design-center

It would be great if I didn't have to hunt down the cf-sketch manual webpage to get a refresher on the usage options :)

"bypath" would work like this in parameters:

{
   "common": { mailto: "[email protected]" },
   "Mysketch": { mailto: { "bypath": [ "common", "mailto" ] } }
}

so essentially it's a way to reuse parameters across the whole configuration.

It was suggested this week that an autoclear mode be added to this sketch.

Looking for thoughts.
should it clear x time from ctime or mtime of the checked file, or x time from the first time CFEngine finds the file?

Clearing based on c or mtime gives the user more control and possibly is more understandable to the user.

What behaviour would you expect to happen if you removed and recreated the trigger file? or just touched the trigger file.

The idea being to make it increasingly more painful to work in cowboy mode but still allow the local admin to make the decision.

Nick, maybe you can help me with this one. It's an issue with the new cf-sketch interface. Everything looks good here; assuming you have DC in ~source/design-center this command works fine:

~/source/design-center/tools/cf-sketch/cf-sketch.pl --install CFEngine::stdlib --install-source=~/source/design-center/sketches/cfsketches --deactivate-all --install Repository::Yum::Client --activate Repository::Yum::Client=/home/tzz/source/design-center/sketches/package_management/yumclient/params/demo.json --generate
cf-agent -KI -f /home/tzz/.cfagent/inputs/cf-sketch-runfile.cf -Dverbose -Ddebug -Dtest

And the output, to me, looks good. But the actual editing is never done! I thought it might be because of the $(editor) bundle name in a variable, but if I use a literal string it's the same result.

R: DEBUG repo_yum_client_config  baseline baseurl=file:///var/www/html/yum_repos/myrepo
R: DEBUG repo_yum_client_config  baseline enabled=1
R: DEBUG repo_yum_client_config  baseline name=myrepo
R: DEBUG repo_yum_client_config  baseline gpgcheck=no
R: DEBUG repo_yum_client_config  empty_section: true, editor=$(editor)
R: DEBUG repo_yum_client_config  required baseline baseurl defined = file:///var/www/html/yum_repos/myrepo
R: DEBUG repo_yum_client_config  ensure_absent is not set; writing /tmp/baseline.repo
R: DEBUG repo_yum_client_config  repos_dir: /tmp
 !! Method invoked repairs
R: DEBUG repo_yum_client_config  repo1 gpgcheck=no
R: DEBUG repo_yum_client_config  repo1 name=repo1
R: DEBUG repo_yum_client_config  repo1 enabled=1
R: DEBUG repo_yum_client_config  repo1 baseurl=file:///var/www/html/yum_repos/myrepo
R: DEBUG repo_yum_client_config  required repo1 baseurl defined = file:///var/www/html/yum_repos/myrepo
R: DEBUG repo_yum_client_config  ensure_absent is not set; writing /tmp/repo1.repo
 !! Method invoked repairs
R: Found repo baseline in array cfsketch_g._001_Repository__Yum__Client_repos.  Will configure it in /tmp
R: Found repo repo1 in array cfsketch_g._001_Repository__Yum__Client_repos.  Will configure it in /tmp
 !! Method invoked repairs

When using defined_only option resolv.conf is edited once for each resolver index slist value on each activation.

As suggested by @goneri but without the Carp::Always dependency or global override.

It would be nice if the README.md files were all consistent format and information based on the sketch metadata.

I know with list I would find it useful to see the version of what’s installed, and it seems like searching based on metadata other than name might also be useful.

We should probably have them.
It would be cool it there was something to test with various versions automatically to help validate compatibility.

I started playing with the default.sub testing thing from core with the latest updates to Security::limits

With these settings:

  "users[jesse][gecos]"          string => "Jesse A";
  "users[jesse][uid]"            string => "502";
  "users[jesse][gid]"            string => "502";
  "users[jesse][home]"           string => "/home/jesse";
  "users[jesse][shell]"          string => "/bin/bash";
  "users[jesse][passwdhash]"     string => "$6$/Zx5Qr9k$wLTAuRpBUQNEEPnJhstQQe/rU/Veq69s.Ysqzm7EmSbQ9QZSxdJRR5YRUYP1jyRP5D4ddtlkzSXIzAGh6iNmV/";
  # Optional settings
  "users[jesse][groupname]"      string => "jesse";

I get weird non-convergent edits in /etc/shadow (the first line is inserted many times):

$(shadowentry[jesse])
 jesse:$6$/Zx5Qr9k$wLTAuRpBUQNEEPnJhstQQe/rU/Veq69s.Ysqzm7EmSbQ9QZSxdJRR5YRUYP1jyRP5D4ddtlkzSXIzAGh6iNmV/:15435:0:99999:7:::

I can't trace the problem but it seems regular expression-related. The same thing works fine for two other users.

What can I say, I'm lazy.

This is why I don't like too-clever AUTOLOADs. Can you please think about changing the code so it doesn't use AUTOLOADs? I'll try to fix this specific issue, but generally I don't want to be debugging layers upon layers of indirection.

perl -MCarp::Always ~/source/design-center/tools/cf-sketch/cf-sketch.pl --expert --search

https://raw.github.com/zzamboni/design-center/features/ease_of_use/sketches/libraries/copbl/sketch.json is not an object at /home/tzz/source/design-center/tools/cf-sketch/perl-lib/DesignCenter/Repository.pm line 39
    DesignCenter::Repository::AUTOLOAD('https://raw.github.com/zzamboni/design-center/features/ease_o...', '/home/tzz/.cfagent/inputs/sketches/CFEngine/stdlib/sketch.json') called at /home/tzz/source/design-center/tools/cf-sketch/perl-lib/DesignCenter/Repository.pm line 305
    DesignCenter::Repository::install('DesignCenter::Repository=HASH(0x10808a0)', 'ARRAY(0x11135a8)', 1) called at /home/tzz/source/design-center/tools/cf-sketch/perl-lib/DesignCenter/Repository.pm line 39
    DesignCenter::Repository::AUTOLOAD('https://raw.github.com/zzamboni/design-center/features/ease_o...', '/home/tzz/.cfagent/inputs/sketches/CFEngine/stdlib/sketch.json') called at /home/tzz/source/design-center/tools/cf-sketch/perl-lib/DesignCenter/Repository.pm line 305
    DesignCenter::Repository::install('DesignCenter::Repository=HASH(0x10808a0)', 'ARRAY(0x11135a8)', 1) called at /home/tzz/source/design-center/tools/cf-sketch/cf-sketch.pl line 97

https://github.com/ahdinosaur/design-center/blob/master/sketches/system/set_hostname/main.cf

Hi,
git doesn't clone if directory already exists. That wasn't okay because the repositories I wanted to mirror were sections of configuration in /etc/, not necessarily entire directories.

Here's a fix I made: https://github.com/ahdinosaur/design-center/blob/09d9045a6a4fade45f75163266393d4cec085d8c/sketches/utilities/vcs_mirror/main.cf. Probably not the best (I'm new to cfengine), but it works for me.

Thanks.

https://github.com/cfengine/design-center/tree/master/sketches/libraries/copbl

Just throwing out the idea for supporting multiple listen address statements.

Oracle RAC uses ssh to communicate, not uncommon to want ssh only listening on management and DB network.

Hi, when I try to use VCS::vcs_mirror with the nowipe option enabled it still wipes the directories of the git repositories clean.

Here's an example config json: https://github.com/ahdinosaur/command-center/blob/master/params/VCS/vcs_mirror/blue-dream-conf.d.json.

My solution was to comment out everything related to wiping, and now it works for me, but that doesn't seem appropriate. Thanks.

I think that cfengine_stdlib.cf is still missing some essential but trivial bodys (see eg. https://cfengine.com/forum/read.php?3,24119,24119#msg-24119). Before contributing sketches, I would like to know whether I should push bodys in cfengine_stdlib.cf or bundle them with sketches. There is also third choice: I can create my_own_stdlib.cf and use only that one.

Feel free to move the discussion elsewhere, if that is required.

If --fullpath is not enabled when running --generate, the produced runfile is badly broken, with at least the following:

• Inputs are specified as "Sketch/dir/file.cf" instead of "sketches/Sketch/dir/file.cf", so they are never found.
• The bundle_home variable is set to "Sketch/dir", which for some sketches (i.e. cloud_services, which uses it to find its shim.pl script) causes breakage.

Please look at this. CFEngine::stdlib should not be installed on a --search command.

perl -MCarp::Always ~/source/design-center/tools/cf-sketch/cf-sketch.pl --expert --search -v
....
Installing CFEngine::stdlib (CFEngine::stdlib) into /home/tzz/.cfagent/inputs/sketches

I think -Dverbose, -Dtest, and -Ddebug should be standard and recommended for all DC sketches, instead of ad-hoc debugging flags.

WDYT?

It would be nice if output could be colorized and prettified in general, when running in interactive mode (with output to a terminal).

Not sure why I didn't see this behaviour previously.

When an interface configuration file is edited it raises a class to signal an interface down and up is needed to apply the new configuration.

When running the agent manually with -KI over ssh the interface is downed after the configuration change but is not brought back up afterwards. Unsure why the ifup promise is not executing.

one solution would be to run a network restart but that's a little heavy handed as it affects all interfaces not just the one being edited.

11:43 @zzamboni the interactive configuration mode is still very basic, no good way to specify complex data structures
11:43 @zzamboni for "packages" it should ask you to enter the elements one by one, doesn't it?
11:46 < atsaloli_home> Parameter 'packages' must be a LIST().
11:46 < atsaloli_home> Please enter packages: php
11:46 < atsaloli_home> Sorry, but an unknown validation type LIST() was requested. We'll fail the validation, too.
11:46 < atsaloli_home> Invalid value, please reenter it.
11:46 < atsaloli_home> Please enter packages:
11:46 @zzamboni OK, that's a bug

(reported by Nick Anderson)

$ cf-sketch --search sysctl
System::sysctl /home/cmdln/src/cfengine/nickanderson/design-center/sketches/system/sysctl
$ cf-sketch --list sysctl
$ cf-sketch --install System::sysctl
Installing System::sysctl
Checking and installing sketch files.
Everything was up to date - nothing changed.
$ cf-sketch --list sysctl

It's taking almost 30 seconds to return a simple search

cf-sketch --expert --search vcs 0.98s user 0.19s system 3% cpu 29.913 total

I didnt do much debugging but I did run through it with perl debugger just looking for "slow" spots

97: $config->repository->install(['CFEngine::stdlib'], 1);
and
135: Parser::command_search(join('|', @{$config->search}));
were slow when I stepped through it with a debugger.
197: my %missing = map { $ => 1 } $self->missing_dependencies($data->{metadata}->{depends});
was slow

Looks like the entry point was lost at some point.

Whenever cf-sketch does a color_die, your command prompt and anything you type stays red until you reset it. Don't know the proper "Perl way" of doing it, but I fixed it awk-style on my end:

END {
    print RESET;
}

Design center seems like a good place for the stigs.

I think this is why I was using set_variable_values2.

I believe this is caused by the classes that set_variable_values sets when a line is detected.

classes => if_ok("$(cindex[$(index)])_in_file"),

so considering that ONBOOT is one of the settings in an interface files that will define something like ONBOOT_in_file if it exists.

All is fine and well for editing the first interface configuration. When I add another interface a new config file is created but ONBOOT isnt added to the file because the class ONBOOT_in_file exists (because of the promises for the first interface).

So where should this be fixed? I would say that the global class set by set_variable_values is not sufficiently unique. Suggestions on what should be added to the set_variable_values classes to make them more unique?

I may just pull set_variable_values into the sketch so that I can make it unique by interface name.

Thoughts?

I'd like to have sketch return values. Within the cf-sketch context, I'm thinking of simply specifying a optional "return_value" key in the metadata, which each sketch is free to write into. There would be a class "can_return_value" available when the "return_value" key is not null. Then other sketches can chain their execution on top of the return value.

Please comment.

Paths in the generated runfile should be relative so that it works well regardless of being copied across systems (for example, copied from /var/cfengine/masterfiles on the server to /var/cfengine/inputs on a client).

If you have a moment, take a look. I have been playing with defaults recently, they are used in the sketch, probably wont ever be triggered if your using cf-sketch to configure the sketch.

https://github.com/nickanderson/design-center/blob/feature/tmux_sessionManager/sketches/utilities/tmux/sessionManager/main.cf

Anyway this sketch manages the existence (or non existence) of a named tmux session. You can give it config options and a command. I use something similar to make sure that my irssi client is up on my server.

1. I'm not convinced about variable names or sketch names
2. I'm not sure about the meta entry for vars[tmux_config][type]" string => "SLIST";
   I dont think SLIST is a valid type. Needs to be a string that gets split inside the sketch?
3. its not namespaced yet.

I have one in the works, https://github.com/nickanderson/design-center/tree/feature/tcpwrapper/sketches/security/tcpwrappers
comments welcome.

Supports 3 editing modes, full_file (is this the best name, or perhaps defined_only?), entries_present, and entries_absent.
Debug mode for lots of reports.
Test file overrride (auto append .allow and .deny to testfile). Perhaps this should be named test_fileprefix to be more correct.

So far the cfsketches file is maintained by hand. We need to implement a mechanism to populate it automatically based on the existing sketches.

I think activations should not be indexed by filename, as this limits functionality and is confusing.

Confusing because the parameter values are copied into the activations file, so the original filename is no longer needed, yet displaying it might give the impression that it is.

Limits functionality because you cannot use the same base file and override certain parameters with --params to generate multiple activations. Consider this example:

cfma-10022:tmp root# cf-sketch -la
cfma-10022:tmp root# cf-sketch --activate VCS::vcs_mirror=/var/cfengine/inputs/sketches/VCS/vcs_mirror/params/cfengine-core.json
Loading activation params from /var/cfengine/inputs/sketches/VCS/vcs_mirror/params/cfengine-core.json
Activated: VCS::vcs_mirror aparams /var/cfengine/inputs/sketches/VCS/vcs_mirror/params/cfengine-core.json
cfma-10022:tmp root# cf-sketch -la
1   VCS::vcs_mirror /var/cfengine/inputs/sketches/VCS/vcs_mirror/params/cfengine-core.json {"activated":true,"runas":"getenv(\"USER\", 128)","origin":"https://github.com/cfengine/core.git","vcs":"/usr/bin/git","path":"/tmp/cfengine-core","bundle_home":"dirname(\"$(this.promise_filename)\")","branch":"master"}

So far so good. Now I want to activate the same sketch again, but change the checkout directory, so I do:

cfma-10022:tmp root# cf-sketch --activate VCS::vcs_mirror=/var/cfengine/inputs/sketches/VCS/vcs_mirror/params/cfengine-core.json --params path=/some/other/path
Loading activation params from /var/cfengine/inputs/sketches/VCS/vcs_mirror/params/cfengine-core.json
Activated: VCS::vcs_mirror aparams /var/cfengine/inputs/sketches/VCS/vcs_mirror/params/cfengine-core.json
cfma-10022:tmp root# cf-sketch -la
1   VCS::vcs_mirror /var/cfengine/inputs/sketches/VCS/vcs_mirror/params/cfengine-core.json {"activated":true,"runas":"getenv(\"USER\", 128)","origin":"https://github.com/cfengine/core.git","vcs":"/usr/bin/git","path":"/some/other/path","bundle_home":"dirname(\"$(this.promise_filename)\")","branch":"master"}

The original activation has been replaced! I think this is wrong.

I'd rather just assign an ID (like the numeric IDs shown in --list-activations), and maybe allow the user to specify a human-readable tag for that activation, for easier reference.

Thoughts?

Related to #105, I think sketches should have categorization through "provides" and "requires" attributes, now what we have bundle return values.

The attributes, in turn, should have formal meaning within the DC context.

For example:

sketch Apache::Install provides webserver
sketch Deploy::Docroot requires webserver
webserver = (vhost, site_name, docroot)

Which would result in Apache::Install returning the webserver attributes in the return array, while Deploy::Docroot would use them.

The key, therefore, is to develop a flexible language to express the provides/requires things. I propose we simply make it a JSON file living in DC and see how it goes.

Please comment.

In System::sysctl in the commands section, on line 76, there is no match for running sysctl -p, which causes it to be run every time. I believe the intent was to have it only run if sysctl_needs_reloaded class is set

https://github.com/cfengine/design-center/blob/master/sketches/system/sysctl/main.cf

diff main.cf main.cf2
77,81c77,80
< sysctl_needs_reloaded::
< "/sbin/sysctl"
< args => "-p",
< classes => if_repaired("sysctl_reloaded"),

< comment => "Reload sysctl after repairing configuration";

"/sbin/sysctl"
args => "-p",
classes => if_repaired("sysctl_reloaded"),
comment => "Reload sysctl after repairing configuration";

Noticed this, I suspect this has to do with uncommenting default values and adding values at the same time. I have not investigated though.

While it doens't cause any problem. I get complaints from anyone who looks at the configfile itself.

grep Listen /etc/ssh/sshd_config
ListenAddress 172.24.10.10
ListenAddress 172.24.10.10

grep Protocol /etc/ssh/sshd_config
Protocol 2
Protocol 2

This is what I have so far.
https://github.com/nickanderson/design-center/tree/feature/sysctl/sketches/system/sysctl

I'm having a hard time configuring the Wordpress sketch... so I looked at the README and this helped a bit... but I had to suspend cf-sketch to go pop up the hood and look under the hood... could you please make the README available from within the cf-sketch shell?

good stuff!
continue!