zombiezen / mcm Goto Github PK

shellify currently does not pass tests on Mac. One failure is the file mode, the other seems to be related to the command status codes for conditions.

Bazel has a script that already does this: https://github.com/bazelbuild/bazel/blob/master/scripts/ci/ci.sh

This would just need to be grafted onto the Travis configuration to use TRAVIS_COMMIT_RANGE, as described in Travis Environment Variables.

Currently, the second argument to mcm.resource can refer to nonexistent resources or cause cycles. This should be validated during Lua::finish.

(Definitely a check that should be included in #12.)

While catalog code should be trusted, a desirable feature is for luacat evaluation to be mostly side-effect free. It would be good to scrub through the standard library and either remove or sanitize functions. I've already removed the io package, but I will update this issue with the list of changes after some research.

In installing from custom sources, it's quite necessary to be able to explicitly trust a GPG key. I forgot why I didn't do this to begin with, but it should act similarly to how Puppet handles this case.

Neither shellify or exec obey the Mode struct for files. They ignore them, not even returning an error if set.

If one of these field types is encountered (not for any of the current schema fields), then luacat errors out.

Similar to environment variables, it would be good to have a mechanism so that something like Facter could be used to allow luacat scripts to be semi-portable.

Once I've converted more of my configurations over to mcm, I want to write up better sample code for luacat, perhaps with some reusable libraries.

This probably won't be an issue until I work on #21, but there's no way to increase the Cap'n Proto reader limit at runtime. dot, exec, and shellify should all grow a flag that allows the user to control this.

As a follow-on to #17, it would be nice to have a -I flag that adds paths to package.path. There probably should also be an environment variable, but it should differ from the normal Lua environment variable, since luacat isn't fully compatible with stock Lua.

It would be nice to do this as part of #1, so that Travis always builds binaries.

mcm-exec has been designed with this in mind, but just for simplicity of testing, I didn't add it. It should have a -j flag to control the number of resources that are applied at one time.

The use case for this is files like /etc/hosts or /etc/ssh/ssh_known_hosts: files where just creating or removing a file in a foo.d directory isn't sufficient. The set of requirements are:

• Must be able to add lines to a file that don't have a keyed identifier.
• Must be able to idempotently remove lines from a file that contains a keyed identifier.
• The keyed identifier may be at the beginning or end of a line. At the moment, the particular use cases have non-whitespace characters as identifiers, but the rest of the desired line may contain whitespace.

If at all possible, I would like to not add new catalog schema to support this feature. I'm hoping that the combination of luacat plus being able to execute arbitrary bash is powerful enough to support this feature. It may also be interesting to try to develop a means for testing luacat scripts as part of feature development on this, but not a must-have.

Set up a Travis CI build that runs bazel test //....

Not all file contents need to be base64 encoded, and since this bloats file size, it would be good to avoid it where possible. Also, I don't have a good scheme for picking EOF markers.

The current Lua package.path is still the default, which does not include the directory that the main script is in. So for instance:

mkdir foo
cat > foo/main.lua <<'EOF'
local lib = require("lib")
-- ...
EOF
cat > foo/lib.lua <<'EOF'
local lib = {}
-- ...
return lib
EOF

mcm-luacat foo/main.lua > foo.out
# fails with can't find "lib"

The depgraph package doesn't do any cycle detection, so I believe that it will error out midway instead of early.

The print built-in function sends to stdout right now, which isn't ideal for pipelines, since it corrupts the catalog output. luacat should override this and send to stderr.

For large on-disk catalog files, we could get some performance gains by using a lazy loading arena. However, this would require a bit of cooperation from catalog producers, since the best gains would come if large blobs (like the file contents) were placed in separate segments.

I'm unsure of exactly the right design for this, but I think packages ought to be promoted to resource types. While the functionality can be gained via exec, there are scheduling benefits to batching package installs (since packages can be downloaded and installed concurrently instead of serially), and the exec commands are usually hard to get right.

I have the most knowledge of Debian packages, so I'll probably add that first. Based on experience from Puppet, each package manager should be treated as a separate resource type, instead of automatically detecting the package manager. This simplifies implementation as it avoids providing a false abstraction: each package manager calls the same package different things.

It would be nice to do this as part of #1, so that Travis always builds binaries.

I'd like to create a tool that takes in a catalog and outputs the same catalog, but has a series of optional transforms and validations. Examples:

• Sort resources and dependency lists (like in #7)
• Detect cycles (tracking in #5)
• Prune no-op resources by adding their direct dependencies as direct dependencies of their dependents
• Check file paths for absoluteness (and other preconditions specified in the schema docs)
• Move large blobs to separate segments (potentially enables optimizations in #11)

Right now, mcm-exec only handles Exec.Command with the argv field, not bash. It's fairly high priority for this case to be handled since so many catalogs will depend on it.

Files specified in the catalog without content are supposed to succeed if a regular file exists at the path, and can optionally set the mode, and fails otherwise.

Use Bazel's stamp functionality to provide version info for built binaries. Makes it easier for end-users to provide bug reports. Blocks #2 and #3, and blocked by bazelbuild/rules_go#233.

Since the catalog is sent to stdout, console users not redirecting the output will get jumbled terminals as binary data messes everything up. It would be good to fail early in this case.

It's a bit cumbersome to have multiple languages in play in one repository, as it increases load on contributors. Since Lua is the only hard dependency that needs C, then this could be a viable candidate for using cgo. I haven't tried using Bazel's cgo support yet, so doing this may require a bit of experimentation.

The order of resources and dependencies should be sorted so that reorderings in the Lua code don't affect the outputed catalog.

At the moment, a catalog can specify Exec.condition.ifDepsChanged with IDs that aren't in the direct deps. Since avoiding the validation can result in undefined behavior in scripts, it would be good to add this validation.

In some circumstances, it is difficult to specify stdout redirection, so it would be nice to support a -o option to have the redirect happen as part of the command line.

I want some mechanism to be able to pass files around into mcm.resource. The idea is that if you just want to include a binary file, it can be much more efficient to have the C++ side of it read it (or mmap it) into a Cap'n Proto segment.

However, luacat programs should be able to read the data into memory if need be (to allow templating etc.). So perhaps a fileref:read() call, along with some flag in luacat to provide a limit on how large of a file that can be read in in this manner.