picostack / pico Goto Github PK

Aside from more unit tests, given the complexity of the system it would be beneficial to set up a full end-to-end test suite that spins up a Vault server and ensures lifecycle operations such as token renewal work correctly.

It would also be good to copy some of the ideas from gitwatch tests - create some mock repositories, commit to them and assert the service has performed the actions specified by their configurations.

Not sure why. Documentation isn't great for the Vault SDK.

In order to work around the problem, it's easiest to just delete the cached git directory.

In the future it would be nice to have a web ui, that can is present on a master server, it would be interesting to have a master to slave relationship in the future that isn't a requirement, but if you are running multiple services across multiple servers, it might be nice to have a single interface for monitoring them.

The site should really provide statistics, logs, and health checks of services. There should be a way to set off manual events such as stopping/restarting/starting repo's at will. A good example of this is, if a bad commit is pushed to a repo, and it gets deployed, it could leave that system vulnerable to an attack, and it would make sense to stop the repo that is affected. So having a new UI to be able to do that would make sense.

Though not a requirement, it might be a nice idea to provide a way to access a containers cli from the ui. Security would need to be in-check for this sort of stuff, to prevent insecurely leaking the access of the container to the outside world. I could think that vault integration could be used for it's token regeneration to provide security here too.

There should also be an API, that can be exposed to grab data about the services being ran, the statuses, health status, logs, etc.

Sometimes I deploy without remembering to set some env vars so it would be useful for wadsworth to skip deployment completely if there are missing env vars.

E({
        name: "some-app",
        url: "...",
        vars: ["SUPER_SECRET_REQUIRED_PASSWORD"],
        up: ["docker-compose", "up", "-d"],
        down: ["docker-compose", "down"]
})

The difference isn't obvious at all. I'm also not sure what APIs need to be used for v2.

There is some code in the vault CLI source: https://github.com/hashicorp/vault/blob/master/command/kv_helpers.go#L99

But none of it is exposed for importing for some reason. So this might require copying/implementing a v2 check.

It seems to manipulate the mount path that's being read: https://github.com/hashicorp/vault/blob/master/command/kv_get.go#L94-L110

Currently, the choice of secrets is Vault or nothing. I'd like to 1. make Vault integration a bit more modular (write a secrets provider interface that it satisfies) then introduce more secrets providers.

One of the reasons for this is, wadsworth is designed for simple deployments that don't need Kubernetes. Deploying and setting up Vault is quite a hassle to get decent security, and is generally advised to run on a separate server for both security and reliability reasons. I'd like to provide a simple middle-ground that fulfils the single-server use-case that Wadsworth was built to satisfy. This may be as simple as an in-memory store or some simple encrypted store.

Needs some thought. Currently, environment variables are propagated to every config engine VM so this can be a valid replacement but it can lead to users simply shoving their secrets into a .env file, which may not be the best security approach.

The Pico Docker image is currently root (the insane default of Docker)

This means that making quick ad-hoc changes to targets or config is quite annoying. You have to chown the targets directory, do your quick changes/tests and chown it back to root (or let Pico overwrite with root).

Instead, there should be a better solution to this, allowing users to deploy new setups easily where they can deploy manually a few times to work out any first-time-deploy issues then commit those changes and leave Pico to handle the rest.

The ideal workflow would be:

• Add a target to Pico, with a no-op up command
• Pico clones this to its targets directory
• You jump in and do your own docker-compose up and figure out any last-minute deployment details you need
• Once you're happy with the setup and the target is running, change the up command to the live version
• Pico takes over
• At any point you can jump in and make ad-hoc changes to the live target then commit your changes

For some reason when Pico see's changes in a repo, it somehow manages to run for all repos of the same url even if they have different branches.

For example today, I pushed to my development branch, and CI passed and pushed a staging container, well what happened was that the production server went down and it seems pico was responsible for doing that.

ERROR: for sag-sffw-prod-gm  Cannot create container for service sffw: Conflict. The container name "/sag-sffw-dev-gm" is already in use by container "cff9bb1eb6527bd18a47ca1bf84a9d5bd01729e21cf6f5da520a80fc5c24e36b". You have to remove (or rename) that container to be able to reuse that name.,
ROR: for sffw  Cannot create container for service sffw: Conflict. The container name "/sag-sffw-dev-gm" is already in use by container "cff9bb1eb6527bd18a47ca1bf84a9d5bd01729e21cf6f5da520a80fc5c24e36b". You have to remove (or rename) that container to be able to reuse that name.,
{"level":"error","ts":"2020-08-04T14:22:01.131Z","caller":"executor/cmd.go:40","msg":"executor task unsuccessful","target":"sffw-gamemode-dev","shutdown":false,"error":"exit status 1","stacktrace":"github.com/picostack/pico/executor.(*CommandExecutor).Subscribe\n\t/home/runner/work/pico/pico/executor/cmd.go:40\ngithub.com/picostack/pico/service.(*App).Start.func1\n\t/home/runner/work/pico/pico/service/service.go:116"},
Encountered errors while bringing up the project.

The production repo should not have been touched at all, since no event changes happened to that branch.

Currently the watcher receives events and executes them. It would be good if these were two separate packages with interface contracts.

This would permit executing events from other sources, such as webhooks and the Web UI when #10 is done.

It would also make unit testing these individual parts a lot easier.

Once the Vault API has provided secrets, they should be stored in an mlock'd page to reduce the risk of access via swap files etc. https://github.com/awnumar/memguard

Right now you can only specify a single auth for running a repo, however this becomes a problem when you have the config file on one host and the repo on a different host.

Lets say you have the config file as a private repo on gitlab, but you want to access a private repo on github, well then you either have to have 2 configurations (which defeats the purpose of a single config file to manage it all).

A solution would be to have a config block which can take an auth type, or allow each repo to take multiple auth types, but then again it might not be easy to know which one is the one for the config, unless all where tried and only report the error if all fail.

I am experiencing an issue, where pico seems to be updating the wrong repo when using branches.

It seems pico is able to correctly perform an update on the first start, but then after it continues to update the wrong branch (it calls the up event) and the service is restarted.

I think a test needs to be performed which pulls from the same repo, and has 2 different branches being used on it. This should make it clear what the problem is.

Public repos should not require auth, but it seems it's still required for some reason.

Not strictly a pico specific thing but worthwhile for the overall Pico Stack.

1. Deploy
2. Follow steps to set up master key portions and root token
3. Create an administrator policy (needs an example)
4. Enable the userpass Auth Method
5. Add a user for yourself
6. Add an entity that maps to that user (using Aliases)
7. Either add the administrator policy to that user or:
   7.5. Create a group with the administrator policy and add self to it
8. Activate the kv secret engine using v2

The picostack/picostack repository should contain this tutorial, as well as a full setup tutorial for the entire stack.

Now that #27 is on the way, the concept of an Executor exists.

So far, the only executor is CommandExecutor and it just executes task.Target jobs. It's a bit misleading because the task.Target is structured to be a "command" anyway but... that's another issue.

It could make sense to introduce more executors that don't just run a command but may perform other tasks that are easier to declaratively set up via Pico configuration instead of creating a whole new git repo for it.

I'm thinking of webhooks as a first, maybe integrations like Slack or Discord.

More feature-creep, I know. But it could be useful.

A Docker-specific executor may be useful, as more granular information can be acquired via the Docker Go library.

Food for thought anyway...

As the title says, Pico is unable to change to the correct directory when a branch is used. It seems like there is no appending branch name to the deployment path, it looks for the name of the service without the appended branch name <service>_<branch>

Waiting for a quick fix ;)

Hi, I strongly suggest to add a custom subpath definition for both "config" and "target" repo.
I would like to organize *.js and *.yml files in subdirectories and not store them in the root folder.

For example let's think about a config repo that contains configuration for many vms.

• Add a license (MIT probably)
• Finish the GoReleaser config (and correct license)
• Finish the Snapcraft config
• Set up TravisCI

https://pkg.go.dev/github.com/hashicorp/vault/api?tab=doc#Renewer

https://docs.aws.amazon.com/sdk-for-go/api/service/secretsmanager/

It'd be good if pico was added to snap store, it's quite easy. You could also push pico to Arch User Repository, and Fedora COPR, however, snap seems to be the best option, as snapd can be installed on pretty much every GNU/Linux distribution and comes preinstalled on distros with large userbase, like Ubuntu.

# github.com/picostack/pico
go\src\github.com\picostack\pico\main.go:28:5: app.Author undefined (type *cli.App has no field or method Author)
go\src\github.com\picostack\pico\main.go:29:5: app.Email undefined (type *cli.App has no field or method Email)
go\src\github.com\picostack\pico\main.go:41:19: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag
 in slice literal:
        cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
go\src\github.com\picostack\pico\main.go:41:38: unknown field 'EnvVar' in struct literal of type cli.StringFlag
go\src\github.com\picostack\pico\main.go:42:19: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag
 in slice literal:
        cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
go\src\github.com\picostack\pico\main.go:42:39: unknown field 'EnvVar' in struct literal of type cli.StringFlag
go\src\github.com\picostack\pico\main.go:43:17: cannot use cli.BoolFlag literal (type cli.BoolFlag) as type cli.Flag in
slice literal:
        cli.BoolFlag does not implement cli.Flag (Apply method has pointer receiver)
go\src\github.com\picostack\pico\main.go:43:34: unknown field 'EnvVar' in struct literal of type cli.BoolFlag
go\src\github.com\picostack\pico\main.go:44:21: cannot use cli.DurationFlag literal (type cli.DurationFlag) as type cli.
Flag in slice literal:
        cli.DurationFlag does not implement cli.Flag (Apply method has pointer receiver)
go\src\github.com\picostack\pico\main.go:44:46: unknown field 'EnvVar' in struct literal of type cli.DurationFlag
go\src\github.com\picostack\pico\main.go:44:21: too many errors

builds fine so... no idea

Containers are failing to be started, because a port allocation already exists, so I feel the containers should be stopped before they are then started again.

Right now it is impossible for me to make continuous updates to a repo, because the docker container fails to be brought back up again.

As a future idea to make Wadsworth even more useful, would be to implement some notification providers like Gotify, Slack, Telegram, Webhooks, Discord, etc.

It would be useful to have Wadsworth to send notification when events happen, that way to track it's progression and also report any faults that happen, this is just another step to help the deployer quickly resolve issues.

This could probably tie into a future web ui, that could make it easier to setup providers.

To pair with #10

A CLI would be a no-brainer at this point.

Replace --no-ssh with --use-ssh or something.

Currently, these two watchers are intertwined into the same code.

It turns out, the state is only really touched inside reconfigure() and passed once to the state diffing function.

This can probably be delegated to a separate package to facilitate easier testing.

It also gave me a new idea: configuration providers (more abstractions, yay!)

Currently, to both use and test pico, you need at least two Git repositories. One for config and one target. I was thinking an abstraction would be nice for testing but it also might be useful for very simple use-cases too.

You might not want your configuration to be tracked in Git. You might just want a simple process that hits 1 or 2 Git repos to run commands and that's it. So, I think abstracting a configuration provider could be beneficial for both these uses.

This might be tricky, but I think using another event bus will make it simpler. I really need to move to using message-bus but that can come later.

This way, the watcher then just receives state on an event, restarts itself and continues.

Targets should be able to specify required secrets that, if not found, emit an error.

Currently, the behaviour is implicit, it will load whatever secrets it can find, which can make configuration mistakes hard to debug.

T({
  name: 'svc',
  repo: '...',
  secrets: ['PASSWORD'] // error if `PASSWORD` not present in Vault

Should make A return the name so you can assign it to a variable and use that in the auth: field. This would reduce the chance of typos when setting auth methods in targets.

https://github.com/Southclaws/wadsworth/blob/610b1fcdcd114e7920202f34aaa0917600008754/service/secrets.go#L17

Should be / and VAULT_PATH should control the entire thing.

{
    "level": "error",
    "ts": "2020-04-13T12:51:51.924Z",
    "caller": "watcher/git.go:88",
    "msg": "failed to handle event",
    "url": "https://github.com/picostack/test",
    "error": "attempt to handle event for unknown target https://github.com/picostack/test at /data/targets/test"
}

attempt to handle event for unknown target https://github.com/picostack/test at /data/targets/test
github.com/picostack/pico/watcher.(*GitWatcher).handle
    /home/runner/work/pico/pico/watcher/git.go:213
github.com/picostack/pico/watcher.(*GitWatcher).Start.func1
    /home/runner/work/pico/pico/watcher/git.go:87
github.com/picostack/pico/watcher.(*GitWatcher).Start
    /home/runner/work/pico/pico/watcher/git.go:101
github.com/picostack/pico/service.(*App).Start.func2
    /home/runner/work/pico/pico/service/service.go:121
runtime.goexit
    /opt/hostedtoolcache/go/1.14.1/x64/src/runtime/asm_amd64.s:1373

So this error attempt to handle event for unknown target https://github.com/picostack/test at /data/targets/test seems to indicate that once a command fails, it's lost from the targets list and future events will not match against it because it's missing.

By default Vault uses the engine version 2 for key/value pairs. Wadsworth seems to expect version 1, and doesn't correctly handle version 2.

This will cause credentials to not be correctly handled/passed to containers.

Since version 2 is the default for engines being created at this current time, I think it makes sense to have an option for vault where you can specify which version you are using and then handle that internally.

Though this may not need to exist, if there is a way in the vault api to detect which versions exist and to use that. Since it's not possible to have 2 versions of the same path, it makes more sense to detect it internally and to not require any more input from the end user.

Currently, there are two ways to securely use Git to pull config repos and target repos:

1. ssh
2. basic auth

SSH is really awkward to automate declaratively, so I generally avoid it. Basic auth works well as GitHub/Lab both provide machine tokens for automated access.

The issue is, you need to pass these tokens in as environment variables. It also means propagating them through to the target config which makes multiple tokens awkward.

So, pico should instead just load all credentials from Vault.

And drop SSH support, it's not worth maintaining for the aforementioned reason.

General architecture:

Now:

• Init
• Ping vault
• Clone config repo
• Get secrets for targets and up targets

Future:

• Init
• Ping vault
• Acquire bootstrapping credentials - user/pass for config repo
• Clone config repo
• Get secrets for target repo credentials, clone target repos, up targets

This also means that there needs to be some form of naming convention for the secrets in Vault so they map to repositories. Maybe based on regex?

This may be out of scope as it can be accomplished with a script.

But I hate scripts. Especially *sh scripts.

So it would be neat to run certain commands only if specific directories have been touched with a commit. This would be beneficial for monorepos.

Consider:

frontend/
backend/

When backend is changed, you want to run docker-compose up --build -d and when frontend is changed, you want to run npm build or something.

My use case is that a project has a backend hosted on a machine and a frontend hosted on Vercel. So when changes are made to the frontend, Pico doesn't need to run a build for the backend.

So you don't need to run picobot run https://user:[email protected]/ns/repo

This can also be used to solve #24

Essentially, any variables prefixed with PICO_ can be reserved for Pico use, such as PICO_GIT_USERNAME to solve #24

Then, any other variables can just be passed to every future task.

The secret can sit at VAULT_CONFIG_PATH which will default to pico. Along with the default base path, this would place the default config path at /secret/pico which seems logical.

Since write access is usually not required for most tasks, HTTPS URLs are quite enough. However, even though it all works fine via HTTPS, the git-client still requires a running SSH agent. While this is not a huge problem, it's quite annoying and it would be cool to to be able to use it without an SSH agent.

This code:

	w.targetsWatcher, err = gitwatch.New(
		context.TODO(),
		targetRepos,
		w.checkInterval,
		w.directory,
		nil,
		false)
	if err != nil {
		return errors.Wrap(err, "failed to watch targets")
	}

	go func() {
		e := w.targetsWatcher.Run()
		if e != nil && !errors.Is(e, context.Canceled) {
			w.errors <- e
		}
	}()
	zap.L().Debug("created targets watcher, awaiting setup")

	<-w.targetsWatcher.InitialDone

	zap.L().Debug("targets watcher initialised")

Is terrible. It was a quick solution but it really should be better.

If the targetsWatcher fails in any way, it pushes errors to an error channel. But, if it never sends the InitialDone signal, the select on w.errors never actually yields - deadlock.

This should instead be a select over the initial wait and the error channel that returns the error.

This is a wild one and not really something necessary for production but maybe worth some investigation.

Currently, Otto only supports ES5, so quite a few neat new JS features aren't available.

I don't really know how Otto fairs with transpiled code, but it could be interesting to try.

Of course, a user can transpile their config files themselves but then they have to set up the environment etc which can be a burden. Especially given they then need to decide where that happens - pre push, on CI, at build/run time?

So the idea is, wadsworth may check if npm is available at runtime then, if it is, set up a pre-configured npm package with a set of sane defaults (babel configured to transpile various popular ES6+ features to ES5, and bundle everything into a single JS file).

I'm not certain if it's even a good idea yet... probably not, but could be interesting to try nonetheless!

https://pkg.go.dev/github.com/picostack/pico?tab=doc

It doesn't need to be all-encompassing, just links out to the Pico project page etc.

Currently, the executor will just copy output from commands to standard out.

It may be useful for log aggregation purposes to read this buffer and place each line inside a Zap log and writing it out as JSON just like all the other logs.

The problem here is with terminal rewrites such as when you pull a docker image and docker attempts to provide a progress bar "UI". Which can sometimes be solved if the program provides a way to disable that. Docker and compose both provide --quiet.

Of course this should be optional too, not everyone runs a log aggregator.

My main use-case for Pico is to run docker-compose up -d for config changes. However, if a container dies for some reason, bringing it back is done in two ways:

• manually ssh into the machine, cd into its target directory and manually run compose (ew!) - this is really annoying/insecure when the container depends on a bunch of secrets from Vault - you have to copy those into the command line or .env file (and risk forgetting to remove it after)
• push an empty commit to the config repo to trigger pico - the slightly safer option, but clutters your history and is a bit of a hack

One of the goals of the web UI (#10) is to provide a way to do this with a click instead of an ssh.

One of the goals of using an event bus (#27) is to allow multiple trigger sources for command execution.

It would make sense to introduce another provider (I guess we're calling event sources "providers" now?) that's based on periodic automated events (cron-tab style) alongside the Git events. So a target will react to Git pushes, but also run its command every now and then anyway.

At first, this felt like feature creep. I didn't really want to move towards a kubeapplier style reconcillation system because that would be very Docker-specific (but maybe...). I gave it some thought and realised this could be a useful solution that applies outside of Docker use-cases (periodically run a container, or a webhook, or any process, all configured via Git instead of via crontab where it gets buried in machine-specific config)

I ran out of disk space (surprise) while a new version was building. The build thus failed but after freeing up space it didn't retry.

The C should stand for "Container(s)" ;)

As of right now, submodules are not pulled in when cloned, it would also be nice to have them pulled when pico detects changes in the repo.

Hopefully this makes it into the next release.

It would make sense to have a way to run a script or command on the very first run of pico, to allow for setting up all the required tools, packages, and configurations for the system it is being run on.

This could be used for installing docker, setting up firewall, permissions, users, etc. Since pico is intended to be used with a master repo, that is responsible for deploying multiple services, it makes sense that the setup script would also be part of that initial repo, and with the option to run it when first being initialized, it just adds to ease of setting up deployments.

To facilitate separating by team/service, config and targets should be cloned recursively by default.