Hetzner Cloud integration about beaker-hcloud HOT 7 CLOSED

Hi @bastelfreak! I prototyped this a little bit and wonder how to deal with SSH keys:

Similarly to AWS, Hetzner requires you to set up SSH Keys (manual upload, or via API) before they can be used to provision servers. The AWS integration for beaker has a nice implementation that generates a unique name for an SSH Key, creates that in AWS in the provisioning phase and then removes the key again during cleanup.

We could do the same, but there is a small caveat: Hetzner enforces uniqueness of keys. You cannot upload the same key with a different name. If you only ever run one beaker job at a time and always use a key that you never want to use within the same project in your hetzner account, this is not a problem. But when concurrent CI jobs are possible and/or you want to run jobs manually using the same key, things could get ugly.

A slightly more robust approach could be to:

1. Search for existing keys at hetzner
   1. If a matching key is found, use that
   2. If no matching key is found, set up a new one
2. Provision with the chosen key
3. During cleanup remove the key if and only if it was created during step 1.

This way we would not clobber existing keys and could safely support different use cases using the same key. But still not concurrently ☹️
For concurrent jobs using the algorithm above would present all kinds of race conditions.

Of course we could simply skip the cleanup of keys, but I would find that a bit messy.

A cheap solution could be to simply skip management of SSH keys altogether. In that case a user would have to set up their key at hetzner in whichever way they prefer and then configure within beaker which key to use. This way we would be rid of all the problems mentioned above, but it would require more manual setup and be less "plug&play".

What do you think?

from beaker-hcloud.

Is generation of SSH keys not cheap enough that you can generate a key for each individual run? beaker-vagrant does generate a Vagrantfile for every setfile. Perhaps a similar approach could be taken to generate a key in the local directory?

from beaker-hcloud.

@oneiros I agree with @ekohl here. I think generating ed25519 keys on the fly isn't that expensive. It doesn't require a lot of randomness and also not a lot of CPU time. We could generate one for each job, upload them to hetzner and maybe prefix them with beaker_ or something similar, and purge them after the job finished/failed. The prefix allows us to easily identify and purge keys manually.

from beaker-hcloud.

I agree and it makes total sense to me.

Being new to beaker and its ecosystem, I was not sure, what the expections were. beaker itself compiles a set of ssh options using existing keys from the user. The aws "hypervisor" builds on that and I thought it might be least surprising to stay as close to this behavior as possible.

But given hetzner's constraints and the fact that a truly "plug&play" solution is probably even less error-prone, I will gladly go ahead with your suggestion.

from beaker-hcloud.

awesome, thanks!

from beaker-hcloud.

I just opened PR #11 with my first attempt at an implementation that "works for me" ™️

Generating ed25519 keys from ruby turned out to be a bit trickier than expected. While there is the ed25519 gem to generate the raw key material and there are a couple of gems that can handle importing different key formats used by OpenSSH, none was able to serialize the private key to OpenSSH's format.

In the end I settled on Github's ssh_data gem and monkey patched the missing methods in, because that gem already had all the necessary low level encoding methods to create the private key file. Additionally, this SO answer was invaluable in figuring out, how OpenSSH's format works: https://stackoverflow.com/a/73902911

A couple of remarks with regards to the requirements mentioned above:

support custom OS images

For now I opted to specify OS images by name. According to the docs this should work for both the ready-made images Hetzner provides as well as custom ones, i.e. VM snapshots stored in your account. I did however not test this.

cleanup stale VMs

I am not sure how and if this can be done. So instead I opted to add a paragraph about cleanup to the README similar to what the beaker-google gem does.

provide the desired availability zone

Sorry, I am not sure what this actually means 🙂.

I did provide an option to specify a datacenter location. Maybe that is it?

(Note that Hetzner has multiple datacenters at these locations and indeed the API allows to name a specific datacenter instead of the location, but I thought giving the location should suffice as that is what you select when creating servers manually as well.)

from beaker-hcloud.

Thanks for all the work! I will try to review/test/respond in detail in the next couple of days.

from beaker-hcloud.