ibotty / openshift-letsencrypt Goto Github PK

License: MIT License

Shell 95.98% Dockerfile 4.02%

openshift-letsencrypt's Introduction

PROJECT NO LONGER ACTIVELY MAINTAINED, use https://github.com/tnozicka/openshift-acme instead (comparison here)

Automatic Certificates for Openshift Routes

It will manage all routes with (by default) butter.sh/letsencrypt-managed=yes labels in the project/namespace, it's deployed in.

Limitations

For now, there are the following limitations.

It only supports domain names of length smaller than 64 characters.
It only implements http-01-type verification, better known as "Well-Known".
Multiple domains per certificate are not supported. See issue #1.
It will not create the letsencrypt account. It needs to be created before deploying. See Section Installation below.
It doesn't work cross-namespace. See issue #4.

Customizing

The following env variables can be used.

LETSENCRYPT_ROUTE_SELECTOR (optional, defaults to butter.sh/letsencrypt-managed=yes), to filter the routes to use;
LETSENCRYPT_RENEW_BEFORE_DAYS (optional, defaults to 30), renew this number of days before the certificate is about to expire;
LETSENCRYPT_CONTACT_EMAIL (required for account generation), the email that will be used by the ACME CA;
LETSENCRYPT_CA (optional, defaults to https://acme-v01.api.letsencrypt.org/directory);
LETSENCRYPT_KEYTYPE (optional, defaults to rsa), the key algorithm to use;
LETSENCRYPT_KEYSIZE (optional, defaults to 4096), the size in bit for the private keys (if applicable);

Troubleshooting

Route does not get admitted

Please test, whether DNS is set up correctly. In particular the hostname to get a certificate for has to point to the router (or the loadbalancer), also from within the cluster!

Implementation Details

Secrets

The ACME key is stored in letsencrypt-creds.

Containers

The pod consists of three containers, each doing exactly one thing. They share the filesystem /var/www/acme-challenge to store the challenges.

Watcher Container, watcher, watches routes and either generates a new certificate or set the already generated certificate.
Cron container, cron, periodically checks whether the certificates need to be regenerated. When Kubernetes cron jobs are implemented, this will move outside the pod.
Webserver Container, nginx, serves .well-known/acme-challenge when asking to sign the certificate. Uses ibotty/s2i-nginx on dockerhub.

Installing Openshift-Letsencrypt

Template

Create the template as usual.

> oc create -f template.yaml

Deploy openshift-letsencrypt

Instanciate the template.

> oc new-app --template=letsencrypt -p [email protected]

Service Account

The "letsencrypt" service account needs to be able to manage its secrets and manage routes.

> oc policy add-role-to-user edit -z letsencrypt

Let's encrypt credentials

Register an account key

You can skip that section, if you already use letsencrypt and already have an account key.

Get dehydrated and run the following commands.

> echo [email protected] > my_config
> /path/to/dehydrated -f config --register --accept-terms

This will generate a key in ./accounts/*/account_key.pem and info about it in ./accounts/*/registration_info.json.

Create the account key secret

Given an account-key, create a secret as follows.

> oc create secret generic letsencrypt-creds \
     --from-file=account-key=/path/to/account-key.pem \
     --from-file=registration-info=./accounts/*/registration_info.json

The registration info is not strictly necessary.

Notes

HPKP

It is necessary to pin at least one key to use for disaster recovery, outside the cluster!

Maybe pre-generate n keys and pin all of them. On key rollover, delete the previous key, use the oldest of the remaining keys to sign the certificate, generate a new key and pin the new keys. That way, the pin can stay valid for (n-1)* lifetime of a key. That is, if no key gets compromised!

openshift-letsencrypt's People

Contributors

Stargazers

Watchers

openshift-letsencrypt's Issues

Watcher dying every 2 minutes

My watcher container always dies and respawns after running for 2 minutes.

The problem seems to be with curl making api calls to the routes resource. It receives data but then after 60 seconds it times out thinking there is outstanding read data. The second (debugging?) curl call does the same thing, giving a total of 2 minutes.

Here's an example trace with curl running verbose:

*   Trying 172.30.0.1...
* Connected to openshift.default (172.30.0.1) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /run/secrets/kubernetes.io/serviceaccount/ca.crt
  CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=10.0.0.10
* 	start date: May 15 11:43:00 2017 GMT
* 	expire date: May 15 11:43:01 2019 GMT
* 	common name: 10.0.0.10
* 	issuer: CN=openshift-signer@1494848580
> GET /oapi/v1/namespaces/foo/routes/?watch&labelSelector=butter.sh/letsencrypt-managed=yes HTTP/1.1
> User-Agent: curl/7.29.0
> Host: openshift.default
> Accept: */*
> Authorization: Bearer 12345678...
> 
< HTTP/1.1 200 OK
< Cache-Control: no-store
< Content-Type: application/json
< Date: Thu, 18 May 2017 12:13:18 GMT
< Transfer-Encoding: chunked
< 
{ [data not shown]
Processing route /oapi/v1/namespaces/foo/routes/jenkins with domain jenkins.example.com.
Certificate for jenkins.example.com still valid long enough.
Processing route /oapi/v1/namespaces/foo/routes/api with domain api.example.com.
Certificate for api.example.com still valid long enough.
Processing route /oapi/v1/namespaces/foo/routes/www with domain www.example.com.
Certificate for www.example.com still valid long enough.

100 25339    0 25339    0     0  21388      0 --:--:--  0:00:01 --:--:-- 21401
100 25339    0 25339    0     0  11592      0 --:--:--  0:00:02 --:--:-- 11596
100 25339    0 25339    0     0   7950      0 --:--:--  0:00:03 --:--:--  7953
100 25339    0 25339    0     0   6051      0 --:--:--  0:00:04 --:--:--  6051
100 25339    0 25339    0     0   4883      0 --:--:--  0:00:05 --:--:--  4884
100 25339    0 25339    0     0   4094      0 --:--:--  0:00:06 --:--:--     0
100 25339    0 25339    0     0   3524      0 --:--:--  0:00:07 --:--:--     0
100 25339    0 25339    0     0   3093      0 --:--:--  0:00:08 --:--:--     0
100 25339    0 25339    0     0   2756      0 --:--:--  0:00:09 --:--:--     0
100 25339    0 25339    0     0   2485      0 --:--:--  0:00:10 --:--:--     0
100 25339    0 25339    0     0   2263      0 --:--:--  0:00:11 --:--:--     0
100 25339    0 25339    0     0   2077      0 --:--:--  0:00:12 --:--:--     0
100 25339    0 25339    0     0   1920      0 --:--:--  0:00:13 --:--:--     0
100 25339    0 25339    0     0   1784      0 --:--:--  0:00:14 --:--:--     0
100 25339    0 25339    0     0   1667      0 --:--:--  0:00:15 --:--:--     0
100 25339    0 25339    0     0   1564      0 --:--:--  0:00:16 --:--:--     0
100 25339    0 25339    0     0   1473      0 --:--:--  0:00:17 --:--:--     0
100 25339    0 25339    0     0   1392      0 --:--:--  0:00:18 --:--:--     0
100 25339    0 25339    0     0   1319      0 --:--:--  0:00:19 --:--:--     0
100 25339    0 25339    0     0   1254      0 --:--:--  0:00:20 --:--:--     0
100 25339    0 25339    0     0   1194      0 --:--:--  0:00:21 --:--:--     0
100 25339    0 25339    0     0   1141      0 --:--:--  0:00:22 --:--:--     0
100 25339    0 25339    0     0   1091      0 --:--:--  0:00:23 --:--:--     0
100 25339    0 25339    0     0   1046      0 --:--:--  0:00:24 --:--:--     0
100 25339    0 25339    0     0   1005      0 --:--:--  0:00:25 --:--:--     0
100 25339    0 25339    0     0    966      0 --:--:--  0:00:26 --:--:--
     0
100 25339    0 25339    0     0    931      0 --:--:--  0:00:27 --:--:--     0
100 25339    0 25339    0     0    898      0 --:--:--  0:00:28 --:--:--     0
100 25339    0 25339    0     0    867      0 --:--:--  0:00:29 --:--:--     0
100 25339    0 25339    0     0    838      0 --:--:--  0:00:30 --:--:--     0
100 25339    0 25339    0     0    811      0 --:--:--  0:00:31 --:--:--     0
100 25339    0 25339    0     0    786      0 --:--:--  0:00:32 --:--:--     0
100 25339    0 25339    0     0    762      0 --:--:--  0:00:33 --:--:--     0
100 25339    0 25339    0     0    740      0 --:--:--  0:00:34 --:--:--     0
100 25339    0 25339    0     0    719      0 --:--:--  0:00:35 --:--:--     0
100 25339    0 25339    0     0    699      0 --:--:--  0:00:36 --:--:--     0
100 25339    0 25339    0     0    680      0 --:--:--  0:00:37 --:--:--     0
100 25339    0 25339    0     0    662      0 --:--:--  0:00:38 --:--:--     0
100 25339    0 25339    0     0    645      0 --:--:--  0:00:39 --:--:--     0
100 25339    0 25339    0     0    629      0 --:--:--  0:00:40 --:--:--     0
100 25339    0 25339    0     0    614      0 --:--:--  0:00:41 --:--:--     0
100 25339    0 25339    0     0    600      0 --:--:--  0:00:42 --:--:--     0
100 25339    0 25339    0     0    586      0 --:--:--  0:00:43 --:--:--     0
100 25339    0 25339    0     0    572      0 --:--:--  0:00:44 --:--:--     0
100 25339    0 25339    0     0    560      0 --:--:--  0:00:45 --:--:--     0
100 25339    0 25339    0     0    548      0 --:--:--  0:00:46 --:--:--     0
100 25339    0 25339    0     0    536      0 --:--:--  0:00:47 --:--:--     0
100 25339    0 25339    0     0    525      0 --:--:--  0:00:48 --:--:--     0
100 25339    0 25339    0     0    514      0 --:--:--  0:00:49 --:--:--     0
100 25339    0 25339    0     0    504      0 --:--:--  0:00:50 --:--:--     0
100 25339    0 25339    0     0    494      0 --:--:--  0:00:51 --:--:--     0
100 25339    0 25339    0     0    485      0 --:--:--  0:00:52 --
:--:--     0
100 25339    0 25339    0     0    475      0 --:--:--  0:00:53 --:--:--     0
100 25339    0 25339    0     0    467      0 --:--:--  0:00:54 --:--:--     0
100 25339    0 25339    0     0    458      0 --:--:--  0:00:55 --:--:--     0
100 25339    0 25339    0     0    450      0 --:--:--  0:00:56 --:--:--     0
100 25339    0 25339    0     0    442      0 --:--:--  0:00:57 --:--:--     0
100 25339    0 25339    0     0    435      0 --:--:--  0:00:58 --:--:--     0
100 25339    0 25339    0     0    427      0 --:--:--  0:00:59 --:--:--     0* transfer closed with outstanding read data remaining

100 25339    0 25339    0     0    421      0 --:--:--  0:01:00 --:--:--     0
* Closing connection 0
curl: (18) NSS: client certificate not found (nickname not specified)

Scheduled Jobs

Fantastic work. Just what I need. Thanks for that.

One idea. Instead of continuously running the containers could this also be done with ScheduledJob?

Unable to create new certificates

This is a fresh deployment, fresh cluster, followed all directions exactly and i get a result similar to #21.

I don't even know what the solution was to #21. Afaik the roles are working correctly. I'm not rate-limited either. It just hasn't been able to request any certificates atall.

Here is log output of the watcher container:

watching routes with selector butter.sh/letsencrypt-managed=yes
Processing route /oapi/v1/namespaces/default/routes/openshift-cloud-console with domain cloud.crl.coloradomesa.edu.
unable to load certificate
139774264334240:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
Getting new certificate for cloud.crl.coloradomesa.edu
Adding well-known route.
Route letsencrypt-cloud.crl.coloradomesa.edu not admitted.

this repeats continuously until i destroy the deployment. I've recreated the deployment numerous times to no avail.

I also created an account-key with dehydrated and saved it as a secret, which is loaded into the pod.

implement multiple domains per certificate

Maybe use a label butter.sh/letsencrypt-cert-id which implements the grouping, or group by namespace only.

An open question is still, whether requesting a new certificate that contains a domain whose original certificate had been signed just recently is a problem. This old certificate has to be properly revoked afterwards as well.

routes with same dns name and different paths

i have routes with same domain but different paths
example:
i have a route with sucefful installed certificate by openshift-letsencrypt
route name: "cdn" with url cdn.mydomain.com route to service nginx-cdn

then i am add several routes
route name: "xxx" with url cdn.mydomain.com/xxx route to service nginx-xxx
route name: "yyy" with url cdn.mydomain.com/yyy route to service nginx-yyy

But watcher's log shows:

Processing route /oapi/v1/namespaces/mynamespace/routes/xxx with domain cdn.mydomain.com.
Certificate for cdn.mydomain.com still valid long enough.
Processing route /oapi/v1/namespaces/mynamespace/routes/yyy with domain cdn.mydomain.com.
Certificate for cdn.mydomain.com still valid long enough.

And certs not be added to these routes

rename due to not using the official client

Add support for recovery key

Depends on dehydrated-io/dehydrated#119

Implement http public key pinning

See https://calomel.org/http_public_key_pinning_hpkp.html.

This needs openshift/origin#9728.

work across namespaces

There can't be routes with the same hostname in different namespaces. That leaves two options for serving the well-known routes (webserver).

Run the letsencrypt management pod with integrated webserver in the same namespace as the routes to use (as implemented now), or
distribute the acme-challenges somehow, so that an external webserver pod can access them.

Integrated Webserver

It has the following big downside: It can't work with multiple namespaces. You'll have to deploy one instance, one service account, one acme credential and set the service account's permissions for every namespace.

It does only use one pod with three containers per namespace though and is conceptional easy.

Separated Webserver

I can think of the following options.

local secrets and respawning one webserver container per namespace (when attached secrets get mounted automatically, respawning might not be necessary. There is a bug about that in kubernetes somewhere). The management pod will then have to attach the secret volume to the webserver pod, wait for the namespace's pod to be available, and remove the volume again. The problem is that there will be one container per namespace pointlessly running constantly. Maybe that can be mitigated with a timeout.
local secret and spawning one webserver per acme-challenge. This will result in many one-shot pods. Maybe if letsencrypt's cron does use one per namespace, that is feasible.

Both will slow down certificate deployments considerable (scheduling, pulling, starting the webserver pod).

I tend to favor a separated webserver and on-demand spawning. Opinions welcome!

include a deployer script

Help in setting up the acme account secret, or when making backwards-incompatible changes, run migration scripts.

Couldn't create certificate PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE

Hi,
We created account_key via dehydrated:

./dehydrated --register --accept-terms

That created our account_key RSA file (4096 bits)
The we added this key as secret.

Afterward, we've launched your application and we can see in logs:

watching routes with selector butter.sh/letsencrypt-managed=yes
Processing route /oapi/v1/namespaces/XXX/XXX/XXX with domain api.XXXX.fr.
unable to load certificate
139686420318112:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
Getting new certificate for api.XXXXX.fr
Deleting well-known route.
Adding well-known route.

We checked rsa pem key and we have good headers:

-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----

We've checked in Pod if the RSA key is well monted, and it's ok.

At this time, we're not able to find why it won't work, and how to fix it.

Any help will be appreciated.

Thanks

are you aware of openshift-acme

Are you aware of https://github.com/tnozicka/openshift-acme.
If so, is there a reason to not use that repo?
If not, could you possibly join efforts?

Handling secrets

I'm opening this issue to see how you feel about dealing with secrets as well as routes.
An example use case that I have is postfix and dovecot deployments. There are no routes involved.

Certificate Not Connected?

I initially had a similar issue to #21. I was able to get past that by changing the insecureEdgeTerminationPolicy from Redirect to Allow.

Now I get the following:
`watching routes with selector butter.sh/letsencrypt-managed=yes

| Processing route /oapi/v1/namespaces/XXX/routes/XXX.com with domain XXX.com.
| unable to load certificate
| 140546834421664:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
| Getting new certificate for XXX.com
| Adding well-known route.
| calling dehydrated with domain name 'XXX.com'
| # INFO: Using main config file /usr/share/letsencrypt-container/config
| Using private key /etc/openshift-letsencrypt/account-key instead of account key
| Processing XXX.com
| + Signing domains...
| + Generating private key...
| + Generating signing request...
| + Requesting challenge for XXX.com...
| + Already validated!
| + Requesting certificate...
| + Checking certificate...
| + Done!
| + Creating fullchain.pem...
| Defer deploying certificate for routes.
| + Done!
| Running exit_hook
`
However, when I access https://XXX.com I still receive either an insecure website warning or a mismatched certificate notification depending on which browser I am using. What am I missing?

add monitoring and /healthz

Might use prometheus node-exporter's textfile exporter.

having a few issues

Hi,

I've been playing around with this and it's really very impressive, but I've noticed a few problems.

With 0.3.0 of dehydrated, the watcher pod was hanging while running dehydrated to perform the challenge response. I've forked the repo (https://github.com/jameseck/openshift-letsencrypt) and updated the dehydrated version in the Dockerfile to 0.4.0 and this has resolved the problem. This did lead to a trivial error where dehydrated is trying to call exit_hook() in the dehydrated callback, so I added this as a simple empty function for now. Let me know if you would like a PR for these changes.

Also, when the deployment is first rolled out, the cron container crashes a couple of times complaining about being unable to get a lock. Not really a big issue, but it looks a little messy when you are monitoring the pod status.

Dehydrate and certs not found

Hi!
First off, fantastic work!

I tried running the pod on openshift origin 1.3 and directly ran into a problem.
The dehydrate script hadn't been downloaded when creating the docker image but after changing the commit hash to the one of the latest commit, building the image and pushing it to our own repo on dockerhub everything seemed fine.

But then common.sh couldn't find the certs in

keyfile() {
    echo "$LETSENCRYPT_DATADIR/$1/key"
}

certfile() {
    echo "$LETSENCRYPT_DATADIR/$1/crt"
}

fullchainfile() {
    echo "$LETSENCRYPT_DATADIR/$1/fullchain"
}

Had to change to:


keyfile() {
    echo "$LETSENCRYPT_DATADIR/$1/privkey.pem"
}

certfile() {
    echo "$LETSENCRYPT_DATADIR/$1/cert.pem"
}

fullchainfile() {
    echo "$LETSENCRYPT_DATADIR/$1/fullchain.pem"
}

Do you think the change of dehydrated commit hash was the cause of pem files not found? Are they supposed to be copied to e.g. $LETSENCRYPT_DATADIR/$1/crt after retrieving them with dehydrate?

I have created new routes after the change and everything seems to work just fine.

Upper limit for domain names of 63 characters

Spent a few hours scratching my head on this one so I thought it might be worth mentioning in case anyone else hits it. Domain names of more than 64 chars cause both watcher and cron to fail in two places.

Route generated for validation puts the domain name into a label, these have a limit of 63 chars and so the route creation fails.
Library used by dehydrated puts the domain name being registered in the CN of the x509 cert, which has an upper limit of 64 chars, so even if you work around the first failure it fails here too.

Certbot issue here: certbot/certbot#1915 (I'm assuming it's the same issue with dehydrated).

Create only one certificate per domain for multiple routes with same domain

According to the log (and my understanding of the code), it seems that a new certificate is requested for each route, even if the host name is the same. This is a problem due to the rate limit of letsencrypt.

In our case, multiple routes are used to map different services to sub paths.

jq and nss_wrapper not found

When running from the latest dockerhub image the logs show me that jq is not found.

I get this when building myself:

Step 4 : RUN curl -sSL https://github.com/lukas2511/dehydrated/raw/$LETSENCRYPT_SH_REF/dehydrated          -o /usr/bin/dehydrated  && chmod +x /usr/bin/dehydrated  && yum install -y openssl curl nss_wrapper jq  && yum clean all
 ---> Running in bac2f3c36df2
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: mirror.nsc.liu.se
 * extras: mirror.nsc.liu.se
 * updates: mirror.nsc.liu.se
Package curl-7.29.0-25.el7.centos.x86_64 already installed and latest version
No package nss_wrapper available.
No package jq available.
Resolving Dependencies
--> Running transaction check
---> Package openssl.x86_64 1:1.0.1e-51.el7_2.7 will be installed
--> Finished Dependency Resolution

patch_route() doesn't work - Route won't be updated.

When I looked at the watcherpod's log, I noticed, that all steps work fine. Even the generation of the data object to patch the route with is correct. I took this data object and passed it to openshift via oc patch route <route-name> -p <data>. This worked and the route was changed to https.

By turning on debugging in common.sh via set -x and removing 2> /dev/null in line 62 i got the message curl: (22) NSS: client certificate not found (nickname not specified).

After this, cleanup proceeds and the route didn't change.

Any ideas, why this is failing?