External TLS cert/key not auto-updating for Hosted CE about slate-catalog HOT 15 OPEN

Sounds good to me

from slate-catalog.

@brianhlin thoughts on how you want to proceed here? I see two paths..

1. We mount the secret somewhere, and then symlink to hostcert.pem and hostkey.pem. We would need to add another little script at startup to patch that together. I presume this would go into the image config scripts that we get from software base.
2. We mount the secret somewhere, and then change the HostedCE configuration appropriately. Perhaps a drop-in configmap for the CE configuration?

from slate-catalog.

I think I like option 2 and we can just set the following with updated paths:

GSI_DAEMON_CERT = /etc/grid-security/hostcert.pem
GSI_DAEMON_KEY = /etc/grid-security/hostkey.pem

If we do 2, though, are we going to be able to set the permissions on the cert/key properly?

from slate-catalog.

@brianhlin Ugh, yeah, I forgot about that whole problem. Thinking..

from slate-catalog.

OK, will experiment and return with a PR!

from slate-catalog.

So I did a bit of looking and thinking on this one-

We currently do the following:

      - name: osg-hosted-ce-hostcertkey-volume
        secret:
          secretName: {{ .Values.HostCredentials.HostCertKeySecret }}
          items:
          - key: tls.crt
            path: hostcert.pem
            mode: 256
          - key: tls.key
            path: hostkey.pem
            mode: 256

As far as permissions are concerned, all of this is above board as far as I can tell.

If we change the mount from

        {{ if .Values.HostCredentials.HostCertKeySecret }}
        - name: osg-hosted-ce-hostcertkey-volume
          mountPath: /etc/grid-security/hostcert.pem
          subPath: hostcert.pem
        - name: osg-hosted-ce-hostcertkey-volume
          mountPath: /etc/grid-security/hostkey.pem
          subPath: hostkey.pem

to

        {{ if .Values.HostCredentials.HostCertKeySecret }}
        - name: osg-hosted-ce-hostcertkey-volume
        mountPath: /media

(or whatever, /media is convenient, exists, and is usually empty)

You can poke the filesystem and see files are properly created with the right mode (256 dec = 0400 oct)

I made a nasty kubectl exec loop to just try to recursively look at /media while I waited between crashloops on my laziest/worst deployment of theHostedCE:

drwxrwxrwt 3 root root  120 Apr 30 20:38 .
drwxr-xr-x 1 root root 4096 Apr 30 20:42 ..
drwxr-xr-x 2 root root   80 Apr 30 20:38 ..2021_04_30_20_38_13.342530518
lrwxrwxrwx 1 root root   31 Apr 30 20:38 ..data -> ..2021_04_30_20_38_13.342530518
lrwxrwxrwx 1 root root   19 Apr 30 20:38 hostcert.pem -> ..data/hostcert.pem
lrwxrwxrwx 1 root root   18 Apr 30 20:38 hostkey.pem -> ..data/hostkey.pem

/media/..2021_04_30_20_38_13.342530518:
total 4
drwxr-xr-x 2 root root   80 Apr 30 20:38 .
drwxrwxrwt 3 root root  120 Apr 30 20:38 ..
-r-------- 1 root root    0 Apr 30 20:38 hostcert.pem
-r-------- 1 root root 1675 Apr 30 20:38 hostkey.pem

That looks like we ought to just be able to symlink to the right locations and things will be Fine (tm) ?

from slate-catalog.

(or whatever, /media is convenient, exists, and is usually empty)

We should be able to pick whatever, k8s should make the dir for us, IIRC. I think @matyasselmeci has used /etc/grid-security/hostcert.orig.d/ or something to that effect.

-r-------- 1 root root 0 Apr 30 20:38 hostcert.pem

We may need this to be world-readable. I forget exactly when the SchedD drops privs and if that's after it reads in the host cert.

Otherwise I think This Is The Way so we'll have to make some minor adjustments to the container to create the symlinks if the certs live in whatever dir we choose.

from slate-catalog.

We should be able to pick whatever, k8s should make the dir for us, IIRC. I think @matyasselmeci has used /etc/grid-security/hostcert.orig.d/ or something to that effect.

Works for me. Let's do that. We'll play along with whatever you guys have precedent for.

-r-------- 1 root root 0 Apr 30 20:38 hostcert.pem

We may need this to be world-readable. I forget exactly when the SchedD drops privs and if that's after it reads in the host cert.

OK, so mode 432 instead of 256. (thanks Kubernetes for making this so intuitive and easy 💩 )

Otherwise I think This Is The Way so we'll have to make some minor adjustments to the container to create the symlinks if the certs live in whatever dir we choose.

Sounds good.

from slate-catalog.

This is what I use in the condor containers:

if [[ -f /etc/grid-security-orig.d/hostcert.pem && -f /etc/grid-security-orig.d/hostkey.pem ]]; then
    install -o root -g root -m 0644 /etc/grid-security-orig.d/hostcert.pem /etc/grid-security/hostcert.pem
    install -o root -g root -m 0600 /etc/grid-security-orig.d/hostkey.pem  /etc/grid-security/hostkey.pem
fi

from slate-catalog.

@matyasselmeci - that probably should be executed every few hours, right?

from slate-catalog.

Yep. I don't have cron in the original htcondor images but in OSG images I put a call to it in /etc/cron.hourly. (It's part of the update-secrets script which also updates passwords and tokens.)

from slate-catalog.

Hi all! What's the next steps here? Not clear who's holding the ball on this.

from slate-catalog.

Container work here: https://opensciencegrid.atlassian.net/browse/SOFTWARE-4623. I believe @LincolnBryant is going to tackle the Helm Chart work.

from slate-catalog.

Indeed, I am trying to focus on this today/tomorrow and get this patch landed. Apologies for the slowness.

from slate-catalog.

PR #496 should be a very simple patch that should address this specific issue, if the container is updated appropriately.

from slate-catalog.