Re-created kubernetes api cert, openunison shows "unauthorized" about openunison-k8s-login-activedirectory HOT 43 CLOSED

Ok, so i should be able to just redeploy the ingress to have it update the controller?

does this look right?

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: https
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.org/ssl-services: openunison-orchestra
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-name: INGRESSCOOKIE
    nginx.ingress.kubernetes.io/session-cookie-hash: sha1
  name: openunison
  namespace: openunison
spec:
  rules:
  - host: my.login.example.org
    http:
      paths:
      - backend:
          serviceName: openunison-orchestra
          servicePort: 443
        path: /
  - host: my.dashboard.example.org
    http:
      paths:
      - backend:
          serviceName: openunison-orchestra
          servicePort: 443
        path: /
  tls:
  - hosts:
    - my.login.example.org
    - my.dashboard.example.org
    secretName: ou-tls-certificate
status:
  loadBalancer: {}

from openunison-k8s-login-activedirectory.

I have re-deployed from scratch and still receive "unauthorize" as the only warning after logging in. Both cluster admin and non-cluster admin users receive the warning. This was working till we re-made the cert following this guide

from openunison-k8s-login-activedirectory.

What is saying unauthorized? Can you provide a screenshot?

from openunison-k8s-login-activedirectory.

The kubernetes dashboard is reporting unauthorized in the yellow banner. I remembered that i needed to replace the ou-ca.pem cert since i re-deployed OpenUnison but copying the cert to all three mgrs and restarting the api containers i still receive the error

from openunison-k8s-login-activedirectory.

When i inspect the cert from the OpenUnison dashboard, the issued to field is no longer the name in the info.props (OU_host or dashboard host) but: openunison.openunison.svc.cluster.local

This cert is in the unison-tls secret. Should it be something else?

from openunison-k8s-login-activedirectory.

The unison-tls certificate is the cert thats used to encrypt traffic between NGINX and OpenUnison. The API server's certificate is pulled directly from the operator's secret mount volume in the container so you shouldn't need to do anything directly related to certs. You might need to update the orchestra CR to trigger a redeploy to pick it up.

Does kubectl work?

from openunison-k8s-login-activedirectory.

Just some more notes:

I was reviewing logs when i saw that i had this log in my operator. It looked suspicious so i found ticket #64 and saw that this was not supposed to be happening.

Is OpenShift : false
Resource Version - null - true
Resource - null - already processed, skipping

i deleted the operator and after coming back it did not produce that error any more but i am still receiving an unauthorized banner.

from openunison-k8s-login-activedirectory.

kubectl does work and i can get to the dashboard with a token based login that i created when i first deployed the dashboard

from openunison-k8s-login-activedirectory.

i have now deleted the orchestra pod as well to force a re-deploy but a re-deploy of either has not worked. i have also added a annotation of "lastupdate": "april32020" to the pod configs inside of the dashboard (via the OOB token user i have) and that has not fixed it either

from openunison-k8s-login-activedirectory.

i don't think this is an openunison issue since kubectl is working. Have you tried deleting the dashboard pod?

from openunison-k8s-login-activedirectory.

i don't think this is an openunison issue since kubectl is working. Have you tried deleting the dashboard pod?

Sorry i realize i have mis-spoke, kubectl works on the mgrs, locally. not from a remote machine using openunison. I will look into re-deploying the dashboard pod

from openunison-k8s-login-activedirectory.

i deleted the dashboard pod and it re-deployed to no avail. Is there a kubernetes log i can look at to diagnose this?

from openunison-k8s-login-activedirectory.

When i try to use kubectl i get:

Unable to connect to the server: Get https://my.login.example.org/auth/idp/k8sIdp/.well-known/openid-config
uration: x509: certificate is valid for openunison.openunison.svc.cluster.local, not my.login.example.org

from openunison-k8s-login-activedirectory.

Sorry i realize i have mis-spoke, kubectl works on the mgrs, locally. not from a remote machine using openunison.

Got it. OK, lets go through a few things:

1. Check the secret ou-tls-certificate's tls.crt data element. It should have the SAN entries for your openunison portal (OU_HOST) and dashboard (K8S_DASHBOARD_HOST). This cert is hosted by your ingress controller (NGINX). It should also be the certificate in the token screen under OpenUnison Server CA Certificate

2. Use the tool sslscan on your openunison host name OU_HOST: sslscan --show-certificate k8sou.host.domain, make sure the certificate showed by the sslscan tool is the same as 1 above.

3. Make sure that the certificate referenced by your API server is also from 1 above

That will ensure that TLS is configured correctly. Are any of those 3 items off?

from openunison-k8s-login-activedirectory.

Sorry i realize i have mis-spoke, kubectl works on the mgrs, locally. not from a remote machine using openunison.

Got it. OK, lets go through a few things:

1. Check the secret ou-tls-certificate's tls.crt data element. It should have the SAN entries for your openunison portal (OU_HOST) and dashboard (K8S_DASHBOARD_HOST). This cert is hosted by your ingress controller (NGINX). It should also be the certificate in the token screen under OpenUnison Server CA Certificate
2. Use the tool sslscan on your openunison host name OU_HOST: sslscan --show-certificate k8sou.host.domain, make sure the certificate showed by the sslscan tool is the same as 1 above.
3. Make sure that the certificate referenced by your API server is also from 1 above

That will ensure that TLS is configured correctly. Are any of those 3 items off?

1. Checked the tls.crt and it shows both SAN entries.
   1.5 It is in the token screen

2. Here are the test results:

Testing SSL server my.login.example.org on port 443 using SNI name my.login.example.org

 TLS Fallback SCSV:
Server does not support TLS Fallback SCSV

 TLS renegotiation:
Session renegotiation not supported

 TLS Compression:
Compression disabled

 Heartbleed:
TLS 1.2 not vulnerable to heartbleed
TLS 1.1 not vulnerable to heartbleed
TLS 1.0 not vulnerable to heartbleed

 Supported Server Cipher(s):
Preferred TLSv1.2  256 bits  AES256-SHA256
Accepted  TLSv1.2  256 bits  AES256-SHA
Accepted  TLSv1.2  128 bits  AES128-SHA256
Accepted  TLSv1.2  128 bits  AES128-SHA

 SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
RSA Key Strength:    2048

Subject:  openunison.openunison.svc.cluster.local
Altnames: DNS:openunison.openunison.svc.cluster.local
Issuer:   kubernetes

Not valid before: Apr  2 18:06:00 2020 GMT
Not valid after:  Apr  2 18:06:00 2021 GMT

3. the API server config points to the ou-ca.pem which is the same tls.crt above. i remembered i had changed the path to /etc/kubernetes/ssl/ou-ca.pem instead of pki/ou-ca.pem since it didn't work for my original install. when i placed the ou-ca.pem inside of pki/ k8s had copied it to the ssl/ dir and in referencing it there it made it work in the original config (hope that makes sense). i have now switched it back to /pki/ou-ca.pem for diagnostics. I re-discovered the reason i had switched it to the ssl/ dir which was that i receive this error from the api dashboard in /var/log:

{"log":"error: invalid authentication config: Failed to read the CA file: open /etc/kubernetes/pki/ou-ca.pem: no such file or directory\n","stream":"stderr","time":"2020-04-03T14:18:38.433273719Z"}.
After seeing that error again i switched it back to /etc/kubernetes/ssl/ou-ca.pem and the apiserver starts again.
After the API server starting again i receive this message in the API server logs:

{"log":"E0403 14:34:17.579491       1 oidc.go:232] oidc authenticator: initializing plugin: Get https://my.login.example.org/auth/idp/k8sIdp/.well-known/openid-configuration: x509: certificate is valid for openunison.openuni
son.svc.cluster.local, not my.login.example.org\n","stream":"stderr","time":"2020-04-03T14:34:17.579839892Z"}

Still receiving the Unauthorized yellow banner... :(

I looked back at the operator and orchestra pods and the orchestra seemed find but the operator started producing this error, perhaps since the api servers were restarted?

Loading Script : '/usr/local/openunison/js/deploy-objs.js'
Loading Script : '/usr/local/openunison/js/globals.js'
Loading Script : '/usr/local/openunison/js/deploy-upstream-k8s.js'
Loading Script : '/usr/local/openunison/js/helpers.js'
Loading Script : '/usr/local/openunison/js/deploy-openshift.js'
Loading Script : '/usr/local/openunison/js/operator.js'
/apis/openunison.tremolo.io/v1/namespaces/openunison/openunisons?watch=true&resourceVersion=8917287
Is OpenShift : false
Error watching /apis/openunison.tremolo.io/v1/namespaces/openunison/openunisons?watch=true&resourceVersion=8917287
java.lang.Exception: No resourceVersion, restartinig watch
	at com.tremolosecurity.kubernetes.artifacts.util.K8sUtils.watchURI(K8sUtils.java:342)
	at com.tremolosecurity.kubernetes.artifacts.run.RunWatch.run(RunWatch.java:25)
	at java.lang.Thread.run(Thread.java:748)

I restarted the operator to clear the error and it seems to be back to a normal output despite still receiving the Unauthorized banner.

from openunison-k8s-login-activedirectory.

huh, check your ingress object (kubectl edit ingress openunison -n openunison), does spec.tls.secretName equal ou-tls-certificate? For some reason the ingress seems to be serving thee unison-tls certificate instead of ou-tls-certificate

from openunison-k8s-login-activedirectory.

So the secret looks to be correct but i wanted to post my ingress to confirm everything else looks right:

{
  "kind": "Ingress",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "openunison-ingress",
    "namespace": "openunison",
    "selfLink": "/apis/extensions/v1beta1/namespaces/openunison/ingresses/openunison-ingress",
    "uid": "adcd2bf8-4ef1-4e3c-98b5-93a0a6320a1d",
    "resourceVersion": "8918080",
    "generation": 1,
    "creationTimestamp": "2020-04-02T19:43:59Z",
    "annotations": {
      "kubernetes.io/ingress.class": "nginx",
      "lastupdated": "April32020.1",
      "nginx.ingress.kubernetes.io/affinity": "cookie",
      "nginx.ingress.kubernetes.io/backend-protocol": "https",
      "nginx.ingress.kubernetes.io/secure-backends": "true",
      "nginx.ingress.kubernetes.io/session-cookie-hash": "sha1",
      "nginx.ingress.kubernetes.io/session-cookie-name": "INGRESSCOOKIE",
      "nginx.org/ssl-services": "openunison-orchestra"
    }
  },
  "spec": {
    "tls": [
      {
        "hosts": [
          "my.login.example.org",
          "my.dashboard.example.org"
        ],
        "secretName": "ou-tls-certificate"
      }
    ],
    "rules": [
      {
        "host": "my.login.example.org",
        "http": {
          "paths": [
            {
              "path": "/",
              "backend": {
                "serviceName": "openunison-orchestra",
                "servicePort": 443
              }
            }
          ]
        }
      },
      {
        "host": "my.dashboard.example.org",
        "http": {
          "paths": [
            {
              "path": "/",
              "backend": {
                "serviceName": "openunison-orchestra",
                "servicePort": 443
              }
            }
          ]
        }
      }
    ]
  },
  "status": {
    "loadBalancer": {
      "ingress": [
        {}
      ]
    }
  }
}

i exec'd into the NGINX controller and i can't seem to find the cert in the nginx.conf. The only cert mentioned is a "default-fake-certificate.pem"

from openunison-k8s-login-activedirectory.

does my.login.example.org point to a load balancer that points to the Ingress or directly to the service port? because the ingress is stetup correctly but it appears my.login.example.org is loading the internal self signed cert, not the cert from the ingress

from openunison-k8s-login-activedirectory.

It points to the load balancer ip

and the external endpoint is configured for the service. Does it need to be restarted?

from openunison-k8s-login-activedirectory.

Something doesn't make sense. The cert being served is from the internal cert but the ingress is hosting the external cert...

Did you make any changes to your ingress? Is it nginx?

from openunison-k8s-login-activedirectory.

Try curl --insecure -v https://my.login.example.org. What's the output?

from openunison-k8s-login-activedirectory.

The ingress controller is nginx and i believe i re-deployed it as a part of my diagnostic steps but i don't believe i have made changes. Happy to gather any information about it.

I took a look at the nginx.conf for the login part. Is this first section of it correct?

## start server my.login.example.org
        server {
                server_name my.login.example.org;

                listen 80  ;
                listen [::]:80  ;
                listen 443  ssl http2 ;
                listen [::]:443  ssl http2 ;

                set $proxy_upstream_name "-";

                # PEM sha: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                ssl_certificate                         /etc/ingress-controller/ssl/default-fake-certificate.pem;
                ssl_certificate_key                     /etc/ingress-controller/ssl/default-fake-certificate.pem;

                ssl_certificate_by_lua_block {
                        certificate.call()
                }

                location / {

                        set $namespace      "openunison";
                        set $ingress_name   "openunison-ingress";
                        set $service_name   "openunison-orchestra";

I'll try the curl here in a sec

from openunison-k8s-login-activedirectory.

Here is the curl output

* Rebuilt URL to: https://my.login.example.org/
*   Trying XXX.XXX.XXX.XXX...
* TCP_NODELAY set
* Connected to my.login.example.org (XXX.XXX.XXX.XXX) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / AES256-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Virgina; L=Somewhere; O=example; OU=my-k8s-cluster; CN=openunison.openunison.svc.cluster.local
*  start date: Apr  2 18:06:00 2020 GMT
*  expire date: Apr  2 18:06:00 2021 GMT
*  issuer: CN=kubernetes
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET / HTTP/1.1
> Host: my.login.example.org
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Connection: keep-alive
< Set-Cookie: autoIdmAppName=scale; path=/; domain=my.login.example.org; secure; HttpOnly
< Set-Cookie: autoIdmSessionCookieName=tremolosession; path=/; domain=my.login.example.org; secure; HttpOnly
< Set-Cookie: openSession=f0de9429929da7b97938c4faeaf2d756d14459b6a; path=/; secure; HttpOnly; Max-Age=0; Expires=Thu, 01-Jan-1970 00:00:00 GMT
< Set-Cookie: tremolosession=eyJpdiI6IkRLZDhuSDQ3ZVVBc0VhWTg2Vi8zdmdcdTAwM2RcdTAwM2QiLCJlbmNyeXB0ZWRSZXF1ZXN0IjoiNDZrSXJlSzVXcTZNd3J5RUorMnJ4ZE9aekhSdGtxcTFLcEFFbFg5NW54YVNyY0lQV0hZdjIzekxoUWZQcjRDaCJ9; path=/; secure; HttpOnly
< Location: https://my.login.example.org/login/ldap
< Content-Length: 0
< Date: Fri, 03 Apr 2020 15:42:50 GMT
<
* Connection #0 to host my.login.example.org left intact

from openunison-k8s-login-activedirectory.

So either one of two things is true:

1. Your load balancer is bypassing ingress
2. Your ingress is passing tls back to the pod instead of terminating

See how there are no Set-Cookie response headers related to the ingress controller? There should be a cookie from the ingress controller for stickyness. Also I think there should be a response headers that mentions nginx

from openunison-k8s-login-activedirectory.

here's an example of what should come back from the response:

Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 302
< date: Fri, 03 Apr 2020 16:16:06 GMT
< content-length: 0
< location: https://xxxxxxxx/auth/forms/loginform.jsp
< server: nginx/1.17.7
< set-cookie: openunison-orchestra=xxxxxx; Path=/; Secure; HttpOnly
< set-cookie: autoIdmAppName=scale; path=/; domain=xxxxxx; secure
< set-cookie: autoIdmSessionCookieName=tremolosession; path=/; domain=xxxxxx; secure
< set-cookie: openSession=xxxxx; path=/; Max-Age=0; Expires=Thu, 01-Jan-1970 00:00:00 GMT
< set-cookie: tremolosession=xxxxxxx; path=/; secure
< strict-transport-security: max-age=15724800; includeSubDomains

There are two headers that show we're going through the ingress:

set-cookie: openunison-orchestra - This is the ingress controller's stickyness cookie
server: nginx/1.17.7 - Says the web server is nginx

I don't see either of those in your response

from openunison-k8s-login-activedirectory.

your ingress objcet looks off. Is it what was created by the operator? The operator created ingress is by default called openunison, not openunison-ingress. Also, it doesn't look like your ingress objcet was picked up. There's no status.loadBalancer.ingreess[0].ip.

from openunison-k8s-login-activedirectory.

I had re-deployed the ingress but with just a different name to avoid it getting mixed up. Is that something that could be causing an issue? I have re-deployed my nginx-controller and am going to test that next.

from openunison-k8s-login-activedirectory.

So i realize now i probably should have saved off the nginx config prior to re-making the nginx controller. Is there an easy way to get the information back in there?

from openunison-k8s-login-activedirectory.

Do you have another host name that points to the load balancer that you can do the curl test on? Or even just try it by ip? You should get an nginx error screen.

from openunison-k8s-login-activedirectory.

There shouldn't be any special config. Everything is done via ingress objects

from openunison-k8s-login-activedirectory.

Before you deploy the ingress controller, does the curl test we just tried work?

from openunison-k8s-login-activedirectory.

the nginx got a new ip from the load balancer and i did the curl against that new ip and received:

* Rebuilt URL to: https://XXX.XXX.XXX.XXX/
*   Trying XXX.XXX.XXX.XXX...
* TCP_NODELAY set
* Connected to XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Apr  3 16:13:24 2020 GMT
*  expire date: Apr  3 16:13:24 2021 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5649902a9580)
> GET / HTTP/2
> Host: XXX.XXX.XXX.XXX
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404
< server: openresty/1.15.8.1
< date: Fri, 03 Apr 2020 16:41:21 GMT
< content-type: text/plain; charset=utf-8
< content-length: 21
< strict-transport-security: max-age=15724800; includeSubDomains
<
* Connection #0 to host XXX.XXX.XXX.XXX left intact

from openunison-k8s-login-activedirectory.

i had already deployed a new nginx but i haven't deployed a new ingress for openunison

from openunison-k8s-login-activedirectory.

OK, so we know that the new IP is forwarding to the default backend. Thats a good sign, its going to thee default backend.

if you go to the openunison host what comes up?

from openunison-k8s-login-activedirectory.

i get a 404 for the host which makes me think i need to make a new ingress object or a new service object since i assume the new nginx doesn't know about them yet till they are re-registered?

from openunison-k8s-login-activedirectory.

Hmm. Check the logs for the nginx container. It ight say why it's not going to the right service

from openunison-k8s-login-activedirectory.

it is saying there are no endpoints for the orchestra. So that means i need to redeploy the service correct?

from openunison-k8s-login-activedirectory.

honestly i'd start overish. You don't need to completely redeploy, what i would do is

1. backup the cr kubectl get openunison orchestra -n openunison -o yaml > /path/to/backup.yaml
2. deletee thee cr kubectl delete openunison orchestra -n openunison
3. wait until the only thing left in the openunison namespace is the operator and its service account and your source secret
4. create the cr kubectl create -f /path/to/backup

that will get you back to a baseline.

Also, just to confirm, are you using the NGINX ingress controller? whats your load balancer in front of thee ingress?

from openunison-k8s-login-activedirectory.

Ok, i'll give it a go. Thanks for all the help! i am using an nginx ingress controller and a metallb loadbalancer

from openunison-k8s-login-activedirectory.

Made progress! everything came back up and i was able to get to the dashboard with the correct cert being passed. Now i am getting the unauthorized banner again but i see the tls.crt changed, so i assume i need to replace that on my managers and things "should" work?

from openunison-k8s-login-activedirectory.

yes sir

from openunison-k8s-login-activedirectory.

It works! Thank you so much! Just for future reference i believe my issue was that in my redeployment after the api-cert update my openunison service got the ip from the loadbalancer not the ingress controller and that is why the certs were incorrect.

from openunison-k8s-login-activedirectory.

yeah, not sure how but your load balancer was going direct to the service, bypassing the ingress.

from openunison-k8s-login-activedirectory.