Unable to create new certificates about openshift-letsencrypt HOT 16 CLOSED

Route letsencrypt-cloud.crl.coloradomesa.edu not admitted.

means that it cannot reach the route. Are you sure cloud.crl.coloradomesa.edu is pointing to your http load balancer? Can you share your routes openshift-cloud-console and letsencrypt-cloud.crl.coloradomesa.edu (scrub certificates and keys if there are any):

oc get route -o yaml -n default openshift-cloud-console letsencrypt-cloud.crl.coloradomesa.edu

You will have to spin up the letsencrypt dc again of course.

Please also include the output of

oc get endpoints -n default letsencrypt

from openshift-letsencrypt.

Also, please try to access http://cloud.crl.coloradomesa.edu/.well-known/acme-challenge/.owner from your browser when openshift-letsencrypt is trying to get a certificate.

from openshift-letsencrypt.

http://cloud.crl.coloradomesa.edu/.well-known/acme-challenge was giving me a 502 before, but now it's just the generic not-found page generated by the router. So, yes, the connection to the router is working, but i guess something with the router isn't working.

output of oc get route -o yaml -n default openshift-cloud-console letsencrypt-cloud.crl.coloradomesa.edu:

apiVersion: v1
items:
- apiVersion: v1
  kind: Route
  metadata:
    creationTimestamp: 2017-07-04T09:28:29Z
    labels:
      butter.sh/letsencrypt-managed: "yes"
    name: openshift-cloud-console
    namespace: default
    resourceVersion: "228513"
    selfLink: /oapi/v1/namespaces/default/routes/openshift-cloud-console
    uid: 23efc6fc-609b-11e7-94fc-005056857075
  spec:
    host: cloud.crl.coloradomesa.edu
    port:
      targetPort: console
    tls:
      destinationCACertificate: "[redacted]"
      termination: reencrypt
    to:
      kind: Service
      name: openshift-cloud-console
      weight: 100
    wildcardPolicy: None
  status:
    ingress:
    - conditions:
      - lastTransitionTime: 2017-07-04T09:28:29Z
        status: "True"
        type: Admitted
      host: cloud.crl.coloradomesa.edu
      routerName: router
      wildcardPolicy: None
- apiVersion: v1
  kind: Route
  metadata:
    annotations:
      openshift.io/generated-by: openshift-letsencrypt
    creationTimestamp: 2017-07-04T10:11:28Z
    labels:
      butter.sh/letsencrypt-domainname: cloud.crl.coloradomesa.edu
      butter.sh/letsencrypt-well-defined: "yes"
    name: letsencrypt-cloud.crl.coloradomesa.edu
    namespace: default
    resourceVersion: "231149"
    selfLink: /oapi/v1/namespaces/default/routes/letsencrypt-cloud.crl.coloradomesa.edu
    uid: 24bb4f8d-60a1-11e7-94fc-005056857075
  spec:
    host: cloud.crl.coloradomesa.edu
    path: /.well-known/acme-challenge
    tls:
      insecureEdgeTerminationPolicy: Allow
      termination: edge
    to:
      kind: Service
      name: letsencrypt
      weight: 100
    wildcardPolicy: None
  status:
    ingress:
    - conditions:
      - lastTransitionTime: 2017-07-04T10:11:28Z
        status: "True"
        type: Admitted
      host: cloud.crl.coloradomesa.edu
      routerName: router
      wildcardPolicy: None
kind: List
metadata: {}
resourceVersion: ""
selfLink: ""

And output of oc get endpoints -n default letsencrypt:

NAME          ENDPOINTS          AGE
letsencrypt   10.131.0.22:8080   59m

from openshift-letsencrypt.

Hmm. I have a hunch, it might be the tls-termination "reencrypt" that does not work well with "edge".

I don't quiet understand what error you got at what point. When a backend is not reachable the router will return a 502 (and that not-found-page), I can't think of any reason why that might happen.

When spinning up openshiftt-letsencrypt again, wait for the route to be added (and admitted) and try to access http://cloud.crl.coloradomesa.edu/.well-known/acme-challenge/.owner. Can you check where the request gets routed? Please also check in the access log of openshift-cloud-console. Because the nginx image does not log accesses, only errors, also try to curl http://cloud.crl.coloradomesa.edu/.well-known/acme-challenge/does-not-exist. You should see some output in the nginx container of the letsencrypt pod (oc logs letsencrypt-... -c nginx).

from openshift-letsencrypt.

When i go to https://cloud.crl.coloradomesa.edu/.well-known/acme-challenge/.owner, it does work. it reaches the nginx. Going to just .well-known/acme-challenge produces a 403 code and i see errors in the nginx log.

from openshift-letsencrypt.

Interesting, I got this in the watcher log this time:

watching routes with selector butter.sh/letsencrypt-managed=yes
Processing route /oapi/v1/namespaces/default/routes/openshift-cloud-console with domain cloud.crl.coloradomesa.edu.
unable to load certificate
140526855919520:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
Getting new certificate for cloud.crl.coloradomesa.edu
Deleting well-known route.
Adding well-known route.
Route letsencrypt-cloud.crl.coloradomesa.edu not admitted.
Processing route /oapi/v1/namespaces/default/routes/openshift-cloud-console with domain cloud.crl.coloradomesa.edu.
unable to load certificate
139644322670496:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
Getting new certificate for cloud.crl.coloradomesa.edu
Deleting well-known route.
Adding well-known route.
2017/07/04 19:42:54 [error] 7#0: *5 directory index of "/opt/app-root/src/html/.well-known/acme-challenge/" is forbidden, client: 10.128.0.1, server: _, request: "GET /.well-known/acme-challenge/ HTTP/1.1", host: "cloud.crl.coloradomesa.edu"
2017/07/04 19:43:03 [error] 7#0: *6 directory index of "/opt/app-root/src/html/.well-known/acme-challenge/" is forbidden, client: 10.128.0.1, server: _, request: "GET /.well-known/acme-challenge/ HTTP/1.1", host: "cloud.crl.coloradomesa.edu"
Route letsencrypt-cloud.crl.coloradomesa.edu not admitted.

Do i need to generate the route with a self-signed cert first or can I leave the fields blank?

from openshift-letsencrypt.

So the route works fine. Can you please oc rsh to the pod and try the curl from there?

from openshift-letsencrypt.

Aha! It was another configuration problem, but this time i'll tell everyone what it was:

Internal dns didnt point the base subdomain (in this case cloud.crl.coloradomesa.edu) to the master, only the external dns was. It's working now, and I found out by running curl inside the container like @ibotty said. I didnt think about whether the container was trying to contact itself.

@ibotty thanks for helping me troubleshoot!

from openshift-letsencrypt.

Great you got it to work. That seems a frequent problem. I (or some other generous person) ought to write some troubleshoot section in the Readme.

from openshift-letsencrypt.

Also i'd like to add: Do not set any routes to redirect insecure traffic. Make the routes allow insecure so that the acme bot can use http. It was stalling on route not admitted until i changed my routes to allow. The redirect insecure applies for all routes on the domain, so it affects the acme-challenge route.

from openshift-letsencrypt.

Yes, that ought to be documented. That's a limitation of openshift's router. These routes always come first.

from openshift-letsencrypt.

Upon investigating, I cannot reproduce the problem you describe with routes. Can you be more specific (or open another bug report)?

from openshift-letsencrypt.

In short, with a edge-terminated route with either Redirect, Allow or no insecure policy, I can successfully use an Allow-edge-terminated path-restricted route.

from openshift-letsencrypt.

So it is expected that a redirect HTTP to HTTPS route should work? I'm getting Route letsencrypt-mydomain not admitted.

https://mydomain/.well-known/acme-challenge will give a cert error, if bypassed it gives Application is not available

http://mydomain/.well-known/acme-challenge will redirect to https.

from openshift-letsencrypt.

Also it does work without the redirect.

from openshift-letsencrypt.

If http://mydomain/.well-known/acme-challenge redirects, openshift's router does not use the create route. Can you please recreate the route (maybe manually) and paste the (redacted) output of oc get route -o yaml? Most routes I have are edge-terminated routes with redirect insecure policy, so I am positive that setup should work.

from openshift-letsencrypt.