Issue when trying to complete ACME Challenge for custom domain used in chart: "failed to perform self check GET request Get \"http://<domain>/.well-known/acme-challenge/<token>": EOF" about dashboard HOT 4 CLOSED

That looks like a protocol mismatch issue (HTTP - HTTPS). Make sure you enabled HTTPS passthrough or tls termination on ingress and that your hostname is using https, not http.

from dashboard.

@floreks Thanks a lot for your answer. However I'm not sure I understand what you mean.

I am not trying to use an already existing TLS cert for the dashboard via a custom ingress, but simply to install the dashboard with the chart and have cert-manager automatically complete the HTTP ACME challenge to get a TLS certificate for the dashboard service.

As per the documentation in the values.yaml file for the project, I simply applied the helm dashboard chart with the override.yml file shared above (in order to add the annotation for cert-manager and the ingressClassName for nginx) and the Ingress resource is entirely created by the Helm chart release, not me.

For all my other service, I have some deployment with pods serving a port e.g. 80 or 8080. Then I map the service and the ingress to the port and the challenge always worked. I never had to use some special configuration such as HTTPS passthrough.

Here's a working example (with automatic TLS cert creation by cert-manager successful) with the kuard test app for reference: https://pastebin.com/Bx1CB5dU (put somewhere else to reduce length of message here)

Here is the resulted Ingress resource created for the Dashboard by the chart install:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: kubernetes-dashboard-letsencrypt-prod
    meta.helm.sh/release-name: kubernetes-dashboard
    meta.helm.sh/release-namespace: kubernetes-dashboard
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  generation: 1
  labels:
    app.kubernetes.io/instance: kubernetes-dashboard
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kubernetes-dashboard
    helm.sh/chart: kubernetes-dashboard-7.1.3
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ingressClassName: nginx
  rules:
  - host: dashboard.<mydomain>
    http:
      paths:
      - backend:
          service:
            name: kubernetes-dashboard-kong-proxy
            port:
              number: 443
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - dashboard.<mydomain>
    secretName: kubernetes-dashboard-certs

I can see that nginx.ingress.kubernetes.io/ssl-passthrough: "true" is set for the Dashboard Ingress, I understand this is what you were refering to as "HTTPS passthrough", right ?

Here is now the Ingress for the solver automatically created by cert-manager:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
  labels:
    acme.cert-manager.io/http-domain: "1377175528"
    acme.cert-manager.io/http-token: "1107562181"
    acme.cert-manager.io/http01-solver: "true"
  name: cm-acme-http-solver-l6mmj
  namespace: kubernetes-dashboard
  ownerReferences:
  - apiVersion: acme.cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: kubernetes-dashboard-certs-1-2599902695-2430526343
spec:
  ingressClassName: nginx
  rules:
  - host: dashboard.<mydomain>
    http:
      paths:
      - backend:
          service:
            name: cm-acme-http-solver-mm5cn
            port:
              number: 8089
        path: /.well-known/acme-challenge/TCvyORbHqI6FZRfEfadnnWO5UlD5i4kQCcFrHqSYlSw
        pathType: ImplementationSpecific

As for the point on https vs http, I believe this is relevant: an http01 ACME challenge uses http mandatorily as it is precisely used to prove ownership of the domain so that letsencrypt then grants us a TLS cert for genuine https later (source). Would this be prevented by Kong ? That is why I'm afraid of.

Here is the listing of my relevant ingresses:

# k get ing -A
NAMESPACE              NAME                        CLASS   HOSTS                  ADDRESS    PORTS     AGE
[...]
kubernetes-dashboard   cm-acme-http-solver-l6mmj   nginx   dashboard.<mydomain>   10.0.0.4   80        3m55s
kubernetes-dashboard   kubernetes-dashboard        nginx   dashboard.<mydomain>   10.0.0.4   80, 443   4m1s

Couldn't Kong absorb the request to http://dashboard.<mydomain>/.well-known/acme-challenge/TCvyORbHqI6FZRfEfadnnWO5UlD5i4kQCcFrHqSYlSw and return an empty response when it should have gone to cm-acme-http-solver-l6mmj instead ?

from dashboard.

Possibly it can but I think port 80 is disabled in Kong by default. Some kong chart values might need to be adjusted to expose it alongside 443.

from dashboard.

Issue was simply port 80 blocked network-wide... Sorry for the bother. Closing.

from dashboard.