Git Product home page Git Product logo

Comments (17)

guidoiaquinti avatar guidoiaquinti commented on August 25, 2024 2

TLDR

This is unfortunately a regression I’ve introduced with #156 and it’s an edge case between the new version of the chart and Helm CRDs management. Unfortunately, this issue is only visible at runtime and with a specific combination of input values (nginx + custom hostname + cert manager enabled)

Apologise for the trouble caused 🙇 I’m going to push a fix/workaround that I hope will work for your use case.

Detailed explanation

This troubleshooting session was a real headache, especially as I’m new to the Helm/k8s world. I’ve initially thought and put my focus on how ingress.letsencrypt gets rendered, but it was following a wrong lead.

In the PR I’ve pushed, we changed how we install cert-manager CRDs as we now manage them directly from the upstream chart. This is good as we are modularising components, reducing code duplication, etc.. What I didn’t expect is that by removing CRDs from the crds/ folder, we also changed the behaviour of Helm.

The error Error: UPGRADE FAILED: unable to recognize “”: no matches for kind “ClusterIssuer” in version “cert-manager.io/v1” is telling us that an apiVersion called cert-manager.io/v1 is not present (yet) so it can't be used.

Due to a Helm ordering issue (upstream they call it "design decision"), the cert-manager CRDs providing this missing API are not yet installed when Helm tries to install the ClusterIssuer resource.

I’ve fixed this weird behaviour by adding a custom Helm hook that should make sure we install the ClusterIssuer definition after everything else.

Follow up notes

Here are some additional follow up tasks to prevent this regression in the future:

  1. make sure the whole code base is unit tested (today I spent quite some time on a wrong path due to this)
  2. add integration tests to catch helm issues like this one

Additional links

Here below are also some interesting links about CRDs and Helm I've used during this investigation:

from charts-clickhouse.

leoMehlig avatar leoMehlig commented on August 25, 2024 1

Thanks a lot @tiina303! After removing PostHog and reinstalling it everything worked as expected!

from charts-clickhouse.

tiina303 avatar tiina303 commented on August 25, 2024

@guidoiaquinti could you look into this one

from charts-clickhouse.

peterjgrainger avatar peterjgrainger commented on August 25, 2024

I'm also getting the same problem :(

from charts-clickhouse.

guidoiaquinti avatar guidoiaquinti commented on August 25, 2024

I'm taking a look 👀

from charts-clickhouse.

guidoiaquinti avatar guidoiaquinti commented on August 25, 2024

@leoMehlig @peterjgrainger do you mind to try again now using the latest charts release (posthog-4.1.1)? Happy to re-open if it didn't fix your specific case 🙇

from charts-clickhouse.

peterjgrainger avatar peterjgrainger commented on August 25, 2024

@guidoiaquinti seems to be working now, thanks!

from charts-clickhouse.

guidoiaquinti avatar guidoiaquinti commented on August 25, 2024

Nice! 🎉 Sorry again for the hiccup Peter!

from charts-clickhouse.

leoMehlig avatar leoMehlig commented on August 25, 2024

@guidoiaquinti Thanks for the quick fix! I'm now able to install everything, but somehow I'm getting a "Ingress Fake Cert" instead of a real one. Is there anything else I need to do?

from charts-clickhouse.

peterjgrainger avatar peterjgrainger commented on August 25, 2024

@leoMehlig You need to add a CNAME record for the custom domain. I had the same issue

It takes about 5 mins for the cert manager to pick up the change.

If you output the logs of the cert manager you can debug it e.g. kubectl --namespace posthog logs posthog-cert-manager-69f4ff7b57-9vkls. You'll have a different random id on the end of the cert manager

It needs to be a CNAME record not an A record. That's where I had an issue.

from charts-clickhouse.

leoMehlig avatar leoMehlig commented on August 25, 2024

Thanks @peterjgrainger, but what alias should the CNAME point to?

My current logs are:

I1027 08:24:56.326670       1 start.go:74] cert-manager "msg"="starting controller"  "git-commit"="969b678f330c68a6429b7a71b271761c59651a85" "version"="v1.2.0"
W1027 08:24:56.326791       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1027 08:24:56.328205       1 controller.go:169] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["10.245.0.10:53"] 
I1027 08:24:56.329703       1 controller.go:129] cert-manager/controller "msg"="starting leader election"  
I1027 08:24:56.330260       1 metrics.go:166] cert-manager/controller/build-context/metrics "msg"="listening for connections on" "address"={"IP":"::","Port":9402,"Zone":""} 
I1027 08:24:56.331272       1 leaderelection.go:243] attempting to acquire leader lease  kube-system/cert-manager-controller...
I1027 08:26:05.557573       1 leaderelection.go:253] successfully acquired lease kube-system/cert-manager-controller
I1027 08:26:05.558133       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="CertificateReadiness" 
I1027 08:26:05.558248       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers" 
I1027 08:26:05.558323       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="issuers" 
I1027 08:26:05.558210       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim" 
I1027 08:26:05.559086       1 reflector.go:207] Starting reflector *v1.Secret (5m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.859274       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="challenges" 
I1027 08:26:05.859290       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned" 
I1027 08:26:05.859324       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="CertificateMetrics" 
I1027 08:26:05.859345       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="CertificateRequestManager" 
I1027 08:26:05.859771       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="orders" 
I1027 08:26:05.860058       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault" 
I1027 08:26:05.860084       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="CertificateIssuing" 
I1027 08:26:05.860362       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="CertificateKeyManager" 
I1027 08:26:05.860365       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi" 
I1027 08:26:05.860678       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="CertificateTrigger" 
I1027 08:26:05.860692       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme" 
I1027 08:26:05.860753       1 reflector.go:207] Starting reflector *v1.Issuer (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.860806       1 reflector.go:207] Starting reflector *v1.Challenge (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.860947       1 reflector.go:207] Starting reflector *v1.Order (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.861018       1 reflector.go:207] Starting reflector *v1.Certificate (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.861201       1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca" 
I1027 08:26:05.861310       1 reflector.go:207] Starting reflector *v1.ClusterIssuer (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.860956       1 reflector.go:207] Starting reflector *v1.Pod (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.861311       1 reflector.go:207] Starting reflector *v1.CertificateRequest (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.861418       1 reflector.go:207] Starting reflector *v1.Service (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.861443       1 reflector.go:207] Starting reflector *v1.Secret (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.861470       1 reflector.go:207] Starting reflector *v1beta1.Ingress (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
W1027 08:26:05.876445       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 08:26:05.884160       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
I1027 08:26:06.158470       1 conditions.go:173] Setting lastTransitionTime for Certificate "nginx-letsencrypt-posthog" condition "Ready" to 2021-10-27 08:26:06.158458068 +0000 UTC m=+69.860708021
I1027 08:26:06.160876       1 conditions.go:173] Setting lastTransitionTime for Certificate "nginx-letsencrypt-posthog" condition "Issuing" to 2021-10-27 08:26:06.16086361 +0000 UTC m=+69.863113566
E1027 08:26:06.181702       1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"nginx-letsencrypt-posthog\": the object has been modified; please apply your changes to the latest version and try again" "key"="posthog/nginx-letsencrypt-posthog" 
I1027 08:26:06.181933       1 conditions.go:173] Setting lastTransitionTime for Certificate "nginx-letsencrypt-posthog" condition "Issuing" to 2021-10-27 08:26:06.181928012 +0000 UTC m=+69.884177942
E1027 08:26:06.534634       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"nginx-letsencrypt-posthog\": the object has been modified; please apply your changes to the latest version and try again" "key"="posthog/nginx-letsencrypt-posthog" 
I1027 08:26:06.546754       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "nginx-letsencrypt-posthog-qxgwv" condition "Ready" to 2021-10-27 08:26:06.546742639 +0000 UTC m=+70.248992571
I1027 08:26:06.547608       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "nginx-letsencrypt-posthog-qxgwv" condition "Ready" to 2021-10-27 08:26:06.547600092 +0000 UTC m=+70.249850025
I1027 08:26:06.547934       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "nginx-letsencrypt-posthog-qxgwv" condition "Ready" to 2021-10-27 08:26:06.547925987 +0000 UTC m=+70.250175927
I1027 08:26:06.546798       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "nginx-letsencrypt-posthog-qxgwv" condition "Ready" to 2021-10-27 08:26:06.546788202 +0000 UTC m=+70.249038145
I1027 08:26:06.548388       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "nginx-letsencrypt-posthog-qxgwv" condition "Ready" to 2021-10-27 08:26:06.548378593 +0000 UTC m=+70.250628530
E1027 08:26:06.592077       1 controller.go:158] cert-manager/controller/certificaterequests-issuer-venafi "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"nginx-letsencrypt-posthog-qxgwv\": the object has been modified; please apply your changes to the latest version and try again" "key"="posthog/nginx-letsencrypt-posthog-qxgwv" 
E1027 08:26:06.594237       1 controller.go:158] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"nginx-letsencrypt-posthog-qxgwv\": the object has been modified; please apply your changes to the latest version and try again" "key"="posthog/nginx-letsencrypt-posthog-qxgwv" 
E1027 08:26:06.598279       1 controller.go:158] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"nginx-letsencrypt-posthog-qxgwv\": the object has been modified; please apply your changes to the latest version and try again" "key"="posthog/nginx-letsencrypt-posthog-qxgwv" 
E1027 08:26:06.606443       1 controller.go:158] cert-manager/controller/certificaterequests-issuer-ca "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"nginx-letsencrypt-posthog-qxgwv\": the object has been modified; please apply your changes to the latest version and try again" "key"="posthog/nginx-letsencrypt-posthog-qxgwv" 
W1027 08:35:58.886848       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 08:44:00.888551       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 08:52:00.891692       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 08:57:54.894246       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 09:03:16.904473       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 09:09:22.907302       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 09:15:32.910932       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

from charts-clickhouse.

peterjgrainger avatar peterjgrainger commented on August 25, 2024

@leoMehlig if you run kubectl get svc --namespace posthog posthog-ingress-nginx-controller you can get the external IP address

Use the external IP address as the CNAME value. Don't use alias.

from charts-clickhouse.

leoMehlig avatar leoMehlig commented on August 25, 2024

My provider won't let me add IPs to CNAMEs, I always thought that CNAME is for redirecting to a different host.

from charts-clickhouse.

peterjgrainger avatar peterjgrainger commented on August 25, 2024

I've only set this up on AWS using these instructions https://posthog.com/docs/self-host/deploy/aws#setting-up-dns I'm not sure on other clouds or locally. CNAME should be DNS name. I use the DNS name for the loadbalancer

from charts-clickhouse.

leoMehlig avatar leoMehlig commented on August 25, 2024

Ok. I think this is different on DO, as you get a IP for the load balancer.

I've setup the A record for the IP, but the certificate is still not created. I already did this a couple of time before 4.0 and there this never was a problem. @guidoiaquinti is there anything else that changed?

from charts-clickhouse.

peterjgrainger avatar peterjgrainger commented on August 25, 2024

Looks like you followed the instructions. Someone from PostHog might be able to help

from charts-clickhouse.

tiina303 avatar tiina303 commented on August 25, 2024

On aws one should be setting a CNAME DNS record (as we get a hostname, not an ip), on all other platforms (we get an ip) should be an A name.

From our user slack: https://posthogusers.slack.com/archives/C01PPBY3G6Q/p1635344471020200?thread_ts=1635342422.018000&cid=C01PPBY3G6Q

Ingress Fake cert might be due to your browser caching, generally if PostHog /preflight tells me TLS is a green checkbox I assume things are good.

You can also check that you set up dns properly by running nslookup <hostname> 1.1.1.1

from charts-clickhouse.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.