Comments (17)
TLDR
This is unfortunately a regression I’ve introduced with #156 and it’s an edge case between the new version of the chart and Helm CRDs management. Unfortunately, this issue is only visible at runtime and with a specific combination of input values (nginx + custom hostname + cert manager enabled)
Apologise for the trouble caused 🙇 I’m going to push a fix/workaround that I hope will work for your use case.
Detailed explanation
This troubleshooting session was a real headache, especially as I’m new to the Helm/k8s world. I’ve initially thought and put my focus on how ingress.letsencrypt
gets rendered, but it was following a wrong lead.
In the PR I’ve pushed, we changed how we install cert-manager
CRDs as we now manage them directly from the upstream chart. This is good as we are modularising components, reducing code duplication, etc.. What I didn’t expect is that by removing CRDs from the crds/
folder, we also changed the behaviour of Helm.
The error Error: UPGRADE FAILED: unable to recognize “”: no matches for kind “ClusterIssuer” in version “cert-manager.io/v1”
is telling us that an apiVersion
called cert-manager.io/v1
is not present (yet) so it can't be used.
Due to a Helm ordering issue (upstream they call it "design decision"), the cert-manager
CRDs providing this missing API are not yet installed when Helm tries to install the ClusterIssuer
resource.
I’ve fixed this weird behaviour by adding a custom Helm hook that should make sure we install the ClusterIssuer
definition after everything else.
Follow up notes
Here are some additional follow up tasks to prevent this regression in the future:
- make sure the whole code base is unit tested (today I spent quite some time on a wrong path due to this)
- add integration tests to catch helm issues like this one
Additional links
Here below are also some interesting links about CRDs and Helm I've used during this investigation:
- kubernetes/enhancements#833
- helm/helm#5871
- helm/helm#2994
- helm/helm#3632
- cert-manager/cert-manager#2961
from charts-clickhouse.
Thanks a lot @tiina303! After removing PostHog and reinstalling it everything worked as expected!
from charts-clickhouse.
@guidoiaquinti could you look into this one
from charts-clickhouse.
I'm also getting the same problem :(
from charts-clickhouse.
I'm taking a look 👀
from charts-clickhouse.
@leoMehlig @peterjgrainger do you mind to try again now using the latest charts release (posthog-4.1.1
)? Happy to re-open if it didn't fix your specific case 🙇
from charts-clickhouse.
@guidoiaquinti seems to be working now, thanks!
from charts-clickhouse.
Nice! 🎉 Sorry again for the hiccup Peter!
from charts-clickhouse.
@guidoiaquinti Thanks for the quick fix! I'm now able to install everything, but somehow I'm getting a "Ingress Fake Cert" instead of a real one. Is there anything else I need to do?
from charts-clickhouse.
@leoMehlig You need to add a CNAME record for the custom domain. I had the same issue
It takes about 5 mins for the cert manager to pick up the change.
If you output the logs of the cert manager you can debug it e.g. kubectl --namespace posthog logs posthog-cert-manager-69f4ff7b57-9vkls
. You'll have a different random id on the end of the cert manager
It needs to be a CNAME record not an A record. That's where I had an issue.
from charts-clickhouse.
Thanks @peterjgrainger, but what alias should the CNAME point to?
My current logs are:
I1027 08:24:56.326670 1 start.go:74] cert-manager "msg"="starting controller" "git-commit"="969b678f330c68a6429b7a71b271761c59651a85" "version"="v1.2.0"
W1027 08:24:56.326791 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1027 08:24:56.328205 1 controller.go:169] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["10.245.0.10:53"]
I1027 08:24:56.329703 1 controller.go:129] cert-manager/controller "msg"="starting leader election"
I1027 08:24:56.330260 1 metrics.go:166] cert-manager/controller/build-context/metrics "msg"="listening for connections on" "address"={"IP":"::","Port":9402,"Zone":""}
I1027 08:24:56.331272 1 leaderelection.go:243] attempting to acquire leader lease kube-system/cert-manager-controller...
I1027 08:26:05.557573 1 leaderelection.go:253] successfully acquired lease kube-system/cert-manager-controller
I1027 08:26:05.558133 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="CertificateReadiness"
I1027 08:26:05.558248 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers"
I1027 08:26:05.558323 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="issuers"
I1027 08:26:05.558210 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim"
I1027 08:26:05.559086 1 reflector.go:207] Starting reflector *v1.Secret (5m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.859274 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="challenges"
I1027 08:26:05.859290 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned"
I1027 08:26:05.859324 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="CertificateMetrics"
I1027 08:26:05.859345 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="CertificateRequestManager"
I1027 08:26:05.859771 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="orders"
I1027 08:26:05.860058 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault"
I1027 08:26:05.860084 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="CertificateIssuing"
I1027 08:26:05.860362 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="CertificateKeyManager"
I1027 08:26:05.860365 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi"
I1027 08:26:05.860678 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="CertificateTrigger"
I1027 08:26:05.860692 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme"
I1027 08:26:05.860753 1 reflector.go:207] Starting reflector *v1.Issuer (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.860806 1 reflector.go:207] Starting reflector *v1.Challenge (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.860947 1 reflector.go:207] Starting reflector *v1.Order (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.861018 1 reflector.go:207] Starting reflector *v1.Certificate (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.861201 1 controller.go:103] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca"
I1027 08:26:05.861310 1 reflector.go:207] Starting reflector *v1.ClusterIssuer (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.860956 1 reflector.go:207] Starting reflector *v1.Pod (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.861311 1 reflector.go:207] Starting reflector *v1.CertificateRequest (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.861418 1 reflector.go:207] Starting reflector *v1.Service (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.861443 1 reflector.go:207] Starting reflector *v1.Secret (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I1027 08:26:05.861470 1 reflector.go:207] Starting reflector *v1beta1.Ingress (10h0m0s) from external/io_k8s_client_go/tools/cache/reflector.go:156
W1027 08:26:05.876445 1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 08:26:05.884160 1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
I1027 08:26:06.158470 1 conditions.go:173] Setting lastTransitionTime for Certificate "nginx-letsencrypt-posthog" condition "Ready" to 2021-10-27 08:26:06.158458068 +0000 UTC m=+69.860708021
I1027 08:26:06.160876 1 conditions.go:173] Setting lastTransitionTime for Certificate "nginx-letsencrypt-posthog" condition "Issuing" to 2021-10-27 08:26:06.16086361 +0000 UTC m=+69.863113566
E1027 08:26:06.181702 1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"nginx-letsencrypt-posthog\": the object has been modified; please apply your changes to the latest version and try again" "key"="posthog/nginx-letsencrypt-posthog"
I1027 08:26:06.181933 1 conditions.go:173] Setting lastTransitionTime for Certificate "nginx-letsencrypt-posthog" condition "Issuing" to 2021-10-27 08:26:06.181928012 +0000 UTC m=+69.884177942
E1027 08:26:06.534634 1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"nginx-letsencrypt-posthog\": the object has been modified; please apply your changes to the latest version and try again" "key"="posthog/nginx-letsencrypt-posthog"
I1027 08:26:06.546754 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "nginx-letsencrypt-posthog-qxgwv" condition "Ready" to 2021-10-27 08:26:06.546742639 +0000 UTC m=+70.248992571
I1027 08:26:06.547608 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "nginx-letsencrypt-posthog-qxgwv" condition "Ready" to 2021-10-27 08:26:06.547600092 +0000 UTC m=+70.249850025
I1027 08:26:06.547934 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "nginx-letsencrypt-posthog-qxgwv" condition "Ready" to 2021-10-27 08:26:06.547925987 +0000 UTC m=+70.250175927
I1027 08:26:06.546798 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "nginx-letsencrypt-posthog-qxgwv" condition "Ready" to 2021-10-27 08:26:06.546788202 +0000 UTC m=+70.249038145
I1027 08:26:06.548388 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "nginx-letsencrypt-posthog-qxgwv" condition "Ready" to 2021-10-27 08:26:06.548378593 +0000 UTC m=+70.250628530
E1027 08:26:06.592077 1 controller.go:158] cert-manager/controller/certificaterequests-issuer-venafi "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"nginx-letsencrypt-posthog-qxgwv\": the object has been modified; please apply your changes to the latest version and try again" "key"="posthog/nginx-letsencrypt-posthog-qxgwv"
E1027 08:26:06.594237 1 controller.go:158] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"nginx-letsencrypt-posthog-qxgwv\": the object has been modified; please apply your changes to the latest version and try again" "key"="posthog/nginx-letsencrypt-posthog-qxgwv"
E1027 08:26:06.598279 1 controller.go:158] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"nginx-letsencrypt-posthog-qxgwv\": the object has been modified; please apply your changes to the latest version and try again" "key"="posthog/nginx-letsencrypt-posthog-qxgwv"
E1027 08:26:06.606443 1 controller.go:158] cert-manager/controller/certificaterequests-issuer-ca "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"nginx-letsencrypt-posthog-qxgwv\": the object has been modified; please apply your changes to the latest version and try again" "key"="posthog/nginx-letsencrypt-posthog-qxgwv"
W1027 08:35:58.886848 1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 08:44:00.888551 1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 08:52:00.891692 1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 08:57:54.894246 1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 09:03:16.904473 1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 09:09:22.907302 1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1027 09:15:32.910932 1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
from charts-clickhouse.
@leoMehlig if you run kubectl get svc --namespace posthog posthog-ingress-nginx-controller
you can get the external IP address
Use the external IP address as the CNAME value. Don't use alias.
from charts-clickhouse.
My provider won't let me add IPs to CNAMEs, I always thought that CNAME is for redirecting to a different host.
from charts-clickhouse.
I've only set this up on AWS using these instructions https://posthog.com/docs/self-host/deploy/aws#setting-up-dns I'm not sure on other clouds or locally. CNAME should be DNS name. I use the DNS name for the loadbalancer
from charts-clickhouse.
Ok. I think this is different on DO, as you get a IP for the load balancer.
I've setup the A record for the IP, but the certificate is still not created. I already did this a couple of time before 4.0 and there this never was a problem. @guidoiaquinti is there anything else that changed?
from charts-clickhouse.
Looks like you followed the instructions. Someone from PostHog might be able to help
from charts-clickhouse.
On aws one should be setting a CNAME DNS record (as we get a hostname, not an ip), on all other platforms (we get an ip) should be an A name.
From our user slack: https://posthogusers.slack.com/archives/C01PPBY3G6Q/p1635344471020200?thread_ts=1635342422.018000&cid=C01PPBY3G6Q
Ingress Fake cert might be due to your browser caching, generally if PostHog /preflight tells me TLS is a green checkbox I assume things are good.
You can also check that you set up dns properly by running nslookup <hostname> 1.1.1.1
from charts-clickhouse.
Related Issues (20)
- Cleanup old `posthog-grafana-annotation` jobs
- Add additional configuration to externalKafka
- Read-only tables in clickhouse
- CI - Ingress: validate traffic routing
- Alertmanager - switch `RedisTooManyConnections` to be % based
- Support nodeSelector, tolerations, affinity for "events" HOT 1
- Plugin server is not starting
- Unable to change clickhouse log level
- Backups error HOT 1
- Trying to install second instance HOT 3
- Only 23.5.1 available for Helm Chart for k8 HOT 1
- Support for external Grafana HOT 3
- External Zookeeper seems ignored - references built-in
- Support GCP CloudSQL Proxy for connecting to Postgres database in GCP deployment HOT 1
- Migration failed to run HOT 1
- Service does not have load balancer ingress IP address: posthog/posthog-ingress-nginx-controller HOT 3
- Add slow lane consumer for ingestion
- Bump version of kafka
- Kafka error: 'posthog' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion). HOT 1
- Unable to reset password using SMTP.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from charts-clickhouse.