GCP deployment results in 502 http error about charts-clickhouse HOT 5 CLOSED

Hey @mands sorry you're having difficulty here.

Some debugging info that might be useful (in the right namespace):

• gcloud beta compute ssl-certificates list and gcloud beta compute ssl-certificates describe CERTor same with get - making sure the ssl certificate is properly being provisioned and not stuck
• kubectl get pods
• kubectl logs SOME_WEB_POD - perhaps you see an issue there in the logs?

Another thing to try: to make sure this is caused by https, try setting web.secureCookies to false and ingress.gcp.forceHttps to false. If things work then (accessing via http), we can debug what's going on with the cert better :)

For deleting the old ClickhouseInstallation: It can get stuck on finalizers. One thing I've done in the past is kubectl edit ClickhouseInstallation posthog.

cc @tiina303

from charts-clickhouse.

Bit more information,

It appears that the GCP load balancer believes that the postgres service is unhealthy

However port-forwarding indicates that it's all up and running, and the ingress configuration in k8s seems correct

$ kubectl -n posthog1 port-forward service/posthog 28015:8000
Forwarding from 127.0.0.1:28015 -> 8000
Forwarding from [::1]:28015 -> 8000
Handling connection for 28015

$ http HEAD http://localhost:28015/preflight
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 2708
Content-Type: text/html; charset=utf-8
Date: Fri, 06 Aug 2021 17:25:00 GMT
Referrer-Policy: same-origin
Server: gunicorn
Vary: Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

Are there any other network fields that require setup in GCP?

from charts-clickhouse.

Hi,

Thanks for the feedback, unfortunately I wasn't able to rescue the namespace and have since deleted it (although as above, it's still stuck deleting due to the ClickHouse finalizer). The gcloud ssl cert was successfully provisioned, it seemed to be an issue with the health check from the load balancer to the pod, howeever port forwarding locally to the web pod seemed to work fine.

We run a similar setup for our own product, also on GCP/GKE, and the networking side looks pretty much the same. The main difference I could see what between using the extensions/v1beta1 Ingress API and the PostHog cluster was using NEG on GKE - perhaps may be related.

For now we're moving to using PostHog Cloud, but may be worth testing the instructions from the readme, using the config above, on a clean-room GKE cluster to see if you can repro. However feel free to close either way.

Cheers!

from charts-clickhouse.

but may be worth testing the instructions from the readme, using the config above, on a clean-room GKE cluster to see if you can repro. However feel free to close either way.

Added to our backlog for this.

from charts-clickhouse.

I've rested gcp install/upgrade a few times now, didn't see the same problem.

from charts-clickhouse.