Helm charts enhancement about charts-clickhouse HOT 5 CLOSED

Here's what we ship by default around alerting: https://github.com/PostHog/charts-clickhouse/blob/main/charts/posthog/values.yaml#L592-L671

Runbooks I think would live alongside our documentation in the handbook w/ troubleshooting sections.

from charts-clickhouse.

Scaling & Documentation

Related issue: #129

As part of the helm charts, we should ship a basic monitoring/alerting stack.

This is kind of done (but undocumented) w/ the prometheus setup.

from charts-clickhouse.

This is kind of done (but undocumented) w/ the prometheus setup.

Do we ship basic alerts and related runbooks as well? This could have caught few of the issues I've seen in the last few days.

from charts-clickhouse.

Priorities, in my view atm:

high:

• vertical scaling for Kafka, ... (ticket #146) this I would say is high priority as we keep seeing problems/questions and we have no docs & the solution to nuke data isn't great.

mid:

• alerting on PostHog not working & troubleshooting easier for users (there's a bunch of random questions in user slack that are around k8s basics & stuff like plugin server got into a bad state - try restart - ok that worked, great)
• docs around maintenance operations/alerts

low:

• horizontal scaling: We have hpa for most things & people use it & I haven't had much user questions about it - I just see them update the charts with improvements to it, so seems like it's working pretty great.

from charts-clickhouse.

In the last year we have implemented the majority of the improvements above. I'm going to close this issue as we are now tracking the remaining tasks individually.

from charts-clickhouse.