Resizing Kafka on all platforms about charts-clickhouse HOT 10 CLOSED

Delete the sts kubectl delete sts posthog-posthog-kafka (this doesn't lose data iiuc)

This will drop data since the pod dies.

You want to --cascade=orphan the command not to delete the pod until update is through.

Also whether your set of commands works or not depends on the platform. It will not work on AWS since based on yesterdays incident the default StorageClass is not resizable. See the original PR description.

from charts-clickhouse.

Made the issue more generic we need to look at all platforms and document how the resize can be done (for others we might just need better docs).

from charts-clickhouse.

Tested on DigitalOcean

when someone just tries a helm upgrade with --set kafka.persistence.size=<bigger_value>Gi we will see an error like

Error: UPGRADE FAILED: cannot patch "posthog-posthog-kafka" with kind StatefulSet: StatefulSet.apps "posthog-posthog-kafka" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

Steps to do the resize (note use df in the kafka pod to confirm we have more space):

1. Size up pvc kubectl edit pvc data-posthog-posthog-kafka-0 -n posthog and change the spec.resources.requests.storage value accordingly
2. Delete the sts kubectl delete sts posthog-posthog-kafka (this doesn't lose data iiuc)
3. helm upgrade with the updated size in values (will recreate the sts as needed)

from charts-clickhouse.

This also blocks (theoretically we could do some magic to use existing value or new value, but let's not) us from making our configs have better default Kafka size #145

from charts-clickhouse.

Thanks, the plan is to test it on all platforms, just started with DO & I should probably just put something in the kafka queue to check that it didn't get dropped.

from charts-clickhouse.

PostHog/posthog.com#2202

from charts-clickhouse.

Let me know if with PostHog/posthog.com#2202 and PostHog/posthog.com#2222 we can consider this ✅

from charts-clickhouse.

Suggestion: Perhaps we should inline the kafka helm chart. This would allow us more flexibility re default options and enforcing storageclass etc and perhaps allow us to get rid of the StatefulSets that prohibit resizing by changing values. :)

from charts-clickhouse.

What's left to do (note that I can pick it up if your plate is getting full/this is not exciting):

• there are some comments in PostHog/posthog.com#2202 (I didn't see in a follow-up yet)
• PostHog/posthog.com#2222 left a bunch of comments (but most importantly we need to have instructions how to get clusters with expandable volumes - sure we don't need to, but we care a lot about PostHog being easy to get started with and that's super helpful to have in a doc for us to figure out once rather than all users needing to go investigate)
• Major upgrade to the chart to make kafka default size = 20Gi with corresponding Major Upgrade notes that tell folks how they can get expandable volumes

Optional nice to have:

• Docs for expanding other pvc's (we can just link to kafka docs if we don't want to do anything extra)

from charts-clickhouse.

GCP: Checked our clusters (benchmarking & ee-test) in gcp and they have allowVolumeExpansion enabled both, makes me assume that's the default for gcp.

from charts-clickhouse.