Comments (10)
Delete the sts kubectl delete sts posthog-posthog-kafka (this doesn't lose data iiuc)
This will drop data since the pod dies.
You want to --cascade=orphan
the command not to delete the pod until update is through.
Also whether your set of commands works or not depends on the platform. It will not work on AWS since based on yesterdays incident the default StorageClass is not resizable. See the original PR description.
from charts-clickhouse.
Made the issue more generic we need to look at all platforms and document how the resize can be done (for others we might just need better docs).
from charts-clickhouse.
Tested on DigitalOcean
when someone just tries a helm upgrade with --set kafka.persistence.size=<bigger_value>Gi
we will see an error like
Error: UPGRADE FAILED: cannot patch "posthog-posthog-kafka" with kind StatefulSet: StatefulSet.apps "posthog-posthog-kafka" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden
Steps to do the resize (note use df
in the kafka pod to confirm we have more space):
- Size up pvc
kubectl edit pvc data-posthog-posthog-kafka-0 -n posthog
and change thespec.resources.requests.storage
value accordingly - Delete the sts
kubectl delete sts posthog-posthog-kafka
(this doesn't lose data iiuc) helm upgrade
with the updated size in values (will recreate the sts as needed)
from charts-clickhouse.
This also blocks (theoretically we could do some magic to use existing value or new value, but let's not) us from making our configs have better default Kafka size #145
from charts-clickhouse.
Thanks, the plan is to test it on all platforms, just started with DO & I should probably just put something in the kafka queue to check that it didn't get dropped.
from charts-clickhouse.
from charts-clickhouse.
Let me know if with PostHog/posthog.com#2202 and PostHog/posthog.com#2222 we can consider this ✅
from charts-clickhouse.
Suggestion: Perhaps we should inline the kafka helm chart. This would allow us more flexibility re default options and enforcing storageclass etc and perhaps allow us to get rid of the StatefulSets that prohibit resizing by changing values. :)
from charts-clickhouse.
What's left to do (note that I can pick it up if your plate is getting full/this is not exciting):
- there are some comments in PostHog/posthog.com#2202 (I didn't see in a follow-up yet)
- PostHog/posthog.com#2222 left a bunch of comments (but most importantly we need to have instructions how to get clusters with expandable volumes - sure we don't need to, but we care a lot about PostHog being easy to get started with and that's super helpful to have in a doc for us to figure out once rather than all users needing to go investigate)
- Major upgrade to the chart to make kafka default size = 20Gi with corresponding Major Upgrade notes that tell folks how they can get expandable volumes
Optional nice to have:
- Docs for expanding other pvc's (we can just link to kafka docs if we don't want to do anything extra)
from charts-clickhouse.
GCP: Checked our clusters (benchmarking & ee-test) in gcp and they have allowVolumeExpansion enabled both, makes me assume that's the default for gcp.
from charts-clickhouse.
Related Issues (20)
- Cleanup old `posthog-grafana-annotation` jobs
- Add additional configuration to externalKafka
- Read-only tables in clickhouse
- CI - Ingress: validate traffic routing
- Alertmanager - switch `RedisTooManyConnections` to be % based
- Support nodeSelector, tolerations, affinity for "events" HOT 1
- Plugin server is not starting
- Unable to change clickhouse log level
- Backups error HOT 1
- Trying to install second instance HOT 3
- Only 23.5.1 available for Helm Chart for k8 HOT 1
- Support for external Grafana HOT 3
- External Zookeeper seems ignored - references built-in
- Support GCP CloudSQL Proxy for connecting to Postgres database in GCP deployment HOT 1
- Migration failed to run HOT 1
- Service does not have load balancer ingress IP address: posthog/posthog-ingress-nginx-controller HOT 3
- Add slow lane consumer for ingestion
- Bump version of kafka
- Kafka error: 'posthog' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion). HOT 1
- Unable to reset password using SMTP.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from charts-clickhouse.