Document backup / retention methods about charts-clickhouse HOT 8 CLOSED

@guidoiaquinti I have also added PostHog/posthog.com#2389

from charts-clickhouse.

@joesaunderson I've also added PostHog/posthog.com#2372 to our docs in order to make it more clear (and apologise as it wasn't before). Let me know if you have all the answers you need and I'll mark this issue as ✅

from charts-clickhouse.

Hi @joesaunderson ! With PostHog/posthog.com#2370 we’ve just published a new ClickHouse section in our runbooks that I hope will answer your questions about this service.

Regarding PostgreSQL: we should make it explicit in our docs but we suggest you to back up the data as well like you were doing before moving analytics queries to ClickHouse.

Let me know if this is 👍 and if we can consider this issue ✅

Have a good day!

from charts-clickhouse.

Hey @guidoiaquinti - thanks for this, it really helps my understanding on how things work. I guess a couple of questions:

• How simple is it to install the automated backup, it looks fairly simple, but would we need to build this into Helm? Could it be added automatically?

• This covers Clickhouse really nicely, do we need to backup the Postgres DB if it's within the K8s stack?

from charts-clickhouse.

How simple is it to install the automated backup, it looks fairly simple, but would we need to build this into Helm? Could it be added automatically?

We are not planning to add ClickHouse backup/restore to Helm for now, but we can re-evaluate this in the future.

This covers Clickhouse really nicely, do we need to backup the Postgres DB if it’s within the K8s stack?

Yes, our suggestion is to back up PostgreSQL as well daily/bi-daily (depending on your requirements). If you don’t have a specific reason to run stateful services in K8S we also suggest that you run them “outside” Kubernetes.

Example: if you are running on AWS, you could use AWS Aurora for PostgreSQL and AWS MSK for Kafka (unfortunately there isn’t yet a ClickHouse as a service solution.

Let me know if this makes sense!

from charts-clickhouse.

We are not planning to add ClickHouse backup/restore to Helm for now, but we can re-evaluate this in the future.

I'm not so familiar with Helm, but does this mean if we add it, and you guys push an update for the chart, we'd need to add it again?

If you don’t have a specific reason to run stateful services in K8S we also suggest that you run them “outside” Kubernetes.

Example: if you are running on AWS, you could use AWS Aurora for PostgreSQL and AWS MSK for Kafka

Makes sense, we are looking into this, would it make sense to mention this on the AWS docs, along with some 'recommendations' on the size of the Postgres / Kafka instances?

from charts-clickhouse.

I’m not so familiar with Helm, but does this mean if we add it, and you guys push an update for the chart, we’d need to add it again?

It depends on how you mange your k8s cluster. You can either:

1. manage the PostHog via our Helm charts + run an additional kubectl command to install the clickhouse-backup related resources.
2. extend our Helm chart and add whatever additional bits you need

Makes sense, we are looking into this, would it make sense to mention this on the AWS docs, along with some ‘recommendations’ on the size of the Postgres / Kafka instances?

Yes, we can add that. For PostgreSQL / Kafka instance size it’s difficult to know upfront without having more data about your expected volume of events, unique people, retention policy, etc... My suggestion is to start with the same values you currently use in Helm and then add capacity if necessary.

from charts-clickhouse.

@guidoiaquinti I think this can be ✅

from charts-clickhouse.