Using HPA with solr-operator about solr-operator HOT 4 CLOSED

My first thought regarding HPA.

The biggest issue with scaling down nodes is that we would need to use the Solr Collections APIs to remove and replicas from the Solr nodes that are being removed. Not sure how we would integrate with the stateful set scaling.

It would be awesome to get the operator to work well with autoscaling though, seems like one of the biggest feature requests from almost everyone I've talked to.

from solr-operator.

That makes sense. Perhaps we do a built in scaling solution within the solr-operator then? I don't think we have to feel forced to use HPA. Other projects like CoreDNS use their own built in kube-dns autoscaler and don't rely upon HPA for scaling.

from solr-operator.

Integrating Operator with AutoScaling framework I think is the way forward. We probably don't want some outside process to add and remove PODs at will? There are three levels of scaling at play here (at least):

Level 1: Nodes (One per host/VM)
Level 2: Pods (One per Solr node)
Level 3: Shards/Replicas/Cores (several per Pod)

Now, AutoScaling assumes a stable number of Nodes and Pods, and will create/remove/move cores around in the Solr cluster depending on CPU, disk, load. In the future, try to balance things out.

So what if AutoScaling could provide an API where it publishes its "external" wishes. I.e. if AutoScaling sees too full disks or too high latency, and cannot compute a plan inside the current Solr cluster to fix it, it could publish something like this:

{ "tooFewNodes": true, "tooManyNodes": false, "avgCpuPct": 80, "memoryPressurePct": 75, "diskFillRatePct": 40 }

With such feedback, the Operator could make decisions on adding Pods (or asking external system to add more VMs), or to reconfigure Pod size depending on memory, disk or CPU pressure in the cluster. No need to start more Pods if all you need is some more disk.

from solr-operator.

Thanks for the overview! I think we can close this and focus on the points discussed. The K8s HPA is probably out of scope for this problem

from solr-operator.