Comments (12)
We don't want to run this in multiple clusters, we have 1 kubernetes cluster that spans 3 AZs. Currently, we have 3 different stateful sets (1 per AZ) so that Solr can be AZ aware in its autoscaling policies. It also used to be a nightmare with PVs/PVCs until the WaitForFirstConsumer
volumeBindingMode
for StorageClasses came out. To really support multi-AZ, I should be able to set a solr autoscaling policy that can reference the different AZs within the cluster.
from solr-operator.
Hey @AceHack , sorry for the delayed response. Things have been pretty crazy, and we have been blocked on the Kubebuilder v2 migration. Finally able to get into these other issues.
In general, Solr is relatively easy to run via multiple kubernetes clusters. What you need to do is make sure that you are using 1 zookeeper cluster for the solr clouds running in all kube clusters, so that they all share the same zookeeper connection information in their SolrCloud Specs. The data volumes don't have to be linked at all, since each Solr node manages it's data independently of other Solr Nodes.
We use different base ingress URLs to differentiate SolrCloud nodes in each kubernetes cluster.
So the solr-operator would be started in Kube cluster A with --base-ingress-domain a.test.com
and in Kube cluster B with --base-ingress-domain b.test.com
. Then when you create a SolrCloud example
in each kube cluster (both connecting to the exact same zookeeper connection information), the nodes will register under their cluster-specific domain names. You just need to make sure that your DNS rules are set up correctly to route a.test.com
and b.test.com
to the correct ingress controllers.
We are hopefully going to be able to support more addressability options in the near future, such as ExternalDNS, and give you more control over how things are created/named.
from solr-operator.
I would really like to set a policy that looks something like
{"replica": "#EQUAL", "shard": "#EACH", "sysprop.zone": "#ANY"}
or
{"replica": "#EQUAL", "shard": "#EACH", "sysprop.zone": ["us-east-1a", "us-east-1b", "us-east-1c"]}
from solr-operator.
More info
https://lucene.472066.n3.nabble.com/Rule-based-replication-or-sharding-td4407125.html#a4407660
from solr-operator.
I completely understand, and don't want the solr-operator to make any assumptions on how people configure their Solr clouds or Kube clusters.
Hopefully we can find a way to pass Node props to the environment variables of Solr. If that's not going to be an option, we might have to look into creating multiple statefulSets for each SolrCloud with multiple AZs.
Linking discussion on creating Kube options to pass Node labels to Pods: kubernetes/kubernetes#40610
from solr-operator.
It's possible in an init container or part of the main container startup you can read what AZ you are in and write it to a file or something. Requires calling kubernetes APIs.
from solr-operator.
@sepulworld, to follow up on the post you made in the PR about autoscaling. I was doing some research and it looks like Kubernetes 1.16 gives us an additional pod Option: Pod.Spec. topologySpreadConstraints
, documented here https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/
This would allow us to generically spread out pods across nodes (and AZs) without having to use autoscaling.
from solr-operator.
Without the solr Auto scaling policy being AZ aware, no matter how the pod spread out you canโt be sure the replicas are spread out across AZs.
from solr-operator.
from solr-operator.
Got a start on implementing this functionality. Let me know how y'all think it could be improved. This was just the first way I could get it done after tinkering for a while. I've very open to suggestions and improvements.
Also we should be good to go on the availability zone thing once the Labels PR & Collection Policy PR are merged, correct? Do y'all have an issue with me merging the policy PR now, or should it be blocked on this?
from solr-operator.
hi - is there any update on this?
from solr-operator.
Hey @alittlec , sorry didn't see your comment come in. Currently there are no plans to merge the currently linked PR. Instead in the v0.5.0 release we plan on support Pod Topology Spread Constraints.
The idea of autoscaling within Solr is completely changing from 8.x
to 9.0
, with the autoscaling framework deprecated and not recommended for production use. We will definitely need to investigate how the Solr Operator ties into the future autoscaling features starting with Solr 9.0.
from solr-operator.
Related Issues (20)
- Improve documentation for additional volumes HOT 1
- Resources limits and requests configuration not set on SolrCloud pod HOT 1
- Add the ability to add Environment variables as a configmap HOT 1
- Not create the StatefulSets when add the custom security.json in helm HOT 4
- Missing permission for "/admin/info/system" endpoint in security.json template in the SolrCloud CRD documentation
- Authentication not woking with solr-cloud. Pods are getting restarted. HOT 4
- Shards in a down state after an HPA scale up / scale down event. HOT 2
- User helm chart 0.8.0 with default values thorw the error in ValidationError(SolrCloud.spec): unknown field "scaling" in org.apache.solr.v1beta1.SolrCloud.spec HOT 1
- gen-pkcs12-keystore init container fails if the tls secret contains no ca.crt HOT 1
- Support running the solr operator on ARM nodes HOT 4
- Solr Backup recurrence/schedule not enabled by helm 0.7.1 HOT 1
- Actual running pod counts are different from the HPA-allocated HOT 1
- Add useful Operator metrics
- Support replicaPlacementFactory in solr.xml HOT 2
- Liveness probe failing for Prometheus Exporter connected to a large SolrCloud
- Disabling PodDisruptionBudgets for both zk pods and solr pods HOT 3
- adding automountServiceAccountToken HOT 1
- Replica allocation after Node is DisabledScheduling HOT 1
- zkHost and zkServer generated incorrectly - helm templates HOT 2
- Solr 8.11 with SolrMetrics produces duplicate samples with prometheus v2.52 HOT 12
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from solr-operator.