Comments (6)
Probably also want to evaluate the Redis Operator and anything else that looks promising.
from open-match.
Some thoughts.
Having only HAProxy and replicated Redis instances assumes manual interventions when Redis master becomes unreachable to clients or to other Redis instances. So that's not real HA, HAProxy only can do the load balancing. At the moment there are two ways to achive HA:
- Set up Redis Cluster - several sharded masters, each replicated to slaves (so quite a lot of boxes to configure and run); clients need to support it.
- Run at least 3 Redis Sentinel boxes that are monitoring several replicated Redis instances: when current master fails they promote some slave to master; clients need to connect to Sentinels to obtain the address of current master (according to https://redis.io/topics/sentinel-clients). Some people however prefer to have HAProxy pointing right to Redis instances and figuring out who of them is master - then clients don't need to support Sentinel.
More information:
Regarding the Operators: there are several implementation, https://github.com/spotahome/redis-operator seems to be most starred. It allows to easily deploy Redis replicas with Sentinels saving us from having to configure everything manually. The downside is that clients have to support Sentinel, and because of autoconfiguration it may be not so easy to workaround that with HAProxy.
Also there's Helm chart with similar setup (3 Redises + 3 Sentinels).
from open-match.
Yeah, I didn't do a good job of explaining, sorry :( I should have said "HA proxy + multiple redis instances set up for sentinel" or the like. We'll need the redis-sentinel for the automatic failover and resiliency, and the HA proxy (or a k8s resource that can do something similar) so that the clients don't have to be 'redis-sentinel' clients, but just regular redis clients to keep this a 'drop-in' replacement. Something like this: https://karlstoney.com/2015/07/23/redis-sentinel-behind-haproxy/
If there's a Redis operator that can get us along the path, that's probably what we want to use.
from open-match.
Redis Operator definitely simplifies Redis management in k8s. If we decide to go with it then I see several ways to still live with 'non-sentinel' client.
-
redis-sentinel-proxy
+ always proxies incomming connections to relevant master address (obtained from Sentinel)
+ easy to deploy and configure - just point it to the Sentinel service (that is managed by RedisFailover)
- seems to be non-zero copy proxy, not sure about its stability & performance (though it works in general)
- requires additional Dockerfile + k8s Deployment + k8s Service -
Have HAProxy configured as here
+ that's HAProxy
- it's not apparent how to pass Redis pods' IPs to configure HAProxy properly and keep it in sync (considering the lifecycle of Pod)
- an alternative to IPs in ^^^ is to have a service-per-redis-pod which looks a bit ugly to me
- theoretically in case of split-brain multiple Redis pods may be claiming that they are masters - not sure how HAProxy will handle the connections then -
Fork Redis Operator to add the ability to label master Redis pod (from StatefulSet) as "Current Master"
+ if implemented, it would require only one more k8s resource to be managed: a Service pointing to the Redis master pod
- additional time to implement that functionality
- ...and get the PR merged in original repo: same feature was already proposed but sort of declined by authors (however hopefully they won't mind adding only the labeling)
- in theory there may occur some issues related to the sync of pods' labels with actual state of Sentinels' cluster (delays? two pods labeled as master during some short periods of time?), we need somebody to think about this more. -
It's possible to have HAProxy always (well, most of time) pointing to the actual Redis master pod IP (obtained from Sentinel):
0) enabled HAproxy Runtime API
1) lookup master address like this:
echo SENTINEL get-master-addr-by-name mymaster | nc -q 5 <RedisFailover-Sentinel-service-name> 26379
2) parse the output to extract IP
3) write new config with that IP address to HAproxy unix socket like the following:
echo "set server bk_redis/redis-master-serv addr <IP> port <PORT>" | sudo socat stdio /var/run/haproxy.sock
4) repeat 1..3 every second?
Changing HAproxy backend server address on fly works well. Still exploring this option.
from open-match.
I like the first option, provided it has reasonable performance characteristics. Thanks for all the research!
from open-match.
PR #48
from open-match.
Related Issues (20)
- Add pod annotations to Helm templates for injecting user-specified pod annotations. HOT 2
- Release 1.8.0-rc.1
- flaky test in testing/e2e/backfill_test.go
- flaky test in testing/e2e/cluster_test.go
- Publish the Release on Open Match Blog.
- Release 1.8.0
- 1.8.0 YAML files still contain some jaeger traces
- How to configure a custom Redis password? HOT 1
- The helm chart have minReplicas set to 3 for query pods.
- add ability to set uo log output
- Helm chart `query.replicas` values are ignored if less than 3
- QUESTION: What must the contents of matchmaker_config_default.yaml and matchmaker_config_override.yaml be for minimatch? HOT 1
- Release 1.8.1
- failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed HOT 1
- The tutorials cannot generate the docker images if the protobuf definitions changed HOT 3
- Ticket Key Leak in proposedTicketIDs ZSet
- Helm Chart - allow to use user-provided secret for Redis auth if not using Bitnami Redis chart
- Inconsistent player ticket extension behavior between regular vs. backfill matches
- Add 'deprecated-public-image' tag to very old versions of the images in the public container registry
- Redis Username is ignored
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from open-match.