Comments (16)
Thanks for reporting .. will check up over the weekend ..
from scala-redis.
Did the changes, pls check. Closing the issue. Will reopen if required.
from scala-redis.
the changes don't allow pipelining to the cluster:
redisCluster.pipeline { p=> // execute redis commands }
from scala-redis.
ouch! Forgot to test pipelining .. will check very soon. Thanks for reporting ..
from scala-redis.
Actually, what does pipelining to a cluster mean ? Select any node from the cluster and do the pipeline ? What do u think ?
from scala-redis.
I am not sure if it should select any node, each command in the pipeline should use the same method for finding a node that the cluster would use for a normal command (i.e. each command should run on the node determined by nodeForKey(key) if valid).
e.g.
cluster.pipeline { p =>
p.incr(key)
p.expire(anotherKey)
}
p.incr would run on nodeForKey(key) and p.expire would run on nodeForKey(anotherKey) - would that not be the correct approach? (the equivalent of running cluster.incr(key) cluster.expire(anotherKey) and sending back the joined results.)
from scala-redis.
On another note, we are using redis cluster for HA, not only for performance / scaling, so any write operation is run with cluster.onAllConns, while read operations are invoked directly, using nodeForKey for a single return. Would be great if this could be handled by the driver..
def faultTolerantRunOnAll[T](body: RedisClient => T): T = {
checkAllHosts()
try {
redis.onAllConns { ac =>
body(ac)
}.head
}
catch {
case err => {
faultTolerantRunOnAll(body)
}
}
}
def checkAllHosts(): Boolean = {
var curHost = ""
var hasError: Boolean = false
redis.clients.foreach { pool =>
try {
curHost = pool.toString()
pool.withClient { client => }
}
catch {
case ex: RuntimeException => {
if (ex.getCause.isInstanceOf[ConnectException]) {
hasError = true
if (nodes.length > 1) {
log.debug("Removing unconnectable redis node: " + curHost)
deadNodes ++= List(curHost)
nodes = nodes diff Seq(curHost)
connect()
checkAllHosts()
}
else throw new ConnectException("No connectable redis nodes in cluster")
}
else throw ex
}
case ex => throw ex
}
}
hasError
}
from scala-redis.
Regarding "any node for pipe-lining", I think all composing commands of a pipeline need to be executed on the same Redis server. The protocol is something like a transaction and the sequence is "MULTI" - [list of commands] - "EXEC" / "DISCARD". The server on getting a MULTI starts queuing responses - hence we need to fix on a server and then execute the whole pipeline there. Hence the thought of selecting a random node ..
from scala-redis.
ah, true. But then this could potentially cause problems:
cluster.set(key, value) // sets on nodeX
cluster.pipeline { p =>
p.get(key) // gets from a random node, not necessarily nodeX
}
from scala-redis.
good catch! I think pipelining doesn't look viable in a cluster implementation based on consistent hashing of keys ..
from scala-redis.
regarding HA, your proposal looks good and possibly can be handled by the driver. Complications come up with failure handling .. if one of the nodes fail, do we consider the write successful ? Then comes up all the complexities of quorums and what not .. Keeping it simple, we can treat all-writes-or-none as the semantics. What do you think ?
from scala-redis.
Unless it was possible to replicate the queuing on the driver side? so a server never gets a multi, it just gets the command, the response is returned to the driver, and then the driver queues up responses locally?
I have limited knowledge on the internals of redis, so not sure about how redis's queuing works, but is it not feasible to just concatenate the results from each command and return that?
HA: this is our own implementation that we use, and considering failure handling makes me wonder if it would be viable for a driver. One issue that we face is that if 1 redis node goes down, if we start it back up and restart the application then 1 node is out of sync with the rest, it may be missing keys. When we run this scenario, the redis nodes have no replication (due to the lack of master master replication currently), so this is a design with inherent consistency flaws
from scala-redis.
Queuing on the driver side is tricky considering the fact that Redis does quite a few optimizations at the server level to improve performance. e.g. see http://redis.io/topics/transactions. Just queuing and concatenating may not work - need to ensure atomicity. In Redis server if the server crashes after executing a part of the pipelined commands, it detects during the subsequent start up and handles the erroneous condition.
Regarding HA and failover I initially thought of implementing something like https://github.com/ryanlecompte/redis_failover, but thought that I should wait till the sentinel stuff (http://redis.io/topics/sentinel) comes up. It is aimed at managing multiple Redis instances.
from scala-redis.
yeah, we have looked at that, and was going to try implement something similar (redis_failover).
It does seem like pipelining in a cluster is not viable.
from scala-redis.
Pipelining in a cluster is viable with the assumption of same-key-within-pipeline. Operations on same key (representing a single Redis structure) are guaranteed to hit same node - and they can form very useful transactions.
from scala-redis.
Pipeling as we use it has the benefit in optimizing network throughput toi the DB system compared to single operation requests. considering the cluster machines will likely be installed in the same rack witin the same switch, networking between node is comparable more performant than from possible data producers (referring to the client side).
Having dedicated queues for pipelining a single cluster node in my eyes is a misdesign - as you need to have knowledge of where data resides, what's the benefit of having a cluster (we've some server farms and I know rehashing because of new nodes is not a daily job - even if that benefit from the cluster is a nice to have)?
The complexity in the driver or client about getting metainformation, reacting on key-not-found-errors results in error prone software and bad maintenance. I'd rather have a two-hop-design within the cluster, so the cluster node itself is able to redirect to the correct node in case the client hit's the wrong node.
from scala-redis.
Related Issues (20)
- incorrect tag name on GitHub. s/v3.3.0/v3.30/ HOT 1
- Add support for variadic EXISTS
- How to create parser from object?
- How to use SSL Context to connect to Redis instance on AWS Elasticache with Encryption In-Transit enabled HOT 1
- Specify TTL for a key before pushing to Redis HOT 1
- Cannot find method "setCreated" in java.net.Socket. Unsupported JVM?
- Getting java.lang.Exception: Protocol error: Got (+,[B@38466868) as initial reply byte while pushing to redis quite frequently HOT 3
- Update: Missing stream commands XINFO, XADD, XTRIM, XDEL, XRANGE, XREVRANGE, XLEN, XREAD, XGROUP, XREADGROUP. XACK, XCLAIM, XAUTOCLAIM, XPENDING HOT 4
- Is server-assisted client side caching supported?
- Akka Dispatcher issue with scala redis
- How to add password to RedisClientPool? HOT 5
- "Exception: ERR wrong number of arguments f or 'expire' command" occurs when attempting to expire a key that contains spaces. HOT 11
- QUESTION - should `RedisClient` be shared or used as a singleton? HOT 3
- RedisClient in BATCH mode (for batchedPipeline) errors when sending commands to password-protected redis servers (initial AUTH command not sent) HOT 1
- can you support sentinel HOT 4
- Rate limit support
- Added LMOVE and BLMOVE but cannot test, blocked on PR
- Using with memcached redis protocol times out
- Read from lowest latency using AWS Elasticache Global datastore?
- Protocol errors while talking to Redis in Amazon ElasticCache HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scala-redis.