Comments (11)
I added a PR .. #288 .. will u able to test with a local build ? Otherwise I can make one more release .. Please take a look at the tests in PipelineSpec
.
from scala-redis.
@noahlz - In case u face any issue with the above PR can u pls send me a reproducible test case here which I can work through.
from scala-redis.
@noahlz Thanks .. closing this issue then.
from scala-redis.
This seems to be the issue: the implementation of batch commands
38 def send[A](command: String, args: Seq[Any])(result: => A)(implicit format: Format): A = try {
39 if (batch == BATCH) {
40 handlers :+= ((command, () => result))
41 commandBuffer.append((List(command) ++ args.toList).mkString(" ") ++ crlf)
42 null.asInstanceOf[A] // hack
Note line 41 - it takes all arguments and concatenates them "as-is" to the Redis.
Do we want the user to be required to escape their key with quotes, or do that automatically? Given this function is completely agnostic as to what args
are, seems like it's up to the user(...?) Unless there are quotes added to the key of functions where appropriate, i.e. RedisClient.expire
https://code.google.com/archive/p/redis/issues/197
from scala-redis.
@debasishg Ok, from reading the code, it's rather clear to me that we should be providing double quotes around keys (and all arguments, really) when using batchedPipeline
. The only question I have is - do we want to do "magic double quotes" (apologies to PHP devs from the early 00's reading this), or merely document for users how to use batchedPipeline
properly, i.e. quote your keys, etc.
from scala-redis.
Ok, I added quotes to the keys, and it got me past EXPIRE
, but now LPUSH seems to be having issues as well
First we
LPUSH "617c85850d04fc7365744c2f:c349p7127:3:Not Available:Not Available:JPY_JPY_L_CEQ" [B@2e14997d [B@7a32edf6 [B@38fe1e9f [B@1fd4a327
Which is supposed to be storing Kryo serialized Objects
Then we try to read them back with zrange, it's clear that it serialized the string representation... not the raw bytes
dev-redis.kube.novus.local:6379> ZRANGE 617c85850d04fc7365744c2f:c349p7127:PIT:3 0 -1
1) "[B@104dd5c2"
2) "[B@106e5566"
3) "[B@107aef31"
4) "[B@107e3da6"
5) "[B@10b9387f"
6) "[B@10d3c559"
7) "[B@10d3f7c2"
8) "[B@10f3b456"
9) "[B@113c9a94"
from scala-redis.
I am able to partially solve this by encoding our byte array data as Base64 strings. Not exactly an ideal solution, as it increases the size of our data, and our code is now more complicated having to go Object > Bytes (via Kryo serialization) > Base64 String > Redis and then back again.
If we can't update the batchedPipeline
implementation to more robustly handle Array[Binary]
command arguments for LPUSH
and such - what is the most effective way to story binary java objects via that function (if not Base64?).
from scala-redis.
from scala-redis.
@noahlz hmm .. Looks like the Format
and Parse
are not passed properly when sending commands in batch. I will work on this very soon and get back to you.
BTW thanks for the other PR - I merged it.
from scala-redis.
@debasishg I opened a PR #290 demonstrating some cases where batchedPipeline errors on keys containing spaces or single quotes.
More interesting:
I ran our application with some data sets that don't have spaces or single-quotes in keys. There was no error, but after we lpush a number of Array[Binary] elements the application threads were hanging on this stack:
"pool-25-thread-5" #614 prio=5 os_prio=0 tid=0x00007f6cc0987000 nid=0x29c6 runnable [0x00007f6bdee35000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x00000007155e0aa8> (a java.io.BufferedInputStream)
at com.redis.IO.readLine(IO.scala:98)
at com.redis.IO.readLine$(IO.scala:91)
at com.redis.Redis.readLine(RedisClient.scala:34)
at com.redis.Reply.receive(RedisProtocol.scala:136)
at com.redis.Reply.receive$(RedisProtocol.scala:136)
at com.redis.Redis.receive(RedisClient.scala:34)
at com.redis.R.asLong(RedisProtocol.scala:251)
at com.redis.R.asLong$(RedisProtocol.scala:251)
at com.redis.Redis.asLong(RedisClient.scala:34)
at com.redis.ListOperations.$anonfun$lpush$1(ListOperations.scala:10)
at com.redis.ListOperations$$Lambda$7779/1438318932.apply(Unknown Source)
at com.redis.RedisClient.$anonfun$batchedPipeline$4(RedisClient.scala:250)
at com.redis.RedisClient$$Lambda$7755/1478899735.apply(Unknown Source)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:273)
at scala.collection.TraversableLike$$Lambda$4295/1723452362.apply(Unknown Source)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:273)
at scala.collection.TraversableLike.map$(TraversableLike.scala:266)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at com.redis.RedisClient.$anonfun$batchedPipeline$2(RedisClient.scala:250)
at com.redis.RedisClient$$Lambda$7725/99231071.apply(Unknown Source)
at com.redis.Redis.send(RedisClient.scala:90)
at com.redis.Redis.send(RedisClient.scala:93)
at com.redis.Redis.send(RedisClient.scala:93)
at com.redis.Redis.send(RedisClient.scala:93)
at com.redis.Redis.send(RedisClient.scala:93)
at com.redis.Redis.send(RedisClient.scala:93)
at com.redis.RedisClient.batchedPipeline(RedisClient.scala:250)
Should I open a different issue for this? This might be more difficult to provide a test case.
from scala-redis.
Can confirm this issue is resolved in the latest changes to #288! Many thanks!
from scala-redis.
Related Issues (20)
- Add support for variadic EXISTS
- How to create parser from object?
- How to use SSL Context to connect to Redis instance on AWS Elasticache with Encryption In-Transit enabled HOT 1
- Specify TTL for a key before pushing to Redis HOT 1
- Cannot find method "setCreated" in java.net.Socket. Unsupported JVM?
- Getting java.lang.Exception: Protocol error: Got (+,[B@38466868) as initial reply byte while pushing to redis quite frequently HOT 3
- Update: Missing stream commands XINFO, XADD, XTRIM, XDEL, XRANGE, XREVRANGE, XLEN, XREAD, XGROUP, XREADGROUP. XACK, XCLAIM, XAUTOCLAIM, XPENDING HOT 4
- Is server-assisted client side caching supported?
- Akka Dispatcher issue with scala redis
- How to add password to RedisClientPool? HOT 5
- QUESTION - should `RedisClient` be shared or used as a singleton? HOT 3
- RedisClient in BATCH mode (for batchedPipeline) errors when sending commands to password-protected redis servers (initial AUTH command not sent) HOT 1
- can you support sentinel HOT 4
- Rate limit support
- Added LMOVE and BLMOVE but cannot test, blocked on PR
- Using with memcached redis protocol times out
- Read from lowest latency using AWS Elasticache Global datastore?
- Protocol errors while talking to Redis in Amazon ElasticCache HOT 1
- Publish release for Scala 3 + LTS
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scala-redis.