Git Product home page Git Product logo

Comments (11)

debasishg avatar debasishg commented on July 28, 2024 1

I added a PR .. #288 .. will u able to test with a local build ? Otherwise I can make one more release .. Please take a look at the tests in PipelineSpec.

from scala-redis.

debasishg avatar debasishg commented on July 28, 2024 1

@noahlz - In case u face any issue with the above PR can u pls send me a reproducible test case here which I can work through.

from scala-redis.

debasishg avatar debasishg commented on July 28, 2024 1

@noahlz Thanks .. closing this issue then.

from scala-redis.

noahlz avatar noahlz commented on July 28, 2024

This seems to be the issue: the implementation of batch commands

 38   def send[A](command: String, args: Seq[Any])(result: => A)(implicit format: Format): A = try {
 39     if (batch == BATCH) {
 40       handlers :+= ((command, () => result))
 41       commandBuffer.append((List(command) ++ args.toList).mkString(" ") ++ crlf)
 42       null.asInstanceOf[A] // hack

Note line 41 - it takes all arguments and concatenates them "as-is" to the Redis.

Do we want the user to be required to escape their key with quotes, or do that automatically? Given this function is completely agnostic as to what args are, seems like it's up to the user(...?) Unless there are quotes added to the key of functions where appropriate, i.e. RedisClient.expire

https://code.google.com/archive/p/redis/issues/197

from scala-redis.

noahlz avatar noahlz commented on July 28, 2024

@debasishg Ok, from reading the code, it's rather clear to me that we should be providing double quotes around keys (and all arguments, really) when using batchedPipeline. The only question I have is - do we want to do "magic double quotes" (apologies to PHP devs from the early 00's reading this), or merely document for users how to use batchedPipeline properly, i.e. quote your keys, etc.

from scala-redis.

noahlz avatar noahlz commented on July 28, 2024

Ok, I added quotes to the keys, and it got me past EXPIRE, but now LPUSH seems to be having issues as well

First we

LPUSH "617c85850d04fc7365744c2f:c349p7127:3:Not Available:Not Available:JPY_JPY_L_CEQ" [B@2e14997d [B@7a32edf6 [B@38fe1e9f [B@1fd4a327

Which is supposed to be storing Kryo serialized Objects

Then we try to read them back with zrange, it's clear that it serialized the string representation... not the raw bytes

dev-redis.kube.novus.local:6379> ZRANGE 617c85850d04fc7365744c2f:c349p7127:PIT:3 0 -1
   1) "[B@104dd5c2"
   2) "[B@106e5566"
   3) "[B@107aef31"
   4) "[B@107e3da6"
   5) "[B@10b9387f"
   6) "[B@10d3c559"
   7) "[B@10d3f7c2"
   8) "[B@10f3b456"
   9) "[B@113c9a94"

from scala-redis.

noahlz avatar noahlz commented on July 28, 2024

I am able to partially solve this by encoding our byte array data as Base64 strings. Not exactly an ideal solution, as it increases the size of our data, and our code is now more complicated having to go Object > Bytes (via Kryo serialization) > Base64 String > Redis and then back again.

If we can't update the batchedPipeline implementation to more robustly handle Array[Binary] command arguments for LPUSH and such - what is the most effective way to story binary java objects via that function (if not Base64?).

from scala-redis.

debasishg avatar debasishg commented on July 28, 2024

from scala-redis.

debasishg avatar debasishg commented on July 28, 2024

@noahlz hmm .. Looks like the Format and Parse are not passed properly when sending commands in batch. I will work on this very soon and get back to you.

BTW thanks for the other PR - I merged it.

from scala-redis.

noahlz avatar noahlz commented on July 28, 2024

@debasishg I opened a PR #290 demonstrating some cases where batchedPipeline errors on keys containing spaces or single quotes.

More interesting:

I ran our application with some data sets that don't have spaces or single-quotes in keys. There was no error, but after we lpush a number of Array[Binary] elements the application threads were hanging on this stack:

"pool-25-thread-5" #614 prio=5 os_prio=0 tid=0x00007f6cc0987000 nid=0x29c6 runnable [0x00007f6bdee35000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
        - locked <0x00000007155e0aa8> (a java.io.BufferedInputStream)
        at com.redis.IO.readLine(IO.scala:98)
        at com.redis.IO.readLine$(IO.scala:91)
        at com.redis.Redis.readLine(RedisClient.scala:34)
        at com.redis.Reply.receive(RedisProtocol.scala:136)
        at com.redis.Reply.receive$(RedisProtocol.scala:136)
        at com.redis.Redis.receive(RedisClient.scala:34)
        at com.redis.R.asLong(RedisProtocol.scala:251)
        at com.redis.R.asLong$(RedisProtocol.scala:251)
        at com.redis.Redis.asLong(RedisClient.scala:34)
        at com.redis.ListOperations.$anonfun$lpush$1(ListOperations.scala:10)
        at com.redis.ListOperations$$Lambda$7779/1438318932.apply(Unknown Source)
        at com.redis.RedisClient.$anonfun$batchedPipeline$4(RedisClient.scala:250)
        at com.redis.RedisClient$$Lambda$7755/1478899735.apply(Unknown Source)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:273)
        at scala.collection.TraversableLike$$Lambda$4295/1723452362.apply(Unknown Source)
        at scala.collection.Iterator.foreach(Iterator.scala:943)
        at scala.collection.Iterator.foreach$(Iterator.scala:943)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
        at scala.collection.IterableLike.foreach(IterableLike.scala:74)
        at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
        at scala.collection.TraversableLike.map(TraversableLike.scala:273)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:266)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at com.redis.RedisClient.$anonfun$batchedPipeline$2(RedisClient.scala:250)
        at com.redis.RedisClient$$Lambda$7725/99231071.apply(Unknown Source)
        at com.redis.Redis.send(RedisClient.scala:90)
        at com.redis.Redis.send(RedisClient.scala:93)
        at com.redis.Redis.send(RedisClient.scala:93)
        at com.redis.Redis.send(RedisClient.scala:93)
        at com.redis.Redis.send(RedisClient.scala:93)
        at com.redis.Redis.send(RedisClient.scala:93)
        at com.redis.RedisClient.batchedPipeline(RedisClient.scala:250)

Should I open a different issue for this? This might be more difficult to provide a test case.

from scala-redis.

noahlz avatar noahlz commented on July 28, 2024

Can confirm this issue is resolved in the latest changes to #288! Many thanks!

from scala-redis.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.