Use case: Stream arrays from redis store about streamz HOT 14 OPEN

Yes, this is interesting. We'd be happy to chat.

from streamz.

I just added pyarrow. It's blazing fast as expected!

gist

from streamz.

It was fun meeting everyone today. I hope we can continue to talk about some of these issues together. I just profiled the send/recv performance of redis. Checkout https://gist.github.com/d7283b77634f52d6c1afd62c67bfc254. I get about 1GB/s of write performance, and 500 MB/s of read performance using redis-py. I am not sure if this is the best possible, because redis-py might be doing some allocations behind the scenes.

from streamz.

It would be interesting to hear what API you would expect from a low-level streaming library that you would want to exist in order to support your project. Do existing solutions work? If not why not?

from streamz.

Note that the other current user group of this library handles image processing pipelines. You might want to chat with people like @CJ-Wright @danielballan and @ordirules to get a sense of their experience so far.

from streamz.

Yes, this is interesting. We'd be happy to chat.

Did this ever happen? If not, are people free today? I'm in use-case collecting mode for streamz, so I would find this helpful.

from streamz.

I'm also available today.

from streamz.

yep, should be able to make time. @mrocklin can you email us all?

from streamz.

from streamz.

I too am available.

from streamz.

None within your time constraints

from streamz.

I have to start running for the last train by about 4:10 ET. [Otherwise no constraints.]

from streamz.

I've bumped the time to 315 ET so @nbren12 and I can join. Hopefully that works for everyone.

from streamz.

thanks for sharing!

from streamz.