ENH: inverse of zip (split?) about streamz HOT 11 CLOSED

from streamz.

I am not familiar with pluck, what is it?

from streamz.

part of toolz lib:
http://toolz.readthedocs.io/en/latest/api.html#toolz.itertoolz.pluck

you can map

s = Stream()
s_first = s.map(partial(pluck, [0]))
s_second = s.map(partial(pluck, [1]))
s.emit([1,2])
# ...

(I knew of pluck but not use case, wondering if this is the line of thinking)

from streamz.

Cool. I think this would work.

from streamz.

Sorry, I assumed that pluck was already part of the API. It seems like a decently common use case and so possibly merits being added as syntactic sugar.

from streamz.

I don't understand the API. I don't know ahead of time how many streams might have been zipped. I need to take these zipped streams and zip more streams, but zipping zipped streams doesn't work for what I'm trying to do. I want something like:

streams_uzip = streamz.unzip(streams_zip)
streams_zip_out = streamz.zip(*streams_uzip, another_stream)
return streams_zip_out

from streamz.

Why not provide the constituent streams to create streamz_zip_out? At some point they were all independent. You could also write a function that unpacks the zipped stream so that all the elements are at the same level in the hierarchy. If that was done right after the zipping you could be confident in how many levels you need to go down.

from streamz.

I don't have the constituent streams. It's a dynamic framework. I know that some zipped streams are passed into a function. I don't know how many have been zipped.

Yes, I need to delve into the API to understand how to unpack the zipped stream. That's what an unzip would be. Can you give an example of this function?

def unzip(zipped_stream):
    if not is_zipped_stream(zipped_stream):
        raise Exception('Not a zipped stream')
    . 
    .
    .
   return streams_uzip 

Basically that's what I need. I don't have any information about how the zipped_stream was formed, just that it's a zip of streams.
A bonus would be some ability to check that this is a zipped stream i.e. how do I achieve is_zipped_stream.

from streamz.

The problem is that it may be difficult to determine how many levels of tuple hierarchy you need to unpack. Was an inner tuple the product of a zip or were the original data elements a tuple.

from streamz.

Each stream is a tuple or array of numbers. Say I have a stream zip of two streams that in a sink would appear as:

(array1, array2)

I want to zip it to another stream so that in a sink it would be:

(array1, array2, array3)

Without ability to unzip and re-zip it appears something like this:

((array1, array2), array3)

Now it could be more than two streams and I might need to zip up more streams. Sure, if I had all the original streams I could formulate this:

# list of length N stream_list with all original streams
stream_zip = streamz.zip(*stream_list)

# this stream_zip would appear as
(array1, array2, array3, ..., arrayN)

But I don't have all the streams so instead I get this mess:

stream_zip = streamz.zip(some_zipped_stream, another_stream)

# this will end up with a bunch of nested tuples of arrays
(...( ( (array1, array2), array3), array4), array5) . . .)

I want this:

streams_uzip_list = streamz.unzip(some_zipped_stream)
stream_zip = streamz.zip(*streams_uzip_list, another_stream)
# this stream_zip would appear as
(array1, array2, array3, ..., arrayN)

Thanks

from streamz.

With an unzip API I would always insure that there's only one level of tuple hierarchy. I don't want to create nested tuples. If someone were to pass in a nested tuple that would not be my problem. I just need to make sure not to produce nested tuple. So if I get something like this:

((array1, array2), array3)

Unzip/re-zip it so that I would produce:

((array1, array2), array3, array4)

I just don't want to produce this:

(((array1, array2), array3), array4)

from streamz.