mila-iqia / picklable-itertools Goto Github PK

• count
• cycle
• repeat
• chain
• compress
• dropwhile
• groupby
• ifilter
• ifilterfalse
• islice
• imap
• starmap
• tee
• takewhile
• izip
• izip_longest
• product
• permutations
• combinations
• combinations_with_replacement
• accumulate
• file_iterator
• range_iterator
• dict_iterator
• ordered_sequence_iterator

https://docs.python.org/3.4/library/itertools.html#itertools.chain.accumulate

https://docs.python.org/2/library/itertools.html#itertools.cycle

Some (maybe all) of our implementations can be swapped in for faster standard library versions on Python 3. We ought to do this maybe.

https://docs.python.org/3.4/library/itertools.html#itertools.starmap

https://docs.python.org/2/library/itertools.html#itertools.count

Don't know how to get the badges in README.rst back to normal.

See #24

https://docs.python.org/3.4/library/itertools.html#itertools.combinations_with_replacement

https://docs.python.org/3.4/library/itertools.html#itertools.chain.from_iterable

It'd be nice to have a picklable version of the equizip operator I wrote in mila-iqia/blocks#458. For example, the Merge transformer from mila-iqia/fuel#31 could use this (optionally, but by default) so that we can raise errors when a user is trying to zip two datastreams of different length.

Travis, maybe scrutinizer. It'll run pretty infrequently but this package is important enough for Blocks that we'd like it not to break.

https://docs.python.org/3.4/library/itertools.html#itertools.islice

Self-explanatory.

• Reorganize files to clean up namespace
• Docstrings
• Naive implementations of combinations and permutations (based on filtering product, permutations, etc. -- see itertools docs)
• At least one serialization test per public function

https://docs.python.org/3.4/library/itertools.html#itertools.combinations

https://docs.python.org/2.7/library/itertools.html#itertools.izip

Would be great to extend this package to contain itertoolz implementations as well. The one that I can think of that are particularly handy are partition and partition_all, so that returning batches becomes: partition(batch_size, xrange(num_examples)).

https://docs.python.org/3.4/library/itertools.html#itertools.tee

https://docs.python.org/3.4/library/itertools.html#itertools.product

https://docs.python.org/2/library/itertools.html#itertools.groupby

https://docs.python.org/2/library/itertools.html#itertools.dropwhile

https://docs.python.org/3.4/library/itertools.html#itertools.permutations

rangeiterator objects aren't picklable either, but a custom iterator should be simple enough.

@dwf Wondering if you consider this within the scope of the framework.

File handles are iterators, and they can't be pickled.

>>> f = open('README.md')
>>> f
<open file 'README.md', mode 'r' at 0x7f13b1360d20>
>>> it = iter(f)
>>> it
<open file 'README.md', mode 'r' at 0x7f13b1360d20>
>>> next(it)
'picklable_itertools\n'
>>> cPickle.dumps(it)
TypeError: can't pickle file objects

Analogously to ordered_sequence_iterator, can I add a file_iterator? It will be a bit more involved, because it needs to handle a variety of things through custom __getstate__ and __setstate__ methods:

• Upon pickling, close and remove file handle
• When unpickling, re-open the file, re-iterate over the lines until we get back to where we were
• Have sensible defaults for if the file is not available in the unpickled environment, is no longer writable, etc. Dill implements all of this, so we can steal a lot of the behaviour from there.

https://docs.python.org/2/library/itertools.html#itertools.takewhile

a.k.a. filterfalse

https://docs.python.org/3.4/library/itertools.html#itertools.filterfalse