Restart Iterator from a particular Epoch and Batch about text HOT 5 CLOSED

It should be possible to do this via pickling (which is also what's called by torch.save), although I think you’ll have to use import dill as pickle because there are lambdas stored as attributes. Let us know if this doesn’t work (I can’t remember off the top of my head but this might be impossible because of difficulty pickling running generator-iterators).

from text.

Hmm, because Iterator is an iterable and not an iterator, that object wouldn’t have the state you’d need to pickle here. Try pickling the result of iter() on your iterator and if that's not possible we can potentially introduce a lightweight class implementing the iterator protocol to replace the generator called by iter().

from text.

Thank you!

An idea to avoid Pickle to incorporate into the library would be:
Is the ability to set the iteration number of the iterator. Then we'd need to restructure the iterator so it can start at an arbitrary iteration number.

Another option is instead of using an iterator to implement getitem instead. That's the option OpenNMT-py choose. Then provided the same dataset and iteration getitem retrieves the same batch.

from text.

Currently Iterator and BucketIterator without shuffling work even for datasets that are only provided as a generator (e.g. infinite synthetic datasets or datasets that are too big to load all at once) and I’d like to keep that.

from text.

This should be implemented in #63. Let us know if it doesn't work for you.

from text.