Guidance on initializable iterators w/ numpy arrays about io HOT 5 CLOSED

Relates to #68

from io.

It looks like the Iterator class even completely disappeared from the r2.0 API documentation.
As such I would concentrate on creating a dynamic modifiable dataset pipeline. This could be a custom dataset op - or a dataset pipeline using TF variables.
For a good example for the later - take a look at the replay buffer implemented in tensorflow/agent:
https://github.com/tensorflow/agents/blob/master/tf_agents/replay_buffers/tf_uniform_replay_buffer.py
https://github.com/tensorflow/agents/blob/master/tf_agents/replay_buffers/table.py

from io.

Using variables is a good idea, thank you!

from io.

@jperl I think it depends on if your data is already in memory (numpy arrays) or not. If the data is already in numpy arrays then I think we could help implement a Dataset to map the numpy arrays into tensors as data continue to feed in. If you have other sources of input (like a file or a stream), we could also help implement a Dataset to take the input directly, not even necessarily to read into numpy array.

Maybe you could share some details or some boilerplate code to show what the data input format is?

from io.

The data is paths to images stored on a local filesystem. It is queried from a database, and then in memory as an array of strings. After each training loop we requery the database for the latest paths (and other metadata), and reinitialize the dataset.

I believe we can accomplish this similar to the TFUniformReplayBuffer example from @suphoff, using a Counter, map and variables or a generator to provide the latest numpy arrays.

from io.