What if the training dataset is too large to fit into memory? about nolearn HOT 5 CLOSED

@paipai880429 One way is to call fit() multiple times. Another one is to only put ids (or filenames) into X and then load the actual data based on these ids from disk in your batch iterator.

from nolearn.

@dnouri Thank you for your timely reply. It seems that the former is more straightforward. By the way, the document for nolearn in the index page(https://pythonhosted.org/nolearn/) seems not a detailed and latest version. Will any plans for a document update ?

from nolearn.

Yes, you're right. nolearn.lasagne sadly has no documentation yet. I'll be hopefully working on that soon.

from nolearn.

@dnouri Dainel, would you please explain Another one is to only put ids (or filenames) into X and then load the actual data based on these ids from disk in your batch iterator. in details? Are you suggesting overriding BatchIterator parameterized with ```file ID list`?

from nolearn.

@dnouri if we call fit() multiple times for training, Then after after completion of 1 epoch say in 1epoch we had to call fit() 1000 times, after completion of epoch How to check it's accuracy given we have an external parameter X_val, y_val and we don't want to recreate validation set 20%,Please clarify on this, even i have external params X_val, y_val it is printing nan on valid loss
since I have use
train_split=TrainSplit(eval_size=0.0)

from nolearn.