Randomizing Batches in training gives bad predictions though validation errors are much less about nolearn HOT 12 CLOSED

What's happening here is that you're probably also shuffling the batches of your test set iterator (batch_iterator_test). When you run net.predict() it'll also use an iterator to go over the samples, so in your case they'll come out in this random order.

from nolearn.

Daniel
I am unaware of batch_iterator_test. Like I said I only changed those three lines in BatchIterator. And I am not sure I understand what you mean. Why would that cause extremely bad test performance when the validation errors are less. Why would the prediction have dependency on the order of the batches ?

from nolearn.

@run2 So if you're running the version that doesn't have batch_iterator_test (which was added three weeks ago), then put a if self.test around your shuffling logic, so it won't shuffle if it's in test mode.

Why would the prediction depend on the order of the batches? Because that order defines the order of predictions on the way out. So imagine you want to predict for values A, B, and C. After your batch iterator's shuffle (assume batch size of 1), the predictions might come out as b, c, and a. If this doesn't make sense, then I suggest you take a closer look at the predict_proba method.

from nolearn.

Ah! I see. Very interesting. Thanks - let me try that out.

from nolearn.

Wow! That makes a big difference now. Much much better prediction results. You might want to try out what I did and see if that gives you much faster learning and lower val errors. It will be great if you can let me know.

from nolearn.

I am closing this issue but waiting to hear from you.

from nolearn.

I think this is a fine idea. FTR, here's some code that I use in another project:

def shuffle(*arrays):
    p = np.random.permutation(len(arrays[0]))
    return [array[p] for array in arrays]


class ShufflingBatchIteratorMixin(object):
    def __iter__(self):
        self.X, self.y = shuffle(self.X, self.y)
        for res in super(ShufflingBatchIteratorMixin, self).__iter__():
            yield res

from nolearn.

@dnouri For the reasons listed here: #36, there are limitations to this approach.

from nolearn.

@cancan101 Not sure I get it. Is nn.enc_ relevant for shuffling?

from nolearn.

I am using is not for shuffling, but rather for subsampling one of the classes.

from nolearn.

So you suggest #36 as an alternative to the code in #13, is that right?

from nolearn.

Ah, yes, #36 instead of #13 (specifically cancan101@00c2ce9). In looking at your code above, I think that is a much cleaner solution and doesn't require the change in #13 and subclassing of NeuralNet. I would be happy to use change/ subclass the Iterator but currently I do need a reference to the NeuralNet.

from nolearn.