Loss is consistenly NaN about keras-fractalnet HOT 5 CLOSED

Can you paste the exact command line on how are you running it?.
I ran it with Theano and Keras versions from python repository (it was Keras 1.0.6 and Theano 0.8.2 iirc).
I don't think Keras has anything to do with it so it must be something in Theano.

from keras-fractalnet.

Nothing fancy. I'm just running python cifar10_fractal.py in the src directory.

Here's the verbose=1 printout after a few iterations with the default setup:

Using Theano backend.
(50000, 3, 32, 32)
Training network
Train on 50000 samples, validate on 10000 samples
Epoch 1/400
 5200/50000 [==>...........................] - ETA: 485s - loss: nan - acc: 0.1012

from keras-fractalnet.

Ok, I'm able to reproduce the loss=nan issue in my laptop (my desktop is doing other stuff). It seems to be something about the theano_multinomial function because if I change it to a static array the bug goes away. The fastest solution is changing the method to something else, the best solution is trying to figure out when Theano introduced this bug.

I checked the multinomial funcion isolated and it's working in my tests so it can be something about guards or optimizations.

@keskarnitish if you want to run the code now you may want to check with the stable versions of Keras and Theano or try writing another method to replace that functionality.

The libraries I used to reproduce the bug:

Theano: Theano/Theano@74c0233
Keras: 1.0.7
Cuda: 7.5-18
CUDNN: Not available

And the config files for keras and theano:

keras.json:

{
    "image_dim_ordering": "th", 
    "epsilon": 1e-07, 
    "floatx": "float32", 
    "backend": "theano"
}

.theano.rc:

[global]
device = gpu
floatX = float32

[lib]
cnmem = 1

[dnn.conv]
algo_fwd = time_on_shape_change
algo_bwd_filter = time_on_shape_change
algo_bwd_data = time_on_shape_change

from keras-fractalnet.

I'm running other tests with THEANO_FLAGS="mode=DebugMode" and THEANO_FLAGS="mode=NanGuardMode" to try to figure out what's going on.

from keras-fractalnet.

Closing as this was fixed in later version of Theano.

from keras-fractalnet.