I am trying to run the code on Amazons AWS (p2.xlarge). I've adjusted the code to theano 1.0 and run it with:
[global]
device = cpu
floatX = float32
Sadly, this pretty slow so I'd rather use the GPU which is a Nvidia Tesla K80.
However, by doing so I run into this error:
Traceback (most recent call last): File "conv_net_train.py", line 483, in <module> activations=[Sigmoid]) File "conv_net_train.py", line 223, in train_conv_net cost_epoch = train_model(minibatch_index) File "/home/ubuntu/anaconda3/envs/theano_p27/lib/python2.7/site-packages/theano/compile/function_module.py", line 917, in __call__ storage_map=getattr(self.fn, 'storage_map', None)) File "/home/ubuntu/anaconda3/envs/theano_p27/lib/python2.7/site-packages/theano/gof/link.py", line 325, in raise_with_op reraise(exc_type, exc_value, exc_trace) File "/home/ubuntu/anaconda3/envs/theano_p27/lib/python2.7/site-packages/theano/compile/function_module.py", line 903, in __call__ self.fn() if output_subset is None else\ File "pygpu/gpuarray.pyx", line 682, in pygpu.gpuarray.pygpu_zeros File "pygpu/gpuarray.pyx", line 693, in pygpu.gpuarray.pygpu_empty File "pygpu/gpuarray.pyx", line 301, in pygpu.gpuarray.array_empty pygpu.gpuarray.GpuArrayException: cuMemAlloc: CUDA_ERROR_OUT_OF_MEMORY: out of memory Apply node that caused the error: GpuAlloc<None>{memset_0=True}(GpuArrayConstant{[[[[0.]]]]}, Elemwise{Composite{(Switch(LT(i0, i1), Switch(LT((i0 + i2), i3), i4, (i0 + i2)), Switch(LT(i0, i2), i0, i2)) - i5)}}.0, Shape_i{1}.0, Shape_i{2}.0, Shape_i{1}.0) Toposort index: 101 Inputs types: [GpuArrayType<None>(float64, (True, True, True, True)), TensorType(int64, scalar), TensorType(int64, scalar), TensorType(int64, scalar), TensorType(int64, scalar)] Inputs shapes: [(1, 1, 1, 1), (), (), (), ()] Inputs strides: [(8, 8, 8, 8), (), (), (), ()] Inputs values: [gpuarray.array([[[[0.]]]]), array(50), array(312), array(153), array(300)] Outputs clients: [[forall_inplace,gpu,grad_of_scan_fn}(Elemwise{maximum,no_inplace}.0, InplaceGpuDimShuffle{0,x,1}.0, InplaceGpuDimShuffle{0,x,1}.0, GpuAlloc<None>{memset_0=True}.0, GpuSubtensor{int64:int64:int64}.0, GpuAlloc<None>{memset_0=True}.0, GpuAlloc<None>{memset_0=True}.0, GpuAlloc<None>{memset_0=True}.0, GpuAlloc<None>{memset_0=True}.0, GpuAlloc<None>{memset_0=True}.0, GpuAlloc<None>{memset_0=True}.0, W_conv, W_conv, W_conv, GpuFromHost<None>.0, InplaceGpuDimShuffle{x,0,x,x}.0, InplaceGpuDimShuffle{x,0,x,x}.0, InplaceGpuDimShuffle{x,0,x,x}.0, GpuFromHost<None>.0, GpuFromHost<None>.0)]]
Any idea how to resolve this (actually, memory should be enough)?