Possible confusion between dropout rate vs keep_prob rate about deeplearningzerotoall HOT 5 CLOSED

I see. So it's not consistent with https://github.com/hunkim/DeepLearningZeroToAll/blob/master/lab-11-2-mnist_deep_cnn.py#L51

from deeplearningzerotoall.

@sxjscience

@zeran4's issue is not about lowering the keep_prob to 0.3 from 0.7, but it's more like a confusing TensorFlow syntax.

In TensorFlow, there are two functions that perform the dropout

• tf.layers.dropout

  • If you look at the doc, it uses the "dropout" rate
  • it behaves like MXNet

• tf.contrib.layers.dropout (same as tf.nn.dropout)

  • It uses the "keep_prob" rate

By dropout rate, I mean the ratio of nodes to be dropped
By keep_prob, I mean the ratio nodes to be kept

dropout rate = 1 - keep_prob

So,

In lab-11-4-mnist_cnn_layers.py, the rate should be changed to 0.3 from 0.7
because it uses tf.layers.dropout and it uses the dropout rate not the keep_prob

from deeplearningzerotoall.

@zeran4 I've tested these two scripts and in fact a keeping probability of 0.7 is correct. If we change the keeping probability to 0.3, too many nodes will be dropped and the performance will be quite bad.

You can refer to the implementation using MXNet https://github.com/hunkim/DeepLearningZeroToAll/blob/master/mxnet/mxlab-11-2-mnist_deep_cnn.py#L27

One more thing is that "dropout" is usually inserted before the FC layers and it's not standard to insert it before the Conv-Layers. The reason is that Conv-Layers already have local sparse connection structures and are less prone to overfitting. Nevertheless, it's okay to use it in the scripts for demonstration.

from deeplearningzerotoall.

In the file DeepLearningZeroToAll/lab-10-5-mnist_nn_dropout.py, we have this problem too

from deeplearningzerotoall.

finally fixed with #205 #206

from deeplearningzerotoall.