Wrong loss for softmax (lab-09-7-sigmoid_back_prop.py) about deeplearningzerotoall HOT 2 CLOSED

Because we chose a "one vs all method" (by choosing a sigmoid instead) instead of softmax.
Since the row sum does not add up to 1.0, we have to take every prediction into account.

For example, y_pred could be even [1.0, 1.0, 1.0].
In softmax, we only need to look at one single probability.

# Suppose there are 3 classes and one sample
target = [1, 0, 0] 
y_pred = [0.9, 0.9, 0.9]

- target * tf.log(y_pred) = -tf.log(0.9)
- (1 - target) * tf.log(1 - y_pred) = -[0, 1, 1] * tf.log([0.1, 0.1, 0.1]) 
                                    = [-tf.log(0.1), -tf.log(0.1)]

And it will be reduced sum later.

We're basically doing logistics regressions multiple times not softmax

The best bet is probably that we should create a softmax version instead.

from deeplearningzerotoall.

Stale issue message

from deeplearningzerotoall.