Questions about batch_normalization about morph-net HOT 3 CLOSED

We have not encountered any problems with the beta... When you say they are "very large", could you be more specific? What is the mean and variance with or without MorphNet (possibly using more than one regularization strength).

In our code we are using tf.contrib version, it should behave the same. We never had problems using these settings (which I think are the same as in the paper:

 'batch_norm_params': {
            'decay': 0.9997,
            'scale': True,
            'center': True,
            'epsilon': 1e-5,
            'param_regularizers': {
                'beta_decay': 0.0,
                'gamma_decay': 5e-5
            }
        }

let me know if it helps.

from morph-net.

Sorry, I didn't talk about my question clearly,it's not training problem, the situation is one of the bn with morphnet, the morphnet limits the gamma to very small value (ex. 1e-8, 1e-7), but in the same channel the beta is like 0.86, 0.89(or larger), maybe it's still an important feature(or not), but morphnet cut that channel?
Or should I let my center = false to avoid this problem?
And I also got the question about the bn output, does have any example shows the value of Channel with and without morphnet?Just want to confirm it works, thank you:).
One of my bn beta mean and var with morphnet is -0.099896885 and 0.12298851.

from morph-net.

I would consider this channel to be dead as the beta is not data dependent and can be absorbed in the bias of the next layer (the bias of a filter is not beta but beta * sum(weights_i).

Furthermore, the next layer also has BN so this bias is actually automatically removed.

With that being said, I will be very interested in seeing what happens if you try experiments without beta. For example I have in mind these four experiments:
flop_regularizer = {on, off} x center = {true, false}.
Please do consider reporting back, I think it interesting to see what happens even if the result is the same.

from morph-net.