keras-mode-normalization's Introduction

Mode Normalization (Keras)

ModeNormalization(axis=-1, k=2, momentum=0.99, epsilon=1e-3, center=True, scale=True, beta_initializer='zeros', gamma_initializer='ones', moving_mean_initializer='zeros', moving_variance_initializer='ones', beta_regularizer=None, gamma_regularizer=None, beta_constraint=None, gamma_constraint=None)

Mode Normalization Lucas Deecke, Iain Murray, Hakan Bilen - 2018

Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1 for K different modes.

Arguments

axis: Integer, the axis that should be normalized (typically the features axis). For instance, after a Conv2D layer with data_format="channels_first", set axis=1 in ModeNormalization.
k: Integer, the number of modes of the normalization.
momentum: Momentum for the moving mean and the moving variance.
epsilon: Small float added to variance to avoid dividing by zero.
center: If True, add offset of beta to normalized tensor. If False, beta is ignored.
scale: If True, multiply by gamma. If False, gamma is not used. When the next layer is linear (also e.g. nn.relu), this can be disabled since the scaling will be done by the next layer.
beta_initializer: Initializer for the beta weight.
gamma_initializer: Initializer for the gamma weight.
moving_mean_initializer: Initializer for the moving mean.
moving_variance_initializer: Initializer for the moving variance.
beta_regularizer: Optional regularizer for the beta weight.
gamma_regularizer: Optional regularizer for the gamma weight.
beta_constraint: Optional constraint for the beta weight.
gamma_constraint: Optional constraint for the gamma weight.

Input shape

Arbitrary. Use the keyword argument input_shape (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.

Output shape

Same shape as input.

References

Mode Normalization

SVHN dataset

https://drive.google.com/file/d/1O4LIHC1ttSeE6HM0haymPeao4nkTFW8y/view?usp=sharing

Run the tests

pytest

keras-mode-normalization's People

Contributors

Stargazers

Watchers

keras-mode-normalization's Issues

Change the gating mechanism

Hey Lucas! Thanks for the great paper
My first question is related to the number of parameters contained in the mode norm layer.
Especially on the gating mechanism.
We have a function psi : X -> [0,1]^k
where X = W . H . C
To my understanding, it translates to
Flatten() # to make it WHC
Dense(1, activation='softmax')
That's the case for K=1
Question: I can have thousands and thousands of learnable parameters in the layer if K >> 1 and W . H . C >> 1. Is this correct?
Yeah, stemming from how normalizations are formalized in the literature this psi business does get a tad bit confusing, I realize we could possibly have done more to clarify it.
So we never go from the full image to K, and instead only apply psi after averaging out H and W. Doing that reduces the number of learnable parameters considerably as the resulting objects lie in some C-dimensional space, and in a medium-sized ResNet the channel number never exceeds 256 or 512 or so.

Recommend Projects