antonyo314 / batch_normalization Goto Github PK

Theoretical part

Batch normalization is a technique for improving the speed, performance, and stability of neural networks. 
batch normalization solves the problem of internal covariate shift (During the training stage of networks, as the parameters of the preceding layers change, the distribution of inputs to the current layer changes accordingly, such that the current layer needs to constantly readjust to new distributions)

Batch normalization has a slight regularization effect
But the main goal of batch normalization is normalize your hidden units activations and therefore speed up learning. The regularization is an almost unintended side effect.
Also batch normalization allows us to use much higher learning rates and be less careful about initialization. 
Batch normalization normalizes the output of a previous activation layer by subtracting the batch mean and dividing by the batch standard deviation.
However, after this shift/scale of activation outputs by some randomly initialized parameters, the weights in the next layer are no longer optimal. SGD ( Stochastic gradient descent) undoes this normalization if itÕs a way for it to minimize the loss function.
Consequently, batch normalization adds two trainable parameters (scale and shift) to each layer, so the normalized output is multiplied by a Òstandard deviationÓ parameter (scale) and add a ÒmeanÓ parameter (shift). In other words, batch normalization lets SGD do the denormalization by changing only these two weights for each activation, instead of losing the stability of the network by changing all the weights.

Practical part: 

I wrote custom layer for batch normalization and became convinced of its fullness.
I saw a very strong improving of neural network training speed.


To run my code use this command:
python3 train_and_evaluate.py