Goal is to get 94% accuracy on cifar10 in 2 minutes is the goal.

cifar10 is a dataset of images of size 32x32 belonging to 10 classes.

Uses DavidNet.

They use fenwicks library which is custom written by David.

You can install fenwicks library as :

if tf.gfile.Exists('./fenwicks'):
 tf.gfile.DeleteRecursively('./fenwicks')
!git clone https://github.com/fenwickslab/fenwicks.git

Set up hyperparameters

BATCH_SIZE = 512 #@param ["512", "256", "128"] {type:"raw"}
MOMENTUM = 0.9 #@param ["0.9", "0.95", "0.975"] {type:"raw"}
WEIGHT_DECAY = 0.000125 #@param ["0.000125", "0.00025", "0.0005"] {type:"raw"}
LEARNING_RATE = 0.4 #@param ["0.4", "0.2", "0.1"] {type:"raw"}
EPOCHS = 24 #@param {type:"slider", min:0, max:100, step:1}
WARMUP = 5 #@param {type:"slider", min:0, max:24, step:1}

BUCKET = 'gs://gs_colab'
PROJECT = 'cifar10'

These hyper parameter values are all taken from David net implementation.

However weight decay used here is only quarter of David Net. Article says that weight decay of 0.005

will underfit and the accuracy will come down to 90%, and the architectural difference between TPU and GPU is said to be cause of this.

Now the training time to read 94% is reduced to 186s all on a single V100 GPU.

The next step is to use CuttOut. Take random 8x8 out of the training images along with the standard data augmentation of padding, clipping and random flipping left-right.

Accelerating the learning rate to 30 epochs, 4/5 times it reaches 94% acc in 30 Epochs, along with Increasing the batch size to 768.

The timings for 30epochs run are 161s at batch size 512 and 154s at batch size 768 all on single GPU. Other hyper parameters (momentum=0.9, weight decay=5e-4) are kept at their original values.

https://github.com/davidcpage/cifar10-fast/blob/master/experiments.ipynb - The code to reproduce this.

Here the main goal is to simplify the Architecture and make the network a lot faster.

The above is the current network architecture.

Training this network with accelerated LR schedule, yeilds a 55.9% test acc in 36s.

The problem with this backbone network is downsampling convolutions have a 1x1 kernel and a stride of 2, so rather than enlarging the receptive field of the network, it's just discarding the information. Replacing 1x1 by 3x3 convolutions, test acc improves to a 85.6% time in 36s.

We can further improve the training accuracy by applying 3x3 convolutions with a stride of one followed a max pooling of (2x2). This leads to a test acc of 89.7% in 43s.

The final pooling layer before the classifier is a concatenation of global average pooling and max pooling layers, inherited from the original network.Replace this with a more standard global max pooling layer and double the output dimension of the final convolution to compensate for the reduction in input dimension to the classifier, leading to a final test accuracy of 90.7% in 47s.

Now use two different types of networks

Class A:

Class B:

Now each of these two classes are then seperately classified into seven different categories, based on adding of convolutiosn with bn-relu and position of pooling layers.

Based on brute force search space test, they choose the Residual Network with L1 + L3 which achieves 93.8% test acuracy in 66s for a 20 epoch run. Extending it to 24 epochs gets the training acc to 94% in 7/10 times in just 79s.

It seems the residual networks (Class B) outperform Extra Networks(Class A).

https://colab.research.google.com/github/davidcpage/cifar10-fast/blob/master/batch_norm_post.ipynb - The code for this part

The learnable parameters of network have some functions that produce a constant output ignoring the input.

First order optimizers such as SGD do not like these constraints.

Batch Norm allows the network to train much more rapidly for high learning rates.

As the LR is increased exponentially, the training acc for the batch norm one seems stable, whereas the one with no batch norm is dead.

However there are many drawbacks of Batch Norm.

It is slow.

It is not much effective for smaller batch sizes.

ReLU introduces non-zero mean channels, which leads to them approximating constant functions regardless of the input.

Since ReLU only returs positive values, output channels all have a positive mean. But if we average over the weights, the effect disappears.