In this practice, you will train a simple neural network classifier and play with many training tricks.

Before we start this practice, you need to understand how PyTorch framework (tensor, gradient, network, loss function, optimizer) works. Please refer to Deep Learning with PyTorch: A 60 Minute Blitz and examples.

• python 3.6.9
• CUDA 10.2
• Pytorch 1.4.0
• numpy 1.18.3
• opencv 4.1.2

pip3 install torch
pip3 install torchvision  

The CIFAR-100 dataset consists of 60000 32x32 colour images in 100 classes, with 600 images per class. There are 50000 training images and 10000 test images.

You can simply download the data with the torchvision API

from torchvision import datasets, transforms

BATCH_SIZE = 64

train_loader = torch.utils.data.DataLoader(
    datasets.CIFAR100('./data', train=True, download=True, transform=transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
    ])), 
    batch_size=BATCH_SIZE, shuffle=True)

test_loader = torch.utils.data.DataLoader(
    datasets.CIFAR100('./data', train=False, transform=transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
    ])), 
    batch_size=BATCH_SIZE, shuffle=True)

Let’s train different models for recognizing CIFAR-100 classes! Please compare the convergence time and test accuracy.

1. Build a softmax classification model with a single linear layer using stochastic gradient descent (SGD).

2. Build a 1-hidden layer neural network with 1024 ReLU units using SGD. This model should improve your test accuracy.

3. Try to get better performance by adding more layers and using learning rate decay.

4. Build a convolutional neural network with two convolutional layers, followed by one fully connected layer using Adam optimizer. (Both Conv1 and Conv2: 16@5x5 filters at stride 2)

5. Now, please replace the strides by a max pooling operation of stride 2, kernel size 2.

6. Apply dropout to the hidden layer of your models. Note that dropout should only be introduced during training, not evaluation.

7. Load ResNet18 pre-trained model and finetune on the CIFAR-100 dataset.

8. Train ResNet18 from scratch and compare the result to problem 7.

[Optional]

7. Apply batch normalization to your models.

8. Replace the ReLU units in your models by LeakyReLU or SELU.

It is important to save your model at any time, especially when you want to reproduce your results or contiune the training procedure. One can easily save the model and the parapeters by using the save/load functions. While please also note that when you need to resume training, you should follow this example to save all required state dictionaries.

Now please save the model which achieves best performance among the above variants. Try to reproduce your results using the save/load functions instead of running a new training procedure.

If you have time, you can use the technique of transfer learning to achieve better performance of semantic segmentation. Detailed discription is in Segmentation Practice.