Instructions to train a model using train.py

• Install the required libraries. Do the following in your command line.

     pip install numpy
     
     pip install wandb
     
     pip install keras
     
     pip install argparse
  

• Go to the directory where train.py is located.

• Run the following command, if you want to sync the run to the cloud: wandb online. It is not necessary to run the script but recommended

• Do python train.py --wandb_entity ENTITY_NAME --wandb_project PROJECT_NAME to run the script, where ENTITY_NAME & PROJECT_NAME is your entity name and proejct name. Currently, the default is set to mine.

• train.py can handle different arguments. The defualts are set to the hyperparameters which gave me the best validation accuracy.

Arguments supported are:

train.py returns the wandb logs generated on Training and Validation dataset. Also, The Real train and test accuracy are printed out.

An example, if you want to train the model on mnist dataset with same configurations.

Run the following command: python train.py --wandb_entity ENTITY_NAME --wandb_project PROJECT_NAME -d mnist

Sone google collab files have been uploaded as well.

• Sample_images_of_classes.ipynb has been used to log the different classes image.

• Final_FNN.ipynb has the whole code, run the cells to train the model, hyperparameter tuning using wandb sweep can be done here, confusion matrix plot have been generated from here. The Cross-Entropy loss & MSE loss comparison plot is also there.

• The old folder contains the older versions of code and MSE_loss.ipynb where I ran sweeps for configurations with MSE loss

Remember to change the name of the entity and project in the code.

The code doesn't make use of NN frameworks like keras or tensorflow. The Neural Network framewrok has been made from scratch using Numpy. The code works only for classification tasks and by default assumes that the activation function for the last layer is softmax. This was done for simplicity.

Function & Classes defined in train.py

• Generate confusion matrix plot, it has been commented out, if requried can be added. To generate plot you have to install matplotlib and seaborn as well

• Activation functions & their derivatives: sigmoid, tanh, relu, identity & softmax

• loss_computation: To compute cross-entropy loss & mse loss

• NeuralNetwork Class

  Functions

  1. weight_bias_initialize: To initialize weight & bias using Xavier/random initialization

  2. forward_propagation: Forward Propagation in Neural Network

  3. backpropagate: Back Propagation to calculate gradients of loss wrt to weight and biases

  4. predict: Predict final class labels

  5. accuracy: Accuracy of the model

  6. loss_plot" To plot loss curve (Not used in train.py but used in .ipynb files)

• NN_optimizers Class: It has functions for Stochastic Gradient descent, Momentum based Gradient Descent, Nesterov Gradient Descent, RMSprop, Adam & NAdam

• early_stopping function

• main takes the argparse arguments.

  1. Data pre-processing is done here, which splits the data to validation data & training data, one hot encodes the class label.

  2. NN_fit_modified function is used to train the model

     Parameters:

         train_data: Training Data
         
         train_labels: Class labels corresponding to training data
         
         test_data: Validation data/ Test data
         
         test_labels: Class labels corresponding to Validation data/ Test data
         
         test: Boolean, default **False**, if set to false Validation is done, if set to True, model is trained on full X and accuracy is given for X_test
         
         patience: required for early stopping, defualt set to 5. If test==True, early stopping is not done
     

     NN_fit_modified framework:

     

The best test accuracy obtained on Fashion-mnist dataset was 88% and 98.24% on mnist dataset with following configurations.