The project is to train a deep neural network to classify images on MNIST and EMNIST datasets.

Summary of Accuracy on MNIST

• training accuracy: 99.92%
• validation accuracy: 99.60%
• test accuracy: 99.48%

Summary of Accuracy on EMNIST

• training accuracy: 98.7%
• validation accuracy: 95.1%
• test accuracy: 94.9%

• Tensorflow
• Keras
• Sklearn
• scipy
• numpy
• cv2
• matplotlib

run "image-classification-mnist-emist.ipynb"

When training the neural network, during the backpropogation in each subsequent layer, the gradient in earlier layers get exponientially smaller and smaller (vanishing), which makes the network learning very little/slow to improve.

Potential Solutions include:

• Batch normalization, which is to normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1 to reduce the oscillations of in the distribution of activation. Deep network can be trained faster and better when the activation is normalized during backpropogation.
• Use ReLu activation will normally solve the vanishing gradient problem (not sigmoid).
• Using Regularization to modify cost function to penalize larger weights.
• Use different weight initialization, mementum, etc.
• Use more data.
• Modify architecture.

• Michael Nielsen's Neural Network and Deep Learning
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Stanford course on Convolutional Neural Networks for Visual Recognition