Build a Traffic Sign Recognition Project

The goals / steps of this project are the following:

• Load the data set (see below for links to the project data set)
• Explore, summarize and visualize the data set
• Design, train and test a model architecture
• Use the model to make predictions on new images
• Analyze the softmax probabilities of the new images
• Summarize the results with a written report

You're reading it! and here is a link to my project code

I used the numpy library to calculate summary statistics of the traffic signs data set:

• The size of training set is 34799
• The size of the validation set is 4410
• The size of test set is 12630
• The shape of a traffic sign image is (32, 32, 3)
• The number of unique classes/labels in the data set is 43

Here is an exploratory visualization of the data set.

First, I plotted a sample image from each class.

Next, I plotted a bar chart showing how the data is distributed across different classes. It shows the total count of images from each class.

Here we can clearly see that some classes were having less number of images when compared to others.

My preprocessing pipeline consisted of the following steps:

1. Grayscale Conversion.
2. Histogram Equalization.
3. Normalization.

As a first step, I converted the color image to grayscale using OpenCV library. This was done to reduce the complexity and to speed up the pre-processing.

Next, I applied histogram equalization using OpenCV library. I preferred the CLAHE (Contrast Limited Adaptive Histogram Equalization) method to equalize the images. This helped in preserving the foreground details while increasing the contrast of the bakground.

Here is an example of a traffic sign image before and after grayscaling plus histogram equalization. We can clearly see that CLAHE version performs better in preserving the foreground details.

As a last step, I normalized the image data because data normalization is an important step which ensures that each input parameter (pixel, in this case) has a similar data distribution. This makes convergence faster while training the network. I have tried different normalization medthods and picked the one which had zero mean and unit variance.

Here is the comparison of the different normalization techniques that were tried:

Here, I plotted a sample image from each class after preprocessing.

I decided to generate additional data because it helped in increasing the training accuracy of the neural network by allowing it to train on an extended dataset when the existing numbers of some classes were relatively low. I augmented the existing data-set with perturbed versions of the existing images. This is done to expose the neural network to a wide variety of variations. This makes it less likely that the neural network recognizes unwanted characteristics in the data-set.

To add more data to the the data set, I used the following techniques:

• Scaling
• Translation
• Rotation

Here is an example where a set of augmented images were produced from a single orignal image:

I had made sure that each class in the dataset contains atleast 1400 images by augmenting the classes which had lesser number of images.

The size of the training set was increased from 34799 to 63880.

My final model consisted of the following layers:

To train the model, I used AdamOptimizer, a batch size of 128, 50 epochs, a learning rate of 0.001 and dropout rate of 0.5 while training. I have tried changing the optimizer to GradientDescentOptimizer but it perfomed much worse.

My final model results were:

• training set accuracy of 1.000
• validation set accuracy of 0.976
• test set accuracy of 0.963

The code for calculating accuracies on my final model is located in the 36th cell of the Ipython notebook.

I have started with the LeNet neural network which I have trained in the LeNet lab as part of the course work and made minor changes to the input and output layers. It is as follows:

1. Input: 32x32x3 for color and 32x32x1 for grayscale.
2. Layer 1: Convolutional. The output shape should be 28x28x6.
3. Activation. Relu.
4. Pooling. The output shape should be 14x14x6.
5. Layer 2: Convolutional. The output shape should be 10x10x16.
6. Activation. Relu.
7. Pooling. The output shape should be 5x5x16.
8. Flatten. 400
9. Layer 3: Fully Connected. This should have 120 outputs.
10. Activation. Relu.
11. Layer 4: Fully Connected. This should have 84 outputs.
12. Activation. Relu.
13. Layer 5: Fully Connected (Logits). This should have 43 outputs.
14. Output: 43.

This gave me an accuracy of 89% out of the box using above architecture. I initally started with RGB images and couldn't get better accuracy so I then moved to grayscale images. Then I applied histogram equalization and normalization and could get the accuracy to 91% to 92%. I've augmented the training data set and it increased the accuracy to 94% to 95% but my test accuracy was not high as expected. So then I introduced dropout layer for fully connected layers and it reduced the validation accuracy a bit. I have played with different learning rates and dropout probabilities. I have even tried to apply L2 regularization but found no improvements. Finally I increased the number of filters in the convolutional layers and it worked as a charm and produced excellent results. I got around 100% training accuracy, 98% validation accuracy and 96.5% testing accuracy. The presence of dropout layers prevented the network from overfitting.

The LeNet architecture was chosen because it is simple and the training dataset consisted of 32x32 images as its input and it performed really well on the MNIST dataset in the LeNet lab. With minimal modifications I was able to get a better testing accuracy using LeNet.

Here are the German traffic signs that I found on the web:

All of the images got plenty of features and should be easily classified. I'm suspecting little difficulty on the last image due to less difference in contrast between it's foreground and background.

Here are the results of the prediction:

The model was able to correctly guess all the traffic signs, which gives an accuracy of 100%. This compares favorably to the accuracy on the test set of 96.3%.

The code for making predictions on my final model is located in the 40th cell of the Ipython notebook.

For the first image, the model is relatively sure that this is a Children crossing sign (probability of 99.99%), and the image does contain a Children crossing sign. The top five soft max probabilities were

For the second image, the model is exactly sure that this is a No entry sign (probability of 100%), and the image does contain a No entry sign. The top five soft max probabilities were

For the third image, the model is exactly sure that this is a Right-of-way at the next intersection sign (probability of 100%), and the image does contain a Right-of-way at the next intersection sign. The top five soft max probabilities were

For the fourth image, the model is relatively sure that this is a Road work sign (probability of 99.79%), and the image does contain a Road work sign. The top five soft max probabilities were

For the fifth image, the model is exactly sure that this is a Roundabout mandatory sign (probability of 100%), and the image does contain a Roundabout mandatory sign. The top five soft max probabilities were

For the sixth image, the model is exactly sure that this is a Speed limit (50km/h) sign (probability of 100%), and the image does contain a Speed limit (50km/h) sign. The top five soft max probabilities were

For the seventh image, the model is exactly sure that this is a Stop sign (probability of 100%), and the image does contain a Stop sign. The top five soft max probabilities were

For the eighth image, the model is relatively sure that this is a Turn right ahead sign (probability of 99.99%), and the image does contain a Turn right ahead sign. The top five soft max probabilities were

Considering the below image as input:

The visulaization of the output of the activations of the first convolutional layer is as follows:

We can clearly see that different feature maps look for different features. The ninth featuremap is activated by inside horizontal edges while the tenth featuremap is activated by outside curves.

I couldn't quite understand the output of the activations of the second convolutional layer.