Building Trees using scikit-learn - Lab

Introduction

Following the simple example you saw in the previous lesson, you'll now build a decision tree for a more complex dataset. This lab covers all major areas of standard machine learning practice, from data acquisition to evaluation of results. We'll continue to use the Scikit-learn and Pandas libraries to conduct this analysis, following the same structure we saw in the previous lesson.

Objectives

In this lab you will:

• Use scikit-learn to fit a decision tree classification model
• Use entropy and information gain to identify the best attribute to split on at each node
• Plot a decision tree using Python

UCI Banknote authentication dataset

In this lab, you'll work with a popular dataset for classification called the "UCI Bank note authentication dataset". This data was extracted from images that were taken from genuine and forged banknotes! The notes were first digitized, followed by a numerical transformation using DSP techniques. The final set of engineered features are all continuous in nature, meaning that our dataset consists entirely of floats, with no strings to worry about. If you're curious about how the dataset was created, you can visit the UCI link here!

We have the following attributes in the dataset:

1. Variance of wavelet transformed image (continuous)
2. Skewness of wavelet transformed image (continuous)
3. Curtosis of wavelet transformed image (continuous)
4. Entropy of image (continuous)
5. Class (integer) - Target/Label

Step 1: Import the necessary libraries

We've imported all the necessary modules you will require for this lab, go ahead and run the following cell:

# Import necessary libraries
import numpy as np 
import pandas as pd 
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier 
from sklearn.metrics import accuracy_score, roc_curve, auc
from sklearn.preprocessing import OneHotEncoder
from sklearn import tree

Step 2: Import data

Now, you'll load our dataset in a DataFrame, perform some basic EDA, and get a general feel for the data you'll be working with.

• Import the file 'data_banknote_authentication.csv' as a pandas DataFrame. Note that there is no header information in this dataset
• Assign column names 'Variance', 'Skewness', 'Kurtosis', 'Entropy', and 'Class' to the dataset in the given order
• View the basic statistics and shape of the dataset
• Check for the frequency of positive and negative examples in the target variable

# Create DataFrame

# Describe the dataset

# Shape of dataset

# Class frequency of target variable 

Step 3: Create features, labels, training, and test data

Now we need to create our feature set X and labels y:

• Create X and y by selecting the appropriate columns from the dataset
• Create a 80/20 split on the dataset for training/test. Use random_state=10 for reproducibility

# Create features and labels

# Perform an 80/20 split

Step 4: Train the classifier and make predictions

• Create an instance of a decision tree classifier with random_state=10 for reproducibility
• Fit the training data to the model
• Use the trained model to make predictions with test data

# Train a DT classifier

# Make predictions for test data

Step 5: Check predictive performance

Use different evaluation measures to check the predictive performance of the classifier:

• Check the accuracy, AUC, and create a confusion matrix
• Interpret the results

# Calculate accuracy 
acc = None
print('Accuracy is :{0}'.format(acc))

# Check the AUC for predictions
false_positive_rate, true_positive_rate, thresholds = None
roc_auc = None
print('\nAUC is :{0}'.format(round(roc_auc, 2)))

# Create and print a confusion matrix 
print('\nConfusion Matrix')
print('----------------')

Level up (Optional)

Re-grow the tree using entropy

The default impurity criterion in scikit-learn is the Gini impurity. We can change it to entropy by passing in the argument criterion='entropy' to the classifier in the training phase.

• Create an instance of a decision tree classifier with random_state=10 for reproducibility. Make sure you use entropy to calculate impurity
• Fit this classifier to the training data
• Run the given code to plot the decision tree

# Instantiate and fit a DecisionTreeClassifier
classifier_2 = None

# Plot and show decision tree
plt.figure(figsize=(12,12), dpi=500)
tree.plot_tree(classifier_2, 
               feature_names=X.columns,
               class_names=np.unique(y).astype('str'),
               filled=True, rounded=True)
plt.show()

• We discussed earlier that decision trees are very sensitive to outliers. Try to identify and remove/fix any possible outliers in the dataset.

• Check the distributions of the data. Is there any room for normalization/scaling of the data? Apply these techniques and see if it improves the accuracy score.

Summary

In this lesson, we looked at growing a decision tree for the banknote authentication dataset, which is composed of extracted continuous features from photographic data. We looked at data acquisition, training, prediction, and evaluation. We also looked at growing trees using entropy vs. gini impurity criteria. In following lessons, we shall look at more pre-training tuning techniques for ensuring an optimal classifier for learning and prediction.