Class Imbalance Problems - Lab
Introduction
Now that you've gone over some techniques for tuning classification models on imbalanced datasets, it's time to practice those techniques. In this lab, you'll investigate credit card fraud and attempt to tune a model to flag suspicious activity.
Objectives
You will be able to:
- Use sampling techniques to address a class imbalance problem within a dataset
- Create a visualization of ROC curves and use it to assess a model
Predicting credit card fraud
The following cell loads all the functions you will be using in this lab. All you need to do is run it:
import pandas as pd
import numpy as np
import itertools
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_curve, auc
from sklearn.metrics import confusion_matrix, plot_confusion_matrix
from imblearn.over_sampling import SMOTE, ADASYN
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inlineUse Pandas to load the compressed CSV file, 'creditcard.csv.gz'.
Note: You need to pass an additional argument (
compression='gzip') to read_csv() in order to load compressed CSV files.
# Load a compressed csv file
df = None
# Print the first five rows of dataPreview the class imbalance
Did you notice that the dataset has 31 columns? The first is a time field followed by columns V1 - V28, created by way of manual feature engineering done on the backend that we have little information about. Finally, there's the amount of the purchase and a binary 'Class' flag. This last column, 'Class', is the indication of whether or not the purchase was fraudulent, and it is what you should be attempting to predict.
Take a look at how imbalanced this dataset is:
# Count the number of fraudulent/infraudulent purchasesDefine the predictor and target variables
Define X and y and perform a standard train-test split. Assign 25% to the test set and random_state to 0.
# Your code here
y = None
X = None
X_train, X_test, y_train, y_test = NoneFind the class imbalance in the training and test sets:
# Training set
print('\n')
# Test setCreate an initial model
As a baseline, train a vanilla logistic regression model. Then plot the ROC curve and print out the AUC. We'll use this as a comparison for how our future models perform.
# Initial Model
logreg = None
# Probability scores for test set
y_score = None
# False positive rate and true positive rate
fpr, tpr, thresholds = None
# Seaborn's beautiful styling
sns.set_style('darkgrid', {'axes.facecolor': '0.9'})
# Print AUC
# Plot the ROC curve
plt.figure(figsize=(10, 8))Use scikit-learn's plot_confusion_matrix function to plot the confusion matrix of the test set:
# Plot confusion matrix of the test set Tune the model
Try some of the various techniques proposed to tune your model. Compare your models using AUC and ROC curve.
# Now let's compare a few different regularization performances on the dataset:
C_param_range = [0.001, 0.01, 0.1, 1, 10, 100]
names = [0.001, 0.01, 0.1, 1, 10, 100]
colors = sns.color_palette('Set2')
plt.figure(figsize=(10, 8))
for n, c in enumerate(C_param_range):
# Fit a model
logreg = None
model_log = None
print(model_log) # Preview model params
# Predict
y_hat_test = None
y_score = None
fpr, tpr, thresholds = None
print('AUC for {}: {}'.format(names[n], auc(fpr, tpr)))
print('-------------------------------------------------------')
lw = 2
plt.plot(fpr, tpr, color=colors[n],
lw=lw, label='ROC curve Normalization Weight: {}'.format(names[n]))
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.yticks([i/20.0 for i in range(21)])
plt.xticks([i/20.0 for i in range(21)])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic (ROC) Curve')
plt.legend(loc='lower right')
plt.show()SMOTE
Use the SMOTE class from the imblearn package in order to improve the model's performance on the minority class.
# Previous original class distribution
print(y_train.value_counts())
# Fit SMOTE to training data
X_train_resampled, y_train_resampled = None
# Preview synthetic sample class distribution
print('\n')
print(pd.Series(y_train_resampled).value_counts()) Similar to what you did above, build models with this resampled training data:
# Now let's compare a few different regularization performances on the dataset
C_param_range = [0.005, 0.1, 0.2, 0.5, 0.8, 1, 1.25, 1.5, 2]
names = [0.005, 0.1, 0.2, 0.5, 0.8, 1, 1.25, 1.5, 2]
colors = sns.color_palette('Set2', n_colors=len(names))
plt.figure(figsize=(10, 8))
# Write a for loop that builds models for each value of C_param_range, prints the AUC and plots the ROC
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.yticks([i/20.0 for i in range(21)])
plt.xticks([i/20.0 for i in range(21)])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic (ROC) Curve')
plt.legend(loc='lower right')
plt.show()Something wrong here?
Describe what is misleading about the AUC score and ROC curves produced by this code:
# Previous original class distribution
print(y.value_counts())
X_resampled, y_resampled = SMOTE().fit_sample(X, y)
# Preview synthetic sample class distribution
print('---------------------------------')
print(pd.Series(y_resampled).value_counts())
# Split resampled data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled, random_state=0)
# Now let's compare a few different regularization performances on the dataset:
C_param_range = [0.005, 0.1, 0.2, 0.3, 0.5, 0.6, 0.7, 0.8]
names = [0.005, 0.1, 0.2, 0.3, 0.5, 0.6, 0.7, 0.8, 0.9]
colors = sns.color_palette('Set2', n_colors=len(names))
plt.figure(figsize=(10, 8))
for n, c in enumerate(C_param_range):
# Fit a model
logreg = LogisticRegression(fit_intercept=False, C=c, solver='liblinear')
model_log = logreg.fit(X_train, y_train)
# Predict
y_hat_test = logreg.predict(X_test)
y_score = logreg.fit(X_train, y_train).decision_function(X_test)
fpr, tpr, thresholds = roc_curve(y_test, y_score)
print('----------------------------------------------')
print('AUC for {}: {}'.format(names[n], auc(fpr, tpr)))
lw = 2
plt.plot(fpr, tpr, color=colors[n],
lw=lw, label='ROC curve Normalization Weight: {}'.format(names[n]))
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.yticks([i/20.0 for i in range(21)])
plt.xticks([i/20.0 for i in range(21)])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic (ROC) Curve')
plt.legend(loc='lower right')
plt.show()Your response here
Summary
In this lab, you got some hands-on practice tuning logistic regression models. In the upcoming labs and lessons, you will continue to dig into the underlying mathematics of logistic regression, taking on a statistical point of view and providing you with a deeper understanding of how the algorithm works. This should give you further insight as to how to tune and apply these models going forward.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent