Case Study (Social Network Ads) :

SKLearrn (Decision Tree Classification)

- Decision tree is a supervised machine learning technique 


    - It splits data according to a certain condition

    - Also referred to as a classification tree or a reduction tree


    - In the tree structures, 

        - leaves represent classifications (also referred to as labels)

        - nonleaf nodes are features

        - branches represent conjunctions of features  


- Overfitting pruning can be used to

        - prevent the tree from being overfitted just for the training set

        - this technique makes the tree general for unlabeled data 

        - can tolerate some mistakenly labeled training data 

        - Pruning is a data compression technique 

        - reduces the size of decision trees 

        - reduces the complexity of the final classifier

        - improves predictive accuracy by the reduction of overfitting

-  Example:

   - whether a customer will click on ads or not based on their savings and age

source1

source2

from IPython.display import Image
from IPython.core.display import HTML 
Image(url= "https://datascienceschools.github.io/Machine_Learning/Classification_Models_Intuition/DT1.jpg", width=600)

from IPython.display import Image
from IPython.core.display import HTML 
Image(url= "https://datascienceschools.github.io/Machine_Learning/Classification_Models_Intuition/DT2.jpg", width=600)

from IPython.display import Image
from IPython.core.display import HTML 
Image(url= "https://datascienceschools.github.io/Machine_Learning/Classification_Models_Intuition/DT3.jpg", width=600)

Dr. Ryan @STEMplicity

Additional Resources

Source1 - Page 250

Source2 - Page 303

Overview

- Importing the Relevant Libraries

- Loading the Data

- Declaring the Dependent and the Independent variables

- Splitting the dataset into the Training set and Test set

- Feature Scaling

- Training the Decision Tree Classification Model

- Predicting the Test Set Results

- Confusion Matrix

- Classification Report

- K-Fold Cross Validation

- Visualising the Training Set Results

- Visualising the Test Set Results

Importing the Relevant Libraries

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

Loading the Data

url = "https://DataScienceSchools.github.io/Machine_Learning/Classification_Models_Intuition/Social_Network_Ads.csv"

df = pd.read_csv(url)

df.head()

	Age	EstimatedSalary	Purchased
0	19	19000	0
1	35	20000	0
2	26	43000	0
3	27	57000	0
4	19	76000	0

Declaring the Dependent & the Independent Variables

X = df.iloc[:, :-1].values

y = df.iloc[:, -1].values

Splitting the Dataset into the Training Set and Test Set

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, random_state = 0)

Feature Scaling

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

X_train = sc.fit_transform(X_train)

X_test = sc.transform(X_test)

Training the Decision Tree Classification Model

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(criterion = 'entropy', random_state = 0)

model.fit(X_train, y_train)

DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='entropy',
                       max_depth=None, max_features=None, max_leaf_nodes=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, presort='deprecated',
                       random_state=0, splitter='best')

Predicting the Test Set Results

y_pred = model.predict(X_test)

Confusion Matrix

 - A confusion matrix used to describe the performance of a classification model

   - TP (True Positive): Model predicted Correctly
   - TN (True Negative): Model predicted Correctly
   - FP (Flase Positive): Model predicted True but it is actually False

       - Type I Error
       - Predicting people have cancer, but actually they do not have cancer
       - Predictiong earthquake will happen, but it actually does not happen

   - FN (False Negative): Model predicted False but it is actually True 

       - Type II Error -> Life-threatening Error (Must avoid it at all cost)
       - Predicting people do not have cancer, but actually they have
       - Predictiong earthquake will not happen, but it actually happens

• Accuracy = Correct/Total
• Error Rate = Wrong/Total

source

from IPython.display import Image
from IPython.core.display import HTML 
Image(url= "https://DataScienceSchools.github.io/Machine_Learning/Classification_Models_Intuition/confusionmatrix.jpg", width=400)

from sklearn.metrics import confusion_matrix, accuracy_score

cm = confusion_matrix(y_test, y_pred)

accuracy = accuracy_score(y_test, y_pred)

print("Accuracy is: {:.2f} %".format(accuracy*100))

sns.heatmap(cm, annot=True, fmt="d")

plt.show()

Accuracy is: 90.00 %

Classification Report

from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred))

              precision    recall  f1-score   support

           0       0.95      0.91      0.93        58
           1       0.79      0.86      0.83        22

    accuracy                           0.90        80
   macro avg       0.87      0.89      0.88        80
weighted avg       0.90      0.90      0.90        80

K-Fold Cross Validation

- Accuracy of test set is often a misleading metric

- A solution to this problem is a procedure called cross-validation

- k-fold cross-validation is used to evaluate machine learning models

- How the performance measure is calculated by k-fold cross-validation? 

    1. The training set is split into k smaller sets  

    1. Each set is used as training data to train the model

    2. The remaining part of the data used as a test set to compute the accuracy 

    3. Then the average of all accuracies is calculated & reported

from sklearn.model_selection import cross_val_score

accuracies = cross_val_score(estimator = model, X = X_train, y = y_train, cv = 10)

print("Accuracy: {:.2f} %".format(accuracies.mean()*100))

print("Standard Deviation: {:.2f} %".format(accuracies.std()*100))

Accuracy: 83.75 %
Standard Deviation: 7.10 %

Visualising the Training Set Results

from matplotlib.colors import ListedColormap

X_set, y_set = sc.inverse_transform(X_train), y_train

X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 10, step = 0.25),
                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1000, step = 0.25))

plt.contourf(X1, X2, model.predict(sc.transform(np.array([X1.ravel(), X2.ravel()]).T)).reshape(X1.shape),
             alpha = 0.75, cmap = ListedColormap(('magenta', 'blue')))
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())

for i, j in enumerate(np.unique(y_set)):
    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1], color = ListedColormap(('magenta', 'blue'))(i), label = j)

plt.title('Decision Tree Classification (Training set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()

Visualising the Test Set Results

from matplotlib.colors import ListedColormap

X_set, y_set = sc.inverse_transform(X_test), y_test

X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 10, step = 0.25),
                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1000, step = 0.25))

plt.contourf(X1, X2, model.predict(sc.transform(np.array([X1.ravel(), X2.ravel()]).T)).reshape(X1.shape),
             alpha = 0.75, cmap = ListedColormap(('magenta', 'blue')))

plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())

for i, j in enumerate(np.unique(y_set)):
    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1], color = ListedColormap(('magenta', 'blue'))(i), label = j)

plt.title('Decision Tree Classification (Test set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()