- Ensemble learning:
- A machine learning technique
- Combines several models to improve overall accuracy
- Random forests or random decision forests
- An ensemble learning method for classification & regression
- operate by constructing a multitude of decision trees at training time
- outputting the class that is the mode of the classes
- Classification
- Regression (mean/average prediction) of the individual trees
- Generally outperform decision trees
- but their accuracy is lower than gradient boosted trees
In [1]:
from IPython.display import Image
from IPython.core.display import HTML
Image(url= "http://datascienceschools.github.io/Machine_Learning/Classification_Models_Intuition/RF1.jpg", width=600)
Out[1]:
In [2]:
from IPython.display import Image
from IPython.core.display import HTML
Image(url= "http://datascienceschools.github.io/Machine_Learning/Classification_Models_Intuition/RF2.jpg", width=600)
Out[2]:
Dr. Ryan @STEMplicity
Overview¶
- Importing the Relevant Libraries
- Loading the Data
- Declaring the Dependent and the Independent variables
- Splitting the dataset into the Training set and Test set
- Feature Scaling
- Training the Random Forest Classification Model
- Predicting the Test Set Results
- Confusion Matrix
- Classification Report
- K-Fold Cross Validation
- Visualising the Training Set Results
- Visualising the Test Set ResultsImporting the Relevant Libraries¶
In [2]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
Loading the Data¶
In [3]:
url = "https://DataScienceSchools.github.io/Machine_Learning/Classification_Models_Intuition/Social_Network_Ads.csv"
df = pd.read_csv(url)
df.head()
Out[3]:
Declaring the Dependent & the Independent Variables¶
In [4]:
X = df.iloc[:, :-1].values
y = df.iloc[:, -1].values
Splitting the Dataset into the Training Set and Test Set¶
In [5]:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, random_state = 0)
Feature Scaling¶
In [6]:
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
Training the Random Forest Classification Model¶
In [7]:
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators = 200, criterion = 'entropy', random_state = 0)
model.fit(X_train, y_train)
Out[7]:
Predicting the Test Set Results¶
In [8]:
y_pred = model.predict(X_test)
Confusion Matrix¶
- A confusion matrix used to describe the performance of a classification model
- TP (True Positive): Model predicted Correctly
- TN (True Negative): Model predicted Correctly
- FP (Flase Positive): Model predicted True but it is actually False
- Type I Error
- Predicting people have cancer, but actually they do not have cancer
- Predictiong earthquake will happen, but it actually does not happen
- FN (False Negative): Model predicted False but it is actually True
- Type II Error -> Life-threatening Error (Must avoid it at all cost)
- Predicting people do not have cancer, but actually they have
- Predictiong earthquake will not happen, but it actually happens
- Accuracy = Correct/Total
- Error Rate = Wrong/Total
In [9]:
from IPython.display import Image
from IPython.core.display import HTML
Image(url= "https://DataScienceSchools.github.io/Machine_Learning/Classification_Models_Intuition/confusionmatrix.jpg", width=400)
Out[9]:
In [10]:
from sklearn.metrics import confusion_matrix, accuracy_score
cm = confusion_matrix(y_test, y_pred)
accuracy = accuracy_score(y_test, y_pred)
print("Accuracy is: {:.2f} %".format(accuracy*100))
sns.heatmap(cm, annot=True, fmt="d")
plt.show()
Classification Report¶
In [11]:
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))
K-Fold Cross Validation¶
- Accuracy of test set is often a misleading metric
- A solution to this problem is a procedure called cross-validation
- k-fold cross-validation is used to evaluate machine learning models
- How the performance measure is calculated by k-fold cross-validation?
1. The training set is split into k smaller sets
1. Each set is used as training data to train the model
2. The remaining part of the data used as a test set to compute the accuracy
3. Then the average of all accuracies is calculated & reportedIn [12]:
from sklearn.model_selection import cross_val_score
accuracies = cross_val_score(estimator = model, X = X_train, y = y_train, cv = 10)
print("Accuracy: {:.2f} %".format(accuracies.mean()*100))
print("Standard Deviation: {:.2f} %".format(accuracies.std()*100))
Visualising the Training Set Results¶
In [14]:
from matplotlib.colors import ListedColormap
X_set, y_set = sc.inverse_transform(X_train), y_train
X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 10, step = 0.25),
np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1000, step = 0.25))
plt.contourf(X1, X2, model.predict(sc.transform(np.array([X1.ravel(), X2.ravel()]).T)).reshape(X1.shape), alpha = 0.75, cmap = ListedColormap(('magenta', 'blue')))
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())
for i, j in enumerate(np.unique(y_set)):
plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1], color = ListedColormap(('magenta', 'blue'))(i), label = j)
plt.title('Random Forest Classification (Training set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()
Visualising the Test Set Results¶
In [13]:
from matplotlib.colors import ListedColormap
X_set, y_set = sc.inverse_transform(X_test), y_test
X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 10, step = 0.25),
np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1000, step = 0.25))
plt.contourf(X1, X2, model.predict(sc.transform(np.array([X1.ravel(), X2.ravel()]).T)).reshape(X1.shape),
alpha = 0.75, cmap = ListedColormap(('magenta', 'blue')))
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())
for i, j in enumerate(np.unique(y_set)):
plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1], color = ListedColormap(('magenta', 'blue'))(i), label = j)
plt.title('Random Forest Classification (Test set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()
