- How K-NN assign the new data point to a category?
- KNN Algorithm Steps:
STEP 1:
- Choose the number of neighbors (K)
- Example: k = 5
STEP 2:
- Calculate the Euclidian distance between the new point and other points
STEP 3:
- Pick the 5 closest data points (5 points with the smallest distances)
STEP 4:
- Among these 5 neighbors, count the number of data points in each category
STEP 5:
- Assign the new data point to the category where you counted the most neighborsOverview¶
- Importing the Relevant Libraries
- Loading the Data
- Declaring the Dependent and the Independent variables
- Splitting the dataset into the Training set and Test set
- Feature Scaling
- Training the K-Nearest Neighbors (K-NN) Model
- Predicting the Test Set Results
- Confusion Matrix
- Classification Report
- k-Fold Cross Validation
- Visualising the Training Set Results
- Visualising the Test Set ResultsImporting the Relevant Libraries¶
In [1]:
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
Loading the Data¶
In [2]:
url = "https://DataScienceSchools.github.io/Machine_Learning/Classification_Models_Intuition/Social_Network_Ads.csv"
df = pd.read_csv(url)
df.head()
Out[2]:
Declaring the Dependent & the Independent Variables¶
In [3]:
X = df.iloc[:, :-1].values
y = df.iloc[:, -1].values
Splitting the Dataset into the Training Set and Test Set¶
In [4]:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, random_state = 0)
Feature Scaling¶
In [5]:
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
Training the K-Nearest Neighbors (K-NN) Model¶
In [6]:
from sklearn.neighbors import KNeighborsClassifier
model = KNeighborsClassifier(n_neighbors = 5, metric = 'minkowski', p = 2)
model.fit(X_train, y_train)
Out[6]:
Predicting the Test Set Results¶
In [7]:
y_pred = model.predict(X_test)
Confusion Matrix¶
- A confusion matrix used to describe the performance of a classification model
- TP (True Positive): Model predicted Correctly
- TN (True Negative): Model predicted Correctly
- FP (Flase Positive): Model predicted True but it is actually False
- Type I Error
- Predicting people have cancer, but actually they do not have cancer
- Predictiong earthquake will happen, but it actually does not happen
- FN (False Negative): Model predicted False but it is actually True
- Type II Error -> Life-threatening Error (Must avoid it at all cost)
- Predicting people do not have cancer, but actually they have
- Predictiong earthquake will not happen, but it actually happens
- Accuracy = Correct/Total
- Error Rate = Wrong/Total
In [8]:
from IPython.display import Image
from IPython.core.display import HTML
Image(url= "https://DataScienceSchools.github.io/Machine_Learning/Classification_Models_Intuition/confusionmatrix.jpg", width=400)
Out[8]:
In [9]:
from sklearn.metrics import confusion_matrix, accuracy_score
cm = confusion_matrix(y_test, y_pred)
accuracy = accuracy_score(y_test, y_pred)
print("Accuracy is: {:.2f}".format(accuracy*100))
sns.heatmap(cm, annot=True,fmt="d")
Out[9]:
Classification Report¶
In [10]:
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))
k-Fold Cross Validation¶
- Accuracy of test set is often a misleading metric
- A solution to this problem is a procedure called cross-validation
- k-fold cross-validation is used to evaluate machine learning models
- How the performance measure is calculated by k-fold cross-validation?
1. The training set is split into k smaller sets
1. Each set is used as training data to train the model
2. The remaining part of the data used as a test set to compute the accuracy
3. Then the average of all accuracies is calculated & reportedIn [11]:
from sklearn.model_selection import cross_val_score
accuracies = cross_val_score(estimator = model, X = X_train, y = y_train, cv = 10)
print("Accuracy: {:.2f} %".format(accuracies.mean()*100))
print("Standard Deviation: {:.2f} %".format(accuracies.std()*100))
Visualising the Training Set Results¶
In [12]:
from matplotlib.colors import ListedColormap
X_set, y_set = X_train, y_train
X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
plt.contourf(X1, X2, model.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),
alpha = 0.75, cmap = ListedColormap(('magenta', 'blue')))
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())
for i, j in enumerate(np.unique(y_set)):
plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],
color = ListedColormap(('magenta', 'blue'))(i), label = j)
plt.title('K-NN (Training set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()
Visualising the Test Set Results¶
In [13]:
from matplotlib.colors import ListedColormap
X_set, y_set = X_test, y_test
X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
plt.contourf(X1, X2, model.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),
alpha = 0.75, cmap = ListedColormap(('magenta', 'blue')))
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())
for i, j in enumerate(np.unique(y_set)):
plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],
color = ListedColormap(('magenta', 'blue'))(i), label = j)
plt.title('K-NN (Test set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()
