Importing the relevant libraries¶
In [53]:
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
Loading data¶
In [54]:
url = "https://datascienceschools.github.io/cars_price.csv"
df = pd.read_csv(url)
df.head()
Out[54]:
Exploring the descriptive statistics of the variables¶
- Descriptive statistics: only descriptives for the numerical variables are shown
- To include the categorical ones, you should specify include='all'
- Categorical variables don't have some types of numerical descriptives
- Numerical variables don't have some types of categorical descriptivesIn [55]:
df.describe(include='all')
Out[55]:
Dropping 'Model'¶
- 'Model' is a categorical variable and has 306 unique values
- 306 unique values -> 306 dummy variables, making it hard to implement regression
- Let's remove itIn [56]:
df = df.drop('Model', axis=1)
Checking Missing Values¶
In [57]:
df.isnull().sum()
Out[57]:
Removing Missing Values¶
In [58]:
df = df.dropna(axis=0)
df.isnull().sum()
Out[58]:
Checking descriptives statistics without the missing values¶
In [59]:
df.describe()
Out[59]:
Exploring Probability Distribution Function (PDF)¶
- Displaying PDF of a variable
- The PDF will show us how that variable is distributed
- Easy way to spot anomalies, such as outliers
- PDF is often the basis on which we decide whether we want to transform a feature
- seaborn.distplot() ->
This function combines the matplotlib hist function (with automatic calculation of a good default bin size) with the seaborn kdeplot() and rugplot() functions. It can also fit scipy.stats distributions and plot the estimated PDF over the dataSeaborn distplot ('Price')¶
- Obviously there are some outliers
- Outliers are situated around the higher prices (right side of the graph)
- Dealing with the problem -> easily by removing 1% of the problematic samplesIn [60]:
sns.distplot(df['Price'])
Out[60]:
Removing the Outliers ('Price')¶
In [61]:
df = df[df['Price'] < df['Price'].quantile(0.99)]
Seaborn distplot ('Price') after removing outliers¶
In [62]:
sns.distplot(df['Price'])
Out[62]:
Seaborn distplot 'Mileage'¶
- Obviously there are some outliers
- Outliers are situated around the higher mileage(right side of the graph)
- Dealing with the problem -> easily by removing 1% of the problematic samplesIn [63]:
sns.distplot(df['Mileage'])
Out[63]:
Removing the Outliers ('Mileage')¶
In [64]:
df = df[df['Mileage']< df['Mileage'].quantile(0.99)]
Seaborn distplot 'Mileage' after removing outliers¶
In [65]:
sns.distplot(df['Mileage'])
Out[65]:
Seaborn distplot 'EngineV'¶
- Obviously there are some outliers
- Outliers are situated around the higher engine volumes(right side of the graph)
- But, car engine volumes are usually below 6.5
- Dealing with the problem -> easily by removing the problematic samples > 6.5In [66]:
sns.distplot(df['EngineV'])
Out[66]:
Removing the Outliers ('EngineV')¶
- Issue comes from the fact that most missing values are indicated with 99.99 or 99
- There are also some incorrect entries like 75
- Car engine volumes are usually below 6.5In [67]:
df = df[df['EngineV']<6.5]
Seaborn distplot 'EngineV' after removing outliers¶
In [68]:
sns.distplot(df['EngineV'])
Out[68]:
Seaborn distplot 'Year'¶
- Obviously there are some outliers
- Outliers are situated left side of the graph
- Dealing with the problem -> easily by removing 1% of the problematic samplesIn [69]:
sns.distplot(df['Year'])
Out[69]:
Removing the Outliers ('Year')¶
In [70]:
df = df[df['Year']> df['Year'].quantile(0.01)]
Seaborn distplot 'Year' after removing outliers¶
In [71]:
sns.distplot(df['Year'])
Out[71]:
Reset the Index¶
- The original indexes are preserved after removing observations
- If we remove observations with indexes 2 and 3, the indexes will be as: 0,1,4,5,6
- Once we reset the index, a new column will be created containing the old index
- To drop that column, add -> 'drop=True'In [72]:
df = df.reset_index(drop=True)
Descriptive Statistics after removing all outliers¶
In [73]:
df.describe()
Out[73]: