Real world data sets contain missing values. These values encoded as blanks, NaNs or other placeholders. we are used to skip that entire row/columns containing missing value while describing a machine learning model. However this comes at the price of losing data.
A good strategy is to fill out the missing values using known part of the data. The Imputer class provides basic strategies for imputing missing values, either using the mean, the median or the most frequent value of the row or column in which the missing values are located. Let's Check
first import our module and grab our data
import pandas as pd data= pd.read_csv(r'titanic.csv')
Let's Grab some column data which has some missing values. Have a look.
data['Age'].values[1:30]
Output :
array([ 38., 26., 35., 35., nan, 54., 2., 27., 14., 4., 58., 20., 39., 14., 55., 2., nan, 31., nan, 35., 34., 15., 28., 8., 38., nan, 19., nan, nan])
Here one can see missing values are represented as 'nan' (nan in Numpy array and NaN in pandas dataframe).Below are some method by which missing value can be handled.
-
Using sklearn.preprocessing Imputer function
-
Using Pandas fillna() method
1. Using Sklearn Imputer Function
import imputer class from sklearn.preprocessing and define our imputer. here we will describe missing_value placeholder, strategy used to fill out the value and axis.
from sklearn.preprocessing import Imputer imp = Imputer(missing_values='NaN', strategy='mean', axis=0) print imp
Output
Imputer(axis=0, copy=True, missing_values='NaN', strategy='mean', verbose=0)
Here some additional variables axis, verbose are set to default. Now let's fit our data to our Imputer.
imp.fit(data['Age'].values.reshape(-1,1))
Output:
Imputer(axis=0, copy=True, missing_values='NaN', strategy='mean', verbose=0)
Now transform our data replacing Nan with appropriate mean value.
age_reformed= imp.transform(data['Age'].values.reshape(-1,1))
Now Check the transformed data.
print(age_reformed[1:10])
Output:
array([[ 38. ], [ 26. ], [ 35. ], [ 35. ], [ 29.69911765], [ 54. ], [ 2. ], [ 27. ], [ 14. ]])
Now these "nan" values has been transformed into mean value(29.69911765). The data after transformation can be used to machine learning model as it doesn't have missing value.
2. Using Pandas fillna()
Python's pandas library provide a direct way to deal with missing value.
First create a new data frame with column value as age.
df= pd.DataFrame(columns=['age']) df['age']= data['Age'].values
We have missing values in newly created Dataframe.
print(df.head(10)) # output age 0 22.0 1 38.0 2 26.0 3 35.0 4 35.0 5 NaN 6 54.0 7 2.0 8 27.0 9 14.0
Pandas fillna method is applied to dataframe as
DataFrame.fillna(value=None, method=None, axis=None, inplace=False, limit=None, downcast=None, **kwargs)
Now apply this to our data and put a value equal to 25 where nan occurs.
filled_value=df.fillna(25) print(fill_value.head(10)) # output age 0 22.0 1 38.0 2 26.0 3 35.0 4 35.0 5 25.0 6 54.0 7 2.0 8 27.0 9 14.0
Here value in 6th row(5th indexed) has been changed from 'NaN' to 25.0. More information about the default variable can be checked from docs.
Pandas's DataFrame fillan:-
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html
Pandas's Series fillna:-
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.fillna.html
Comment and let me know what are you using.
