机器学习—编程基础1
预处理黑色星期五数据（熟悉流程，后续文章有进行预测部分）
主要步骤流程：
- 1. 导入包和数据集
- 2. 处理缺失数据
- 3. 特征工程
- 4. 处理类别型字段
- 5. 得到自变量和因变量
- 6. 拆分训练集和测试集
- 7. 特征缩放
1. 导入包和数据集
In [2]:# 导入包
import numpy as np
import pandas as pd
In [3]:
# 导入数据集
data = pd.read_csv('BlackFriday.csv')
data.head(5)
Out[3]:
| User_ID | Product_ID | Gender | Age | Occupation | City_Category | Stay_In_Current_City_Years | Marital_Status | Product_Category_1 | Product_Category_2 | Product_Category_3 | Purchase | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1000001 | P00069042 | F | 0-17 | 10 | A | 2 | 0.0 | 3 | NaN | NaN | 8370 |
| 1 | 1000001 | P00248942 | F | 0-17 | 10 | A | 2 | 0.0 | 1 | 6.0 | 14.0 | 15200 |
| 2 | 1000001 | P00087842 | F | 0-17 | 10 | A | 2 | NaN | 12 | NaN | NaN | 1422 |
| 3 | 1000001 | P00085442 | F | 0-17 | 10 | A | 2 | 0.0 | 12 | 14.0 | NaN | 1057 |
| 4 | 1000002 | P00285442 | M | 55+ | 16 | C | 4+ | 0.0 | 8 | NaN | NaN | 7969 |
2. 处理缺失数据
In [4]:# 处理缺失数据
# 检测缺失值
null_df = data.isnull().sum()
null_df
Out[4]:
User_ID 0
Product_ID 0
Gender 0
Age 0
Occupation 0
City_Category 0
Stay_In_Current_City_Years 0
Marital_Status 3
Product_Category_1 0
Product_Category_2 15721
Product_Category_3 34817
Purchase 0
dtype: int64
Marital_Status字段有3个缺失值；根据业务场景，缺失的默认是未婚，即用0填补。 Product_Category_2字段有15721个缺失值；根据业务场景，这个字段不重要，删除。 Product_Category_3字段有34817个缺失值；根据业务场景，这个字段不重要，删除。
In [5]:# 删除2个缺失列
data = data.drop(['Product_Category_2', 'Product_Category_3'], axis = 1)
In [6]:
# 填补缺失列
data['Marital_Status'].fillna(0, inplace = True)
In [7]:
# 再次检测缺失值
null_df = data.isnull().sum()
null_df
Out[7]:
User_ID 0
Product_ID 0
Gender 0
Age 0
Occupation 0
City_Category 0
Stay_In_Current_City_Years 0
Marital_Status 0
Product_Category_1 0
Purchase 0
dtype: int64
3. 特征工程
特征工程（Feature Engineering）是将原始数据转化成更好的表达问题本质的特征的过程
In [8]:# 特征工程
# 删除无用的列
data = data.drop(['User_ID', 'Product_ID'], axis = 1)
In [9]:
# 处理Stay_In_Current_City_Years列
data['Stay_In_Current_City_Years'].replace('4+', 4, inplace = True)
data['Stay_In_Current_City_Years'] = data['Stay_In_Current_City_Years'].astype('int64')
4. 处理类别型字段
In [10]:# 处理类别型字段
# 检查类别型变量
print(data.dtypes)
Gender object
Age object
Occupation int64
City_Category object
Stay_In_Current_City_Years int64
Marital_Status float64
Product_Category_1 int64
Purchase int64
dtype: object
根据业务场景，Occupation列、Marital_Status列和Product_Category_1列应该是类别型字段。需要转换。
In [11]:# 转换变量类型
data['Product_Category_1'] = data['Product_Category_1'].astype('object')
data['Occupation'] = data['Occupation'].astype('object')
data['Marital_Status'] = data['Marital_Status'].astype('object')
In [12]:
# 检查类别型变量
print(data.dtypes)
Gender object
Age object
Occupation object
City_Category object
Stay_In_Current_City_Years int64
Marital_Status object
Product_Category_1 object
Purchase int64
dtype: object
In [13]:
# 字符编码&独热编码
data = pd.get_dummies(data, drop_first = True)
data.head(5)
Out[13]:
| Stay_In_Current_City_Years | Purchase | Gender_M | Age_18-25 | Age_26-35 | Age_36-45 | Age_46-50 | Age_51-55 | Age_55+ | Occupation_1 | ... | Product_Category_1_9 | Product_Category_1_10 | Product_Category_1_11 | Product_Category_1_12 | Product_Category_1_13 | Product_Category_1_14 | Product_Category_1_15 | Product_Category_1_16 | Product_Category_1_17 | Product_Category_1_18 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 8370 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 2 | 15200 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 2 | 2 | 1422 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| 3 | 2 | 1057 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| 4 | 4 | 7969 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
5 rows × 49 columns
5. 得到自变量和因变量
In [14]:# 得到自变量和因变量
y = data['Purchase'].values
print(y.shape)
data = data.drop(['Purchase'], axis = 1)
x = data.values
print(x.shape)
(50000,)
(50000, 48)
6. 拆分训练集和测试集
In [15]:# 拆分训练集和测试集
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.3, random_state = 205)
print(x_train.shape)
print(x_test.shape)
print(y_train.shape)
print(y_test.shape)
(35000, 48)
(15000, 48)
(35000,)
(15000,)
自变量保存到x_train和x_test中，因变量保存到y_train和y_test中。
7. 特征缩放
In [16]:y_train.shape
Out[16]:
(35000,)
In [17]:
a = y_train.reshape(-1, 1)
a.shape
Out[17]:
(35000, 1)
In [18]:
# 特征缩放
from sklearn.preprocessing import StandardScaler
sc_x = StandardScaler()
x_train = sc_x.fit_transform(x_train)
x_test = sc_x.transform(x_test)
sc_y = StandardScaler()
y_train = np.ravel(sc_y.fit_transform(y_train.reshape(-1, 1)))
In [19]:
x_train[:3,:]
Out[19]:
array([[ 0.10343173, -1.77050054, -0.4873769 , 1.24137799, -0.49566581,
-0.2961413 , -0.27386969, -0.20083646, -0.30534824, -0.22524545,
-0.18825963, -0.38533409, -0.1388486 , -0.18621612, -0.33838987,
-0.05537615, -0.10670196, -0.16153035, -0.14868325, -0.24336363,
-0.12529891, -0.22657272, -0.1519627 , -0.2198004 , -0.28316115,
-0.11100179, -0.12739959, -0.26453064, -0.84608185, -0.66227816,
-0.83770833, -0.2140971 , -0.19340603, -0.14727185, 1.59734051,
-0.19412034, -0.08274392, -0.51575763, -0.02726553, -0.09620978,
-0.21851165, -0.08396038, -0.09756185, -0.05216966, -0.10900638,
-0.13139418, -0.03339953, -0.0726975 ],
[-1.43966733, 0.56481203, 2.05180016, -0.80555641, -0.49566581,
-0.2961413 , -0.27386969, -0.20083646, -0.30534824, -0.22524545,
-0.18825963, 2.5951506 , -0.1388486 , -0.18621612, -0.33838987,
-0.05537615, -0.10670196, -0.16153035, -0.14868325, -0.24336363,
-0.12529891, -0.22657272, -0.1519627 , -0.2198004 , -0.28316115,
-0.11100179, -0.12739959, -0.26453064, 1.18191875, -0.66227816,
-0.83770833, -0.2140971 , -0.19340603, -0.14727185, -0.62604059,
-0.19412034, -0.08274392, 1.93889522, -0.02726553, -0.09620978,
-0.21851165, -0.08396038, -0.09756185, -0.05216966, -0.10900638,
-0.13139418, -0.03339953, -0.0726975 ],
[-0.6681178 , 0.56481203, -0.4873769 , 1.24137799, -0.49566581,
-0.2961413 , -0.27386969, -0.20083646, -0.30534824, -0.22524545,
-0.18825963, -0.38533409, -0.1388486 , -0.18621612, -0.33838987,
-0.05537615, -0.10670196, -0.16153035, 6.7257073 , -0.24336363,
-0.12529891, -0.22657272, -0.1519627 , -0.2198004 , -0.28316115,
-0.11100179, -0.12739959, -0.26453064, 1.18191875, -0.66227816,
1.19373291, -0.2140971 , -0.19340603, -0.14727185, 1.59734051,
-0.19412034, -0.08274392, -0.51575763, -0.02726553, -0.09620978,
-0.21851165, -0.08396038, -0.09756185, -0.05216966, -0.10900638,
-0.13139418, -0.03339953, -0.0726975 ]])
In [20]:
y_train[:3]
Out[20]:
array([-0.12211265, -1.46218147, -0.78499224])
缩放后的x_train和y_train，所有特征的值处于相似范围内。
结论： 数据预处理有固定的方法。 Python提供了丰富的库，方便人们做数据预处理工作。 最初的数据通过数据预处理生成了x_train、y_train、x_test、y_test。在下一章中，前2个变量将训练模型，后2个变量将评估模型。