人工智能机器学习：Avazu移动互联在线广告点击预测

当前在线广告服务中，广告的点击率（CTR）是评估广告效果的一个非常重要的指标。因此，点击率预测系统是必不可少的，并广泛用于赞助搜索和实时出价。那么如何计算广告的点击率呢？

广告的点击率 = 广告点击量/广告的展现量

如果一个广告被展现了100次，其中被点击了20次，那么点击率就是20%。

今天我们就来动手开发一个移动广告点击率的预测系统，我们数据来自于kaggle，数据包含了10天的Avazu的广告点击数据。

数据

你可以在这里下载移动广告点击数据,由于总数据量达到了4千多万条，数据量过于庞大,为了不影响我们的计算速度,因此我们要从中随机抽样100万条数据，同时我们要对数据的相关字段类型进行重置,这有助于我们以后的计算以及可视化。

In [1]:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
import xgboost as xgb
import lightgbm as lgb
import gzip
import random
import seaborn as sns
import matplotlib.pyplot as plt
plt.rcParams['font.sans-serif'] = ['SimHei']  
plt.rcParams['axes.unicode_minus'] = False 
%matplotlib inline

In [2]:

types_train = {
    'id': np.dtype(int), 
    'click': np.dtype(int),       #是否点击,1表示被点击,0表示没被点击
    'hour': np.dtype(int),        #广告被展现的日期+时间
    'C1': np.dtype(int),          #匿名分类变量
    'banner_pos': np.dtype(int),  #广告位置
    'site_id': np.dtype(str),     #站点Id
    'site_domain': np.dtype(str),  #站点域名
    'site_category': np.dtype(str), #站点分类
    'app_id': np.dtype(str),        # appId 
    'app_domain': np.dtype(str),    # app域名
    'app_category': np.dtype(str),  # app分类
    'device_id': np.dtype(str),     #设备Id
    'device_ip': np.dtype(str),     #设备Ip
    'device_model': np.dtype(str),  #设备型号
    'device_type': np.dtype(int),   #设备型号
    'device_conn_type': np.dtype(int),
    'C14': np.dtype(int),   #匿名分类变量
    'C15': np.dtype(int),   #匿名分类变量
    'C16': np.dtype(int),   #匿名分类变量
    'C17': np.dtype(int),   #匿名分类变量
    'C18': np.dtype(int),   #匿名分类变量
    'C19': np.dtype(int),   #匿名分类变量
    'C20': np.dtype(int),   #匿名分类变量
    'C21':np.dtype(int)    #匿名分类变量
}

In [3]:

n = 40428967  #数据集中的记录总数
sample_size = 1000000
skip_values = sorted(random.sample(range(1,n), n-sample_size)) 
parse_date = lambda val : pd.datetime.strptime(val, '%y%m%d%H')

with gzip.open('./data/ctr/train.gz') as f:
    train = pd.read_csv(f, parse_dates = ['hour'], date_parser = parse_date, dtype=types_train, skiprows = skip_values)
print(len(train))
train.head()

Out[3]:

	id	click	hour	C1	banner_pos	site_id	site_domain	site_category	app_id	app_domain	...	device_type	C14	C15	C16	C17	C18	C19	C20	C21
0	-1636923355	0	2014-10-21	1005	0	85f751fd	c4e18dd6	50e219e0	e2a1ca37	2347f47a	...	1	15708	320	50	1722	0	35	-1	79
1	-193497663	0	2014-10-21	1005	1	e151e245	7e091613	f028772b	ecad2386	7801e8d9	...	1	17747	320	50	1974	2	39	100021	33
2	1315205890	0	2014-10-21	1002	0	85f751fd	c4e18dd6	50e219e0	a37bf1e4	7801e8d9	...	0	21691	320	50	2495	2	167	-1	23
3	1336077603	0	2014-10-21	1005	0	1fbe01fe	f3845767	28905ebd	ecad2386	7801e8d9	...	1	15701	320	50	1722	0	35	-1	79
4	-1368186722	1	2014-10-21	1005	0	1fbe01fe	f3845767	28905ebd	ecad2386	7801e8d9	...	1	15708	320	50	1722	0	35	100084	79

5 rows × 24 columns

特征工程

接下来我们要做的就是数据的探索性分析(EDA)和特征工程(Feature Engineering),首先我们要确定哪些目标变量，哪些是特征变量,根据kaggle中对数据的描述信息中我们可知,目标变量就是"click"字段它表示广告是否被点击过(1表示被点击,0未被点击)，其余所有的字段都是特征变量。在特征变量中C1,C14~C21表示匿名的分类变量(我们不知道它的含义)，其余的特征变量都是和站点,app,设备相关的变量。我们搞清了变量的大概含义以后,接下来我们要分析一下目标变量"click"，首先看看它的数据分布情况:

In [4]:

print(train['click'].value_counts())
print()
print(train['click'].value_counts()/len(train))
# train['click'].value_counts().plot(label='dd',kind = 'bar')
sns.countplot(x='click',data=train, palette='hls')
plt.show()

0    830365
1    169635
Name: click, dtype: int64

0    0.830365
1    0.169635
Name: click, dtype: float64

在“click”变量的统计数据中，点击的数量大约占17%，未点击的数量大约占83%。也就是说广告的平均点击率大概是在17%左右。

接下来我们来分析另外一个关键的特征变量:hour,它可能表示广告被展现的日期+时间，我们要看看不同的日期和时间对广告点击量的影响:

In [5]:

train.hour.describe()

Out[5]:

count                 1000000
unique                    240
top       2014-10-22 09:00:00
freq                    11357
first     2014-10-21 00:00:00
last      2014-10-30 23:00:00
Name: hour, dtype: object

In [ ]: In [6]:

print(train.hour.describe())

train.groupby('hour').agg({'click':'sum'}).plot(figsize=(12,6))
plt.ylabel('点击量')
plt.title('时间和点击量')

count                 1000000
unique                    240
top       2014-10-22 09:00:00
freq                    11357
first     2014-10-21 00:00:00
last      2014-10-30 23:00:00
Name: hour, dtype: object

Out[6]:

Text(0.5, 1.0, '时间和点击量')

由上面的统计结果可知数据的开始时间是2014-10-21 00:00:00,结束时间是2014-10-30 23:00:00，一共10天，点击量高峰的时刻是在10月22日和10月28日这两天，10月24日点击量最低。

对Hour的特征工程

我们知道hour变量包含了具体的日期和时间,接下来我们想知道点击量和具体的时间是什么关系,此时我们忽略日期,只关注具体时间和点击量。接下来我们从hour变量中抽取时间,然后查看时间和点击量之间的关系:

In [7]:

train['time'] = train.hour.apply(lambda x: x.hour)
train.sample(5)

Out[7]:

	id	click	hour	C1	banner_pos	site_id	site_domain	site_category	app_id	app_domain	...	device_conn_type	C14	C15	C16	C17	C18	C19	C20	C21	time
940121	49763770	0	2014-10-30 10:00:00	1005	0	a82179f8	62b6befe	3e814130	ecad2386	7801e8d9	...	0	23908	320	50	2741	0	163	-1	17	10
479783	-1721082668	0	2014-10-25 18:00:00	1005	1	b7e9786d	b12b9f85	f028772b	ecad2386	7801e8d9	...	0	19771	320	50	2227	0	679	-1	48	18
838574	1186530652	0	2014-10-29 09:00:00	1005	0	1fbe01fe	f3845767	28905ebd	ecad2386	7801e8d9	...	0	23161	320	50	2667	0	35	100148	221	9
344962	-2065125200	1	2014-10-24 04:00:00	1002	0	887a4754	e3d9ca35	50e219e0	ecad2386	7801e8d9	...	0	6563	320	50	572	2	39	-1	32	4
936566	1805891415	0	2014-10-30 09:00:00	1005	0	85f751fd	c4e18dd6	50e219e0	685d1c4c	2347f47a	...	2	23224	320	50	2676	0	299	100176	221	9

5 rows × 25 columns

In [8]:

train.groupby('time').agg({'click':'sum'}).plot(figsize=(12,6),grid=True)
plt.ylabel('点击次数')
plt.title('时间和点击量')

Out[8]:

Text(0.5, 1.0, '时间和点击量')

我们看到点击量的高峰大约是在每天下午的13点到14点之间 ,点击量的最低点是在每天的零点左右。这应该是合理的，因为下午1点到2点应该是人们精力最旺盛的时候，而晚上零点大部分人都进入了梦乡。

接下来我们要查看一下在不同的时间点的情况下，广告的展现量和点击量的关系:

In [9]:

train.groupby(['time', 'click']).size().unstack().plot(kind='bar', figsize=(12,6))
plt.ylabel('数量')
plt.title('展现量与点击量');

我们将时间按每个时间点展开,这里没有特别之处下午1点的展现量最大,所以点击量也是最大,我们发现展现量和点击量似乎是成正比的。这似乎也告诉我们，如果您要投放在线广告，请在下午1点至2点之间投放,因为此时广告的展现量和点击量都是最大的。

接下来我们来计算一下各个时间点的广告点击率，并查看点击率的数据分布。

In [10]:

df_click = train[train['click'] == 1]
df_hour = train[['time','click']].groupby(['time']).count().reset_index()
df_hour = df_hour.rename(columns={'click': 'impressions'})
df_hour['clicks'] = df_click[['time','click']].groupby(['time']).count().reset_index()['click']
df_hour['CTR'] = df_hour['clicks']/df_hour['impressions']*100
df_hour.head()

Out[10]:

	time	impressions	clicks	CTR
0	0	21234	3793	17.862861
1	1	24463	4534	18.534113
2	2	30313	5388	17.774552
3	3	34386	5999	17.446054
4	4	47211	7543	15.977209

In [11]:

plt.figure(figsize=(12,6))
sns.barplot(y='CTR', x='time', data=df_hour)
plt.title('点击率的时间分布')

Out[11]:

Text(0.5, 1.0, '点击率的时间分布')

在这里我们发现了一件有趣的事，广告点击率最高的时间点居然在凌晨1点，上午7点，下午16点，而从之前的分析中我知道广告展现量最高的时间点是在下午的13点，但是从上图中我们可知13点的广告点击率并非是最高的。这似乎说明高的展现量和高的点击量并不意味着就有高的点击率。凌晨1点上网的“夜游神”们才是点击率的真正贡献者。

按星期特征工程

前面我们我们已经分别实现了按日期和按时间两种方式来分析点击率,接下来我们再继续扩展对hour变量的分析,这回我们要按星期来分析点击率。我们首先要把hour变量转换成星期。

In [12]:

train['day_of_week'] = train['hour'].apply(lambda val: val.weekday_name)
cats = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']
train.groupby('day_of_week').agg({'click':'sum'}).reindex(cats).plot(figsize=(12,6))
ticks = list(range(0, 7, 1)) 
labels = "周一 周二 周三 周四 周五 周六 周日".split()
plt.xticks(ticks, labels)
plt.title('星期的点击量')

d:\ProgramData\Anaconda3\lib\site-packages\ipykernel_launcher.py:1: FutureWarning: `weekday_name` is deprecated and will be removed in a future version. Use `day_name` instead
  """Entry point for launching an IPython kernel.

Out[12]:

Text(0.5, 1.0, '星期的点击量')

In [13]:

train.groupby(['day_of_week','click']).size().unstack().reindex(cats).plot(kind='bar', title="Day of the Week", figsize=(12,6))
ticks = list(range(0, 7, 1)) 
labels = "周一 周二 周三 周四 周五 周六 周日".split()
plt.xticks(ticks, labels)
plt.title('星期的展现量和点击量分布')

Out[13]:

Text(0.5, 1.0, '星期的展现量和点击量分布')

从上图可知星期二的展现量和点击量是最高的，接下来是星期三和星期四，不过展现量和点击量较高并不意味着点击率也较高，因此接下来我们要按星期来计算一下点击率。

In [14]:

df_click = train[train['click'] == 1]
df_dayofweek = train[['day_of_week','click']].groupby(['day_of_week']).count().reset_index()
df_dayofweek = df_dayofweek.rename(columns={'click': 'impressions'})
df_dayofweek['clicks'] = df_click[['day_of_week','click']].groupby(['day_of_week']).count().reset_index()['click']
df_dayofweek['CTR'] = df_dayofweek['clicks']/df_dayofweek['impressions']*100

plt.figure(figsize=(12,6))
# sns.set(style="whitegrid")
sns.barplot(y='CTR', x='day_of_week', data=df_dayofweek, 
            order=['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'])
plt.title('星期的点击率')

Out[14]:

Text(0.5, 1.0, '星期的点击率')

通过之前的我们知道星期二和星期三有着最高的展现量和点击量，可是他们的点击率却是最低的。相反星期六和星期天却有着最高的点击率。这是否说明星期六和星期天是人民群众最空闲的时候，有了空闲时间大家才会想到去购物，所以广告的点击率才会是最高的？

通过对数据的严谨分析，我们就会从中发现人民群众的日常行为举止以及他们的活动规律都会在数据中得到体现，只要你足够努力，就可以让数据说话！

匿名特征变量C1

C1是一个匿名的分类型变量，我们不知道它的含义，我们先查看一下c1的数据分布

In [15]:

print(train.C1.value_counts()/len(train))

1005    0.918282
1002    0.055076
1010    0.022543
1012    0.002868
1007    0.000855
1001    0.000239
1008    0.000137
Name: C1, dtype: float64

因为C1是分类型变量，它的值包含了1005,1002,1010,1012,1007,1001,1008七种，其中1005的所占比例高达91.87%，接下来我们看看C1的不同的值对点击率的贡献

In [16]:

C1_values = train.C1.unique()
C1_values.sort()
ctr_avg_list=[]
for i in C1_values:
    ctr_avg=train.loc[np.where((train.C1 == i))].click.mean()
    ctr_avg_list.append(ctr_avg)
    print(" C1 value: {},  点击率: {}".format(i,ctr_avg))

 C1 value: 1001,  点击率: 0.03347280334728033
 C1 value: 1002,  点击率: 0.21112644345994624
 C1 value: 1005,  点击率: 0.16909838154292473
 C1 value: 1007,  点击率: 0.03976608187134503
 C1 value: 1008,  点击率: 0.1386861313868613
 C1 value: 1010,  点击率: 0.09754691034911059
 C1 value: 1012,  点击率: 0.16283124128312412

从上面的统计结果可知,虽然1005数据量所占比重最高,但是它的点击率并不是最高,1002的点击率最高达到了21.3%。接下来我们看一下C1的展现量和点击量的分布

In [17]:

train.groupby(['C1', 'click']).size().unstack().plot(kind='bar', figsize=(12,6), title='C1 展现量和点击量分布');

从上图可知，1005的展现量和点击量是最高的，但这并不意味着点击率也是最高的，下面我们看一下C1的点击率的分布

In [18]:

df_c1 = train[['C1','click']].groupby(['C1']).count().reset_index()
df_c1 = df_c1.rename(columns={'click': 'impressions'})
df_c1['clicks'] = df_click[['C1','click']].groupby(['C1']).count().reset_index()['click']
df_c1['CTR'] = df_c1['clicks']/df_c1['impressions']*100

plt.figure(figsize=(12,6))
sns.barplot(y='CTR', x='C1', data=df_c1)
plt.title('C1的点击率分布')

Out[18]:

Text(0.5, 1.0, 'C1的点击率分布')

我们看到点击率最高的并不是1005，而是1002. 它的点击率达到了21%，下面我们总结一下C1数据量和点击率的分布:

从上表中我们可以看出,1002的数据比例是5.5%，它贡献的点击率为21.33%远大于17%的平均点击率，1002的数据比例为0.28%却贡献了17.66%的点击率，1008的数据比例是0.01%，它贡献了14.84%的点击率，性价比非常高。

banner_pos

banner_pos表示广告在网页中的位置，广告摆放在网页的不同位置可能会带来不同的点击量和点击量，下面我们就来分析一下banner_pos这个分类型变量。首先我们查看一下banner_pos的数据分布

In [19]:

print(train.banner_pos.value_counts()/len(train))

0    0.719386
1    0.278839
7    0.001117
2    0.000309
4    0.000162
5    0.000137
3    0.000050
Name: banner_pos, dtype: float64

从上面的统计结果可知banner_pos包含了7个值,它可能代表网页中的7个不同位置，其中位置0和位置1占据了机会99%的数据比例，也就是说绝大多数广告都房子了位置0或者位置1的地方。

下面我们看看不同位置对点击率的贡献:

In [20]:

banner_pos = train.banner_pos.unique()
banner_pos.sort()
ctr_avg_list=[]
for i in banner_pos:
    ctr_avg=train.loc[np.where((train.banner_pos == i))].click.mean()
    ctr_avg_list.append(ctr_avg)
    print(" banner 位置: {},  点击率: {}".format(i,ctr_avg))

 banner 位置: 0,  点击率: 0.164067691058764
 banner 位置: 1,  点击率: 0.18354677788975
 banner 位置: 2,  点击率: 0.12297734627831715
 banner 位置: 3,  点击率: 0.16
 banner 位置: 4,  点击率: 0.1419753086419753
 banner 位置: 5,  点击率: 0.1386861313868613
 banner 位置: 7,  点击率: 0.3034914950760967

位置0和位置1的点击率分别为16.4%和18.2%，它们的点击率并不是最高。位置7和位置3的点击率分别达到了33%和24%，它们的点击率要比位置1和位置0高很多。

In [21]:

train.groupby(['banner_pos', 'click']).size().unstack().plot(kind='bar', figsize=(12,6), title='banner 位置的广告展现量和点击量的分布')

Out[21]:

下面我们再看一下banner_pos的点击率的分布

In [22]:

df_banner = train[['banner_pos','click']].groupby(['banner_pos']).count().reset_index()
df_banner = df_banner.rename(columns={'click': 'impressions'})
df_banner['clicks'] = df_click[['banner_pos','click']].groupby(['banner_pos']).count().reset_index()['click']
df_banner['CTR'] = df_banner['clicks']/df_banner['impressions']*100
sort_banners = df_banner.sort_values(by='CTR',ascending=False)['banner_pos'].tolist()
plt.figure(figsize=(12,6))
sns.barplot(y='CTR', x='banner_pos', data=df_banner, order=sort_banners)
plt.title('banner 位置的点击率的分布')

Out[22]:

Text(0.5, 1.0, 'banner 位置的点击率的分布')

从上图可知位置7和位置3的点击率是最高的，但是他们的数据比例并不是最高的，相反位置0和位置1的数据比例，展现量和点击量都是最高的，但是他们的点击率并非最高。

device_type

device_type表示设备类型，广告可能会在多种设备上展示，下面我们看一下device_type的数据分布

In [23]:

print((train.device_type.value_counts()/len(train)))

1    0.922380
0    0.055076
4    0.019286
5    0.003257
2    0.000001
Name: device_type, dtype: float64

我们看到一共有4种设备，其中设备1所占比例最大达到了92%，绝大多数广告都是在设备1上展示的。下面我们看一下展现量和点击量的分布

In [24]:

train[['device_type','click']].groupby(['device_type','click']).size().unstack().plot(kind='bar', title='设备类型')

Out[24]:

我们看到设备1上的广告展现量和点击量都是最大的。其他设备的展现量和点击量相对较少。为此我们要详细分析一下设备1上的点击量的情况，我们按照不同的时间点对设备1的点击量进行一下分析。

In [25]:

df_click[df_click['device_type']==1].groupby(['time', 'click'])\
                                     .size().unstack()\
                                     .plot(kind='bar', title="设备1的点击量分布", figsize=(12,6))

Out[25]:

从上图可知，设备1上的点击量最高点位于下午1点，这和我们之前按时间分析点击量的结果是一致的。

下面我们分别统计出所有类型的设备的点击量、展现量和点击率。

In [26]:

device_type_click = df_click.groupby('device_type').agg({'click':'sum'}).reset_index()
device_type_impression = train.groupby('device_type').agg({'click':'count'}).reset_index().rename(columns={'click': 'impressions'})
merged_device_type = pd.merge(left = device_type_click , right = device_type_impression, how = 'inner', on = 'device_type')

merged_device_type['CTR'] = merged_device_type['click'] / merged_device_type['impressions']*100

merged_device_type

Out[26]:

	device_type	click	impressions	CTR
0	0	11628	55076	21.112644
1	1	155808	922380	16.891953
2	4	1878	19286	9.737634
3	5	321	3257	9.855695

我们看到点击率最高的设备是设备0，并不是设备1.所以说展现量和点击量都较高并不意味着点击率也会较高。

app features

In [27]:

print("app_id的唯一值有 {} 个".format(train.app_id.nunique()))
print("app_domain的唯一值有 {} 个".format(train.app_domain.nunique()))
print("app_category的唯一值有 {} 个".format(train.app_category.nunique()))

app_id的唯一值有 3122 个
app_domain的唯一值有 197 个
app_category的唯一值有 26 个

In [28]:

print((train.app_category.value_counts()/len(train)))

07d7df22    0.647846
0f2161f8    0.236653
cef3e649    0.042757
8ded1f7a    0.035973
f95efa07    0.027883
d1327cf5    0.003055
dc97ec06    0.001336
09481d60    0.001328
75d80bbe    0.000947
fc6fa53d    0.000564
4ce2e9fc    0.000519
879c24eb    0.000325
a3c42688    0.000291
4681bb9d    0.000171
0f9a328c    0.000133
2281a340    0.000059
a86a3e89    0.000057
8df2e842    0.000048
79f0b860    0.000019
0bfbc358    0.000010
a7fd01ec    0.000008
7113d72a    0.000006
18b1e0be    0.000005
5326cf99    0.000003
2fc4f2aa    0.000003
bd41f328    0.000001
Name: app_category, dtype: float64

In [29]:

train['app_category'].value_counts().plot(kind='bar', title='App Category v/s Clicks',figsize=(12,6))

Out[29]:

In [30]:

train_app_category = train.groupby(['app_category', 'click']).size().unstack()
train_app_category.div(train_app_category.sum(axis=1), axis=0).plot(kind='bar', stacked=True, title="Intra-category CTR",figsize=(12,6))

Out[30]:

C14 - C21 features

In [31]:

print("C14的唯一值有 {} 个".format(train.C14.nunique()))
print("C15的唯一值有 {} 个".format(train.C15.nunique()))
print("C16的唯一值有 {} 个".format(train.C16.nunique()))
print("C17的唯一值有 {} 个".format(train.C17.nunique()))
print("C18的唯一值有 {} 个".format(train.C18.nunique()))
print("C19的唯一值有 {} 个".format(train.C19.nunique()))
print("C20的唯一值有 {} 个".format(train.C20.nunique()))

C14的唯一值有 2253 个
C15的唯一值有 8 个
C16的唯一值有 9 个
C17的唯一值有 421 个
C18的唯一值有 4 个
C19的唯一值有 66 个
C20的唯一值有 165 个

In [32]:

train.groupby(['C15', 'click']).size().unstack().plot(kind='bar', stacked=True, title='C15 distribution')

Out[32]:

In [33]:

train.groupby(['C16', 'click']).size().unstack().plot(kind='bar', stacked=True, title='C16 distribution')

Out[33]:

In [34]:

train.groupby(['C18', 'click']).size().unstack().plot(kind='bar', stacked=True, title='C18 distribution')

Out[34]:

建模

has简介

由于我们的的数据样本量有100万条，特征变量有20个左右，那么总共的特征值将会有100万X20=2000万个左右，为了减少系统内存的消耗，我们要使用python的内置hash函数来映射某些特征变量，我们要将那些类型为object的特征变量映射为一定范围内的整数(原来的string被映射成了integer)，这样就可以大大降低内存的消耗。下面我们看看未使用hash之前我们的样本数据:

In [35]:

train.head()

Out[35]:

	id	click	hour	C1	banner_pos	site_id	site_domain	site_category	app_id	app_domain	...	C14	C15	C16	C17	C18	C19	C20	C21	day_of_week
0	-1636923355	0	2014-10-21	1005	0	85f751fd	c4e18dd6	50e219e0	e2a1ca37	2347f47a	...	15708	320	50	1722	0	35	-1	79	Tuesday
1	-193497663	0	2014-10-21	1005	1	e151e245	7e091613	f028772b	ecad2386	7801e8d9	...	17747	320	50	1974	2	39	100021	33	Tuesday
2	1315205890	0	2014-10-21	1002	0	85f751fd	c4e18dd6	50e219e0	a37bf1e4	7801e8d9	...	21691	320	50	2495	2	167	-1	23	Tuesday
3	1336077603	0	2014-10-21	1005	0	1fbe01fe	f3845767	28905ebd	ecad2386	7801e8d9	...	15701	320	50	1722	0	35	-1	79	Tuesday
4	-1368186722	1	2014-10-21	1005	0	1fbe01fe	f3845767	28905ebd	ecad2386	7801e8d9	...	15708	320	50	1722	0	35	100084	79	Tuesday

5 rows × 26 columns

下面我们要将hash函数将类型为object的变量映射成integer型

In [36]:

def convert_obj_to_int(self):
    
    object_list_columns = self.columns
    object_list_dtypes = self.dtypes
    new_col_suffix = '_int'
    for index in range(0,len(object_list_columns)):
        if object_list_dtypes[index] == object :
            self[object_list_columns[index]+new_col_suffix] = self[object_list_columns[index]].map( lambda  x: hash(x))
            self.drop([object_list_columns[index]],inplace=True,axis=1)
    return self
train = convert_obj_to_int(train)
train.head()

Out[36]:

	id	click	hour	C1	banner_pos	device_type	C14	C15	C16	...	site_id_int	site_domain_int	site_category_int	app_id_int	app_domain_int	app_category_int	device_id_int	device_ip_int	device_model_int	day_of_week_int
0	-1636923355	0	2014-10-21	1005	0	1	15708	320	50	...	3294295106353209792	-7871480762403810028	-1726633989484689459	3967604608139905137	-387442148922046908	3587892912659238008	-6860200936241454220	-7863861599293986987	8122318252297648011	-6056180354402690469
1	-193497663	0	2014-10-21	1005	1	1	17747	320	50	...	-1088018682312011499	-8030684975096413899	-7766034975137587207	-5153835905196929001	9059885572487474882	5089309385467275041	5788537550175848951	588395654717179763	7072708345363904469	-6056180354402690469
2	1315205890	0	2014-10-21	1002	0	0	21691	320	50	...	3294295106353209792	-7871480762403810028	-1726633989484689459	5064714988463203132	9059885572487474882	5089309385467275041	-8229604351924704167	8009012252837729359	-6538374976293708962	-6056180354402690469
3	1336077603	0	2014-10-21	1005	0	1	15701	320	50	...	4322696643391637070	-3123241050271775153	7354276305300575896	-5153835905196929001	9059885572487474882	5089309385467275041	5788537550175848951	-5845137796421209069	-3805608367665034636	-6056180354402690469
4	-1368186722	1	2014-10-21	1005	0	1	15708	320	50	...	4322696643391637070	-3123241050271775153	7354276305300575896	-5153835905196929001	9059885572487474882	5089309385467275041	5788537550175848951	8085682286269281170	7088212619388353806	-6056180354402690469

5 rows × 26 columns

LightGBM 模型

LightGBM是个快速的，分布式的，高性能的基于决策树算法的梯度提升框架。可用于排序，分类，回归以及很多其他的机器学习任务中,接下来我们要使用LightGBM作为我们的分类模型.

In [37]:

train.drop('hour', axis=1, inplace=True)
train.drop('id', axis=1, inplace=True)

In [38]:

X_train = train.loc[:, train.columns != 'click']
y_target = train.click.values

msk = np.random.rand(len(X_train)) < 0.8
lgb_train = lgb.Dataset(X_train[msk], y_target[msk])
lgb_eval = lgb.Dataset(X_train[~msk], y_target[~msk], reference=lgb_train)

In [39]:

# 配置模型参数
params = {
    'task': 'train',
    'boosting_type': 'gbdt',
    'objective': 'binary',
    'metric': { 'binary_logloss'},
    'num_leaves': 31, # 每棵树的默认叶子数
    'learning_rate': 0.08,
    'feature_fraction': 0.7, # 将在训练每棵树之前选择70％的特征
    'bagging_fraction': 0.3, #随机选择30%的特征。
    'bagging_freq': 5, #  每5次迭代执行bagging
    'verbose': 0
}

print('开始训练...')

gbm = lgb.train(params,
                lgb_train,
                num_boost_round=4000,
                valid_sets=lgb_eval,
                early_stopping_rounds=500)

开始训练...
[1]	valid_0's binary_logloss: 0.448408
Training until validation scores don't improve for 500 rounds.
[2]	valid_0's binary_logloss: 0.444125
[3]	valid_0's binary_logloss: 0.440432
[4]	valid_0's binary_logloss: 0.437069
[5]	valid_0's binary_logloss: 0.434273
[6]	valid_0's binary_logloss: 0.431874
[7]	valid_0's binary_logloss: 0.429536
[8]	valid_0's binary_logloss: 0.427666
[9]	valid_0's binary_logloss: 0.426059
[10]	valid_0's binary_logloss: 0.424549
[11]	valid_0's binary_logloss: 0.423223
[12]	valid_0's binary_logloss: 0.422081
[13]	valid_0's binary_logloss: 0.420984
[14]	valid_0's binary_logloss: 0.420014
[15]	valid_0's binary_logloss: 0.419035
[16]	valid_0's binary_logloss: 0.418202
[17]	valid_0's binary_logloss: 0.417401
[18]	valid_0's binary_logloss: 0.416707
[19]	valid_0's binary_logloss: 0.416105
[20]	valid_0's binary_logloss: 0.415556
[21]	valid_0's binary_logloss: 0.414978
[22]	valid_0's binary_logloss: 0.414532
[23]	valid_0's binary_logloss: 0.414154
[24]	valid_0's binary_logloss: 0.413748
[25]	valid_0's binary_logloss: 0.413313
[26]	valid_0's binary_logloss: 0.412819
[27]	valid_0's binary_logloss: 0.412523
[28]	valid_0's binary_logloss: 0.412208
[29]	valid_0's binary_logloss: 0.411876
[30]	valid_0's binary_logloss: 0.41158
[31]	valid_0's binary_logloss: 0.411332
[32]	valid_0's binary_logloss: 0.41108
[33]	valid_0's binary_logloss: 0.410797
[34]	valid_0's binary_logloss: 0.410582
[35]	valid_0's binary_logloss: 0.410326
[36]	valid_0's binary_logloss: 0.410076
[37]	valid_0's binary_logloss: 0.409782
[38]	valid_0's binary_logloss: 0.409591
[39]	valid_0's binary_logloss: 0.409389
[40]	valid_0's binary_logloss: 0.409214
[41]	valid_0's binary_logloss: 0.409087
[42]	valid_0's binary_logloss: 0.408875
[43]	valid_0's binary_logloss: 0.408695
[44]	valid_0's binary_logloss: 0.408544
[45]	valid_0's binary_logloss: 0.408372
[46]	valid_0's binary_logloss: 0.408143
[47]	valid_0's binary_logloss: 0.408052
[48]	valid_0's binary_logloss: 0.407944
[49]	valid_0's binary_logloss: 0.407811
[50]	valid_0's binary_logloss: 0.407675
[51]	valid_0's binary_logloss: 0.407503
[52]	valid_0's binary_logloss: 0.40729
[53]	valid_0's binary_logloss: 0.407154
[54]	valid_0's binary_logloss: 0.407029
[55]	valid_0's binary_logloss: 0.406929
[56]	valid_0's binary_logloss: 0.406796
[57]	valid_0's binary_logloss: 0.406662
[58]	valid_0's binary_logloss: 0.40663
[59]	valid_0's binary_logloss: 0.406464
[60]	valid_0's binary_logloss: 0.406347
[61]	valid_0's binary_logloss: 0.406287
[62]	valid_0's binary_logloss: 0.406233
[63]	valid_0's binary_logloss: 0.406154
[64]	valid_0's binary_logloss: 0.406007
[65]	valid_0's binary_logloss: 0.405914
[66]	valid_0's binary_logloss: 0.405831
[67]	valid_0's binary_logloss: 0.405754
[68]	valid_0's binary_logloss: 0.405688
[69]	valid_0's binary_logloss: 0.405645
[70]	valid_0's binary_logloss: 0.405567
[71]	valid_0's binary_logloss: 0.405448
[72]	valid_0's binary_logloss: 0.405379
[73]	valid_0's binary_logloss: 0.405272
[74]	valid_0's binary_logloss: 0.405226
[75]	valid_0's binary_logloss: 0.405185
[76]	valid_0's binary_logloss: 0.405085
[77]	valid_0's binary_logloss: 0.405022
[78]	valid_0's binary_logloss: 0.404936
[79]	valid_0's binary_logloss: 0.404886
[80]	valid_0's binary_logloss: 0.404862
[81]	valid_0's binary_logloss: 0.404814
[82]	valid_0's binary_logloss: 0.404761
[83]	valid_0's binary_logloss: 0.404703
[84]	valid_0's binary_logloss: 0.404646
[85]	valid_0's binary_logloss: 0.404612
[86]	valid_0's binary_logloss: 0.404498
[87]	valid_0's binary_logloss: 0.404432
[88]	valid_0's binary_logloss: 0.404359
[89]	valid_0's binary_logloss: 0.404282
[90]	valid_0's binary_logloss: 0.404239
[91]	valid_0's binary_logloss: 0.404184
[92]	valid_0's binary_logloss: 0.404149
[93]	valid_0's binary_logloss: 0.404099
[94]	valid_0's binary_logloss: 0.404093
[95]	valid_0's binary_logloss: 0.404057
[96]	valid_0's binary_logloss: 0.404024
[97]	valid_0's binary_logloss: 0.403958
[98]	valid_0's binary_logloss: 0.403943
[99]	valid_0's binary_logloss: 0.403918
[100]	valid_0's binary_logloss: 0.403917
[101]	valid_0's binary_logloss: 0.403856
[102]	valid_0's binary_logloss: 0.403835
[103]	valid_0's binary_logloss: 0.403773
[104]	valid_0's binary_logloss: 0.403742
[105]	valid_0's binary_logloss: 0.403706
[106]	valid_0's binary_logloss: 0.403683
[107]	valid_0's binary_logloss: 0.403656
[108]	valid_0's binary_logloss: 0.403606
[109]	valid_0's binary_logloss: 0.403565
[110]	valid_0's binary_logloss: 0.403531
[111]	valid_0's binary_logloss: 0.403492
[112]	valid_0's binary_logloss: 0.403453
[113]	valid_0's binary_logloss: 0.403424
[114]	valid_0's binary_logloss: 0.403385
[115]	valid_0's binary_logloss: 0.403342
[116]	valid_0's binary_logloss: 0.403292
[117]	valid_0's binary_logloss: 0.403246
[118]	valid_0's binary_logloss: 0.403216
[119]	valid_0's binary_logloss: 0.403195
[120]	valid_0's binary_logloss: 0.403154
[121]	valid_0's binary_logloss: 0.403111
[122]	valid_0's binary_logloss: 0.403079
[123]	valid_0's binary_logloss: 0.403057
[124]	valid_0's binary_logloss: 0.403024
[125]	valid_0's binary_logloss: 0.402983
[126]	valid_0's binary_logloss: 0.402942
[127]	valid_0's binary_logloss: 0.402906
[128]	valid_0's binary_logloss: 0.402891
[129]	valid_0's binary_logloss: 0.402866
[130]	valid_0's binary_logloss: 0.402841
[131]	valid_0's binary_logloss: 0.402794
[132]	valid_0's binary_logloss: 0.402766
[133]	valid_0's binary_logloss: 0.402736
[134]	valid_0's binary_logloss: 0.402719
[135]	valid_0's binary_logloss: 0.402699
[136]	valid_0's binary_logloss: 0.402665
[137]	valid_0's binary_logloss: 0.40265
[138]	valid_0's binary_logloss: 0.402634
[139]	valid_0's binary_logloss: 0.402608
[140]	valid_0's binary_logloss: 0.402595
[141]	valid_0's binary_logloss: 0.402549
[142]	valid_0's binary_logloss: 0.402528
[143]	valid_0's binary_logloss: 0.402508
[144]	valid_0's binary_logloss: 0.402472
[145]	valid_0's binary_logloss: 0.402448
[146]	valid_0's binary_logloss: 0.402432
[147]	valid_0's binary_logloss: 0.402396
[148]	valid_0's binary_logloss: 0.402361
[149]	valid_0's binary_logloss: 0.402335
[150]	valid_0's binary_logloss: 0.402324
[151]	valid_0's binary_logloss: 0.402321
[152]	valid_0's binary_logloss: 0.40228
[153]	valid_0's binary_logloss: 0.402251
[154]	valid_0's binary_logloss: 0.402232
[155]	valid_0's binary_logloss: 0.402221
[156]	valid_0's binary_logloss: 0.402225
[157]	valid_0's binary_logloss: 0.402211
[158]	valid_0's binary_logloss: 0.40217
[159]	valid_0's binary_logloss: 0.40216
[160]	valid_0's binary_logloss: 0.402166
[161]	valid_0's binary_logloss: 0.402155
[162]	valid_0's binary_logloss: 0.402144
[163]	valid_0's binary_logloss: 0.402112
[164]	valid_0's binary_logloss: 0.402083
[165]	valid_0's binary_logloss: 0.402072
[166]	valid_0's binary_logloss: 0.402067
[167]	valid_0's binary_logloss: 0.402039
[168]	valid_0's binary_logloss: 0.402002
[169]	valid_0's binary_logloss: 0.40199
[170]	valid_0's binary_logloss: 0.401963
[171]	valid_0's binary_logloss: 0.40194
[172]	valid_0's binary_logloss: 0.40192
[173]	valid_0's binary_logloss: 0.401903
[174]	valid_0's binary_logloss: 0.401901
[175]	valid_0's binary_logloss: 0.401873
[176]	valid_0's binary_logloss: 0.401832
[177]	valid_0's binary_logloss: 0.401813
[178]	valid_0's binary_logloss: 0.401782
[179]	valid_0's binary_logloss: 0.401741
[180]	valid_0's binary_logloss: 0.401722
[181]	valid_0's binary_logloss: 0.401716
[182]	valid_0's binary_logloss: 0.401698
[183]	valid_0's binary_logloss: 0.40171
[184]	valid_0's binary_logloss: 0.401712
[185]	valid_0's binary_logloss: 0.401708
[186]	valid_0's binary_logloss: 0.401681
[187]	valid_0's binary_logloss: 0.401682
[188]	valid_0's binary_logloss: 0.401678
[189]	valid_0's binary_logloss: 0.401685
[190]	valid_0's binary_logloss: 0.401663
[191]	valid_0's binary_logloss: 0.401671
[192]	valid_0's binary_logloss: 0.401662
[193]	valid_0's binary_logloss: 0.401637
[194]	valid_0's binary_logloss: 0.401644
[195]	valid_0's binary_logloss: 0.401661
[196]	valid_0's binary_logloss: 0.401661
[197]	valid_0's binary_logloss: 0.401657
[198]	valid_0's binary_logloss: 0.401654
[199]	valid_0's binary_logloss: 0.401609
[200]	valid_0's binary_logloss: 0.401596
[201]	valid_0's binary_logloss: 0.40157
[202]	valid_0's binary_logloss: 0.401561
[203]	valid_0's binary_logloss: 0.401548
[204]	valid_0's binary_logloss: 0.401547
[205]	valid_0's binary_logloss: 0.401536
[206]	valid_0's binary_logloss: 0.401518
[207]	valid_0's binary_logloss: 0.401496
[208]	valid_0's binary_logloss: 0.40147
[209]	valid_0's binary_logloss: 0.40146
[210]	valid_0's binary_logloss: 0.401459
[211]	valid_0's binary_logloss: 0.401444
[212]	valid_0's binary_logloss: 0.401421
[213]	valid_0's binary_logloss: 0.40139
[214]	valid_0's binary_logloss: 0.401369
[215]	valid_0's binary_logloss: 0.40134
[216]	valid_0's binary_logloss: 0.401326
[217]	valid_0's binary_logloss: 0.401297
[218]	valid_0's binary_logloss: 0.401269
[219]	valid_0's binary_logloss: 0.401262
[220]	valid_0's binary_logloss: 0.401248
[221]	valid_0's binary_logloss: 0.401244
[222]	valid_0's binary_logloss: 0.401197
[223]	valid_0's binary_logloss: 0.401158
[224]	valid_0's binary_logloss: 0.401161
[225]	valid_0's binary_logloss: 0.401165
[226]	valid_0's binary_logloss: 0.401155
[227]	valid_0's binary_logloss: 0.401115
[228]	valid_0's binary_logloss: 0.401108
[229]	valid_0's binary_logloss: 0.401084
[230]	valid_0's binary_logloss: 0.401072
[231]	valid_0's binary_logloss: 0.401057
[232]	valid_0's binary_logloss: 0.401036
[233]	valid_0's binary_logloss: 0.401023
[234]	valid_0's binary_logloss: 0.401003
[235]	valid_0's binary_logloss: 0.400999
[236]	valid_0's binary_logloss: 0.40098
[237]	valid_0's binary_logloss: 0.400979
[238]	valid_0's binary_logloss: 0.400955
[239]	valid_0's binary_logloss: 0.400947
[240]	valid_0's binary_logloss: 0.400923
[241]	valid_0's binary_logloss: 0.400917
[242]	valid_0's binary_logloss: 0.400911
[243]	valid_0's binary_logloss: 0.400907
[244]	valid_0's binary_logloss: 0.400906
[245]	valid_0's binary_logloss: 0.400915
[246]	valid_0's binary_logloss: 0.400885
[247]	valid_0's binary_logloss: 0.400871
[248]	valid_0's binary_logloss: 0.40087
[249]	valid_0's binary_logloss: 0.400867
[250]	valid_0's binary_logloss: 0.400862
[251]	valid_0's binary_logloss: 0.400853
[252]	valid_0's binary_logloss: 0.400838
[253]	valid_0's binary_logloss: 0.40084
[254]	valid_0's binary_logloss: 0.40084
[255]	valid_0's binary_logloss: 0.400828
[256]	valid_0's binary_logloss: 0.400806
[257]	valid_0's binary_logloss: 0.400787
[258]	valid_0's binary_logloss: 0.400767
[259]	valid_0's binary_logloss: 0.400748
[260]	valid_0's binary_logloss: 0.400721
[261]	valid_0's binary_logloss: 0.400708
[262]	valid_0's binary_logloss: 0.400706
[263]	valid_0's binary_logloss: 0.400708
[264]	valid_0's binary_logloss: 0.400706
[265]	valid_0's binary_logloss: 0.400711
[266]	valid_0's binary_logloss: 0.400712
[267]	valid_0's binary_logloss: 0.400704
[268]	valid_0's binary_logloss: 0.4007
[269]	valid_0's binary_logloss: 0.400671
[270]	valid_0's binary_logloss: 0.400668
[271]	valid_0's binary_logloss: 0.40065
[272]	valid_0's binary_logloss: 0.400624
[273]	valid_0's binary_logloss: 0.400602
[274]	valid_0's binary_logloss: 0.400588
[275]	valid_0's binary_logloss: 0.400578
[276]	valid_0's binary_logloss: 0.400555
[277]	valid_0's binary_logloss: 0.400544
[278]	valid_0's binary_logloss: 0.400547
[279]	valid_0's binary_logloss: 0.400548
[280]	valid_0's binary_logloss: 0.400546
[281]	valid_0's binary_logloss: 0.400516
[282]	valid_0's binary_logloss: 0.4005
[283]	valid_0's binary_logloss: 0.400481
[284]	valid_0's binary_logloss: 0.400461
[285]	valid_0's binary_logloss: 0.400466
[286]	valid_0's binary_logloss: 0.400443
[287]	valid_0's binary_logloss: 0.400426
[288]	valid_0's binary_logloss: 0.400401
[289]	valid_0's binary_logloss: 0.400394
[290]	valid_0's binary_logloss: 0.400388
[291]	valid_0's binary_logloss: 0.400384
[292]	valid_0's binary_logloss: 0.400375
[293]	valid_0's binary_logloss: 0.400379
[294]	valid_0's binary_logloss: 0.400356
[295]	valid_0's binary_logloss: 0.400328
[296]	valid_0's binary_logloss: 0.400312
[297]	valid_0's binary_logloss: 0.400293
[298]	valid_0's binary_logloss: 0.400257
[299]	valid_0's binary_logloss: 0.400253
[300]	valid_0's binary_logloss: 0.400254
[301]	valid_0's binary_logloss: 0.400243
[302]	valid_0's binary_logloss: 0.400243
[303]	valid_0's binary_logloss: 0.400233
[304]	valid_0's binary_logloss: 0.400229
[305]	valid_0's binary_logloss: 0.400225
[306]	valid_0's binary_logloss: 0.400219
[307]	valid_0's binary_logloss: 0.400195
[308]	valid_0's binary_logloss: 0.400179
[309]	valid_0's binary_logloss: 0.400159
[310]	valid_0's binary_logloss: 0.400168
[311]	valid_0's binary_logloss: 0.400156
[312]	valid_0's binary_logloss: 0.400155
[313]	valid_0's binary_logloss: 0.400151
[314]	valid_0's binary_logloss: 0.400152
[315]	valid_0's binary_logloss: 0.400152
[316]	valid_0's binary_logloss: 0.400138
[317]	valid_0's binary_logloss: 0.400128
[318]	valid_0's binary_logloss: 0.400102
[319]	valid_0's binary_logloss: 0.400096
[320]	valid_0's binary_logloss: 0.40009
[321]	valid_0's binary_logloss: 0.400085
[322]	valid_0's binary_logloss: 0.400072
[323]	valid_0's binary_logloss: 0.400053
[324]	valid_0's binary_logloss: 0.400049
[325]	valid_0's binary_logloss: 0.400037
[326]	valid_0's binary_logloss: 0.400013
[327]	valid_0's binary_logloss: 0.400012
[328]	valid_0's binary_logloss: 0.400009
[329]	valid_0's binary_logloss: 0.400004
[330]	valid_0's binary_logloss: 0.400001
[331]	valid_0's binary_logloss: 0.399997
[332]	valid_0's binary_logloss: 0.399978
[333]	valid_0's binary_logloss: 0.399971
[334]	valid_0's binary_logloss: 0.399958
[335]	valid_0's binary_logloss: 0.399952
[336]	valid_0's binary_logloss: 0.399949
[337]	valid_0's binary_logloss: 0.399945
[338]	valid_0's binary_logloss: 0.399939
[339]	valid_0's binary_logloss: 0.39994
[340]	valid_0's binary_logloss: 0.399937
[341]	valid_0's binary_logloss: 0.39993
[342]	valid_0's binary_logloss: 0.399933
[343]	valid_0's binary_logloss: 0.399937
[344]	valid_0's binary_logloss: 0.399937
[345]	valid_0's binary_logloss: 0.399937
[346]	valid_0's binary_logloss: 0.399919
[347]	valid_0's binary_logloss: 0.39992
[348]	valid_0's binary_logloss: 0.399917
[349]	valid_0's binary_logloss: 0.399919
[350]	valid_0's binary_logloss: 0.399928
[351]	valid_0's binary_logloss: 0.399925
[352]	valid_0's binary_logloss: 0.399916
[353]	valid_0's binary_logloss: 0.399904
[354]	valid_0's binary_logloss: 0.399911
[355]	valid_0's binary_logloss: 0.39992
[356]	valid_0's binary_logloss: 0.399933
[357]	valid_0's binary_logloss: 0.399929
[358]	valid_0's binary_logloss: 0.399924
[359]	valid_0's binary_logloss: 0.399925
[360]	valid_0's binary_logloss: 0.399931
[361]	valid_0's binary_logloss: 0.399913
[362]	valid_0's binary_logloss: 0.399904
[363]	valid_0's binary_logloss: 0.399892
[364]	valid_0's binary_logloss: 0.3999
[365]	valid_0's binary_logloss: 0.399895
[366]	valid_0's binary_logloss: 0.399887
[367]	valid_0's binary_logloss: 0.399884
[368]	valid_0's binary_logloss: 0.399879
[369]	valid_0's binary_logloss: 0.399877
[370]	valid_0's binary_logloss: 0.39988
[371]	valid_0's binary_logloss: 0.399875
[372]	valid_0's binary_logloss: 0.399853
[373]	valid_0's binary_logloss: 0.399841
[374]	valid_0's binary_logloss: 0.399827
[375]	valid_0's binary_logloss: 0.399826
[376]	valid_0's binary_logloss: 0.399817
[377]	valid_0's binary_logloss: 0.399797
[378]	valid_0's binary_logloss: 0.3998
[379]	valid_0's binary_logloss: 0.399805
[380]	valid_0's binary_logloss: 0.399788
[381]	valid_0's binary_logloss: 0.399787
[382]	valid_0's binary_logloss: 0.39979
[383]	valid_0's binary_logloss: 0.399788
[384]	valid_0's binary_logloss: 0.399786
[385]	valid_0's binary_logloss: 0.399786
[386]	valid_0's binary_logloss: 0.399788
[387]	valid_0's binary_logloss: 0.399774
[388]	valid_0's binary_logloss: 0.399775
[389]	valid_0's binary_logloss: 0.399776
[390]	valid_0's binary_logloss: 0.399781
[391]	valid_0's binary_logloss: 0.399779
[392]	valid_0's binary_logloss: 0.399766
[393]	valid_0's binary_logloss: 0.399762
[394]	valid_0's binary_logloss: 0.399757
[395]	valid_0's binary_logloss: 0.399769
[396]	valid_0's binary_logloss: 0.399762
[397]	valid_0's binary_logloss: 0.399753
[398]	valid_0's binary_logloss: 0.399748
[399]	valid_0's binary_logloss: 0.399742
[400]	valid_0's binary_logloss: 0.39973
[401]	valid_0's binary_logloss: 0.399726
[402]	valid_0's binary_logloss: 0.39972
[403]	valid_0's binary_logloss: 0.399705
[404]	valid_0's binary_logloss: 0.399686
[405]	valid_0's binary_logloss: 0.399679
[406]	valid_0's binary_logloss: 0.399671
[407]	valid_0's binary_logloss: 0.399662
[408]	valid_0's binary_logloss: 0.399655
[409]	valid_0's binary_logloss: 0.399649
[410]	valid_0's binary_logloss: 0.399641
[411]	valid_0's binary_logloss: 0.399641
[412]	valid_0's binary_logloss: 0.399636
[413]	valid_0's binary_logloss: 0.399644
[414]	valid_0's binary_logloss: 0.399641
[415]	valid_0's binary_logloss: 0.399641
[416]	valid_0's binary_logloss: 0.399628
[417]	valid_0's binary_logloss: 0.399625
[418]	valid_0's binary_logloss: 0.399627
[419]	valid_0's binary_logloss: 0.399635
[420]	valid_0's binary_logloss: 0.399623
[421]	valid_0's binary_logloss: 0.399611
[422]	valid_0's binary_logloss: 0.399615
[423]	valid_0's binary_logloss: 0.399613
[424]	valid_0's binary_logloss: 0.399608
[425]	valid_0's binary_logloss: 0.399591
[426]	valid_0's binary_logloss: 0.399596
[427]	valid_0's binary_logloss: 0.399601
[428]	valid_0's binary_logloss: 0.399602
[429]	valid_0's binary_logloss: 0.399592
[430]	valid_0's binary_logloss: 0.399601
[431]	valid_0's binary_logloss: 0.399606
[432]	valid_0's binary_logloss: 0.399603
[433]	valid_0's binary_logloss: 0.399603
[434]	valid_0's binary_logloss: 0.399614
[435]	valid_0's binary_logloss: 0.39961
[436]	valid_0's binary_logloss: 0.39961
[437]	valid_0's binary_logloss: 0.399587
[438]	valid_0's binary_logloss: 0.399596
[439]	valid_0's binary_logloss: 0.39959
[440]	valid_0's binary_logloss: 0.399591
[441]	valid_0's binary_logloss: 0.399589
[442]	valid_0's binary_logloss: 0.399577
[443]	valid_0's binary_logloss: 0.399574
[444]	valid_0's binary_logloss: 0.39958
[445]	valid_0's binary_logloss: 0.399568
[446]	valid_0's binary_logloss: 0.399569
[447]	valid_0's binary_logloss: 0.399556
[448]	valid_0's binary_logloss: 0.399552
[449]	valid_0's binary_logloss: 0.399554
[450]	valid_0's binary_logloss: 0.399545
[451]	valid_0's binary_logloss: 0.39953
[452]	valid_0's binary_logloss: 0.399522
[453]	valid_0's binary_logloss: 0.399512
[454]	valid_0's binary_logloss: 0.399507
[455]	valid_0's binary_logloss: 0.399496
[456]	valid_0's binary_logloss: 0.39948
[457]	valid_0's binary_logloss: 0.399477
[458]	valid_0's binary_logloss: 0.399475
[459]	valid_0's binary_logloss: 0.399475
[460]	valid_0's binary_logloss: 0.399467
[461]	valid_0's binary_logloss: 0.399457
[462]	valid_0's binary_logloss: 0.399457
[463]	valid_0's binary_logloss: 0.399447
[464]	valid_0's binary_logloss: 0.399452
[465]	valid_0's binary_logloss: 0.399445
[466]	valid_0's binary_logloss: 0.399446
[467]	valid_0's binary_logloss: 0.399415
[468]	valid_0's binary_logloss: 0.399386
[469]	valid_0's binary_logloss: 0.399371
[470]	valid_0's binary_logloss: 0.399348
[471]	valid_0's binary_logloss: 0.399343
[472]	valid_0's binary_logloss: 0.399351
[473]	valid_0's binary_logloss: 0.399352
[474]	valid_0's binary_logloss: 0.399348
[475]	valid_0's binary_logloss: 0.399351
[476]	valid_0's binary_logloss: 0.399352
[477]	valid_0's binary_logloss: 0.399341
[478]	valid_0's binary_logloss: 0.399317
[479]	valid_0's binary_logloss: 0.399308
[480]	valid_0's binary_logloss: 0.39929
[481]	valid_0's binary_logloss: 0.399296
[482]	valid_0's binary_logloss: 0.399293
[483]	valid_0's binary_logloss: 0.399294
[484]	valid_0's binary_logloss: 0.399286
[485]	valid_0's binary_logloss: 0.399294
[486]	valid_0's binary_logloss: 0.399288
[487]	valid_0's binary_logloss: 0.399285
[488]	valid_0's binary_logloss: 0.39929
[489]	valid_0's binary_logloss: 0.399286
[490]	valid_0's binary_logloss: 0.399293
[491]	valid_0's binary_logloss: 0.399281
[492]	valid_0's binary_logloss: 0.399279
[493]	valid_0's binary_logloss: 0.399275
[494]	valid_0's binary_logloss: 0.399278
[495]	valid_0's binary_logloss: 0.399276
[496]	valid_0's binary_logloss: 0.399276
[497]	valid_0's binary_logloss: 0.39927
[498]	valid_0's binary_logloss: 0.39927
[499]	valid_0's binary_logloss: 0.399277
[500]	valid_0's binary_logloss: 0.399278
[501]	valid_0's binary_logloss: 0.399273
[502]	valid_0's binary_logloss: 0.399265
[503]	valid_0's binary_logloss: 0.399271
[504]	valid_0's binary_logloss: 0.399261
[505]	valid_0's binary_logloss: 0.399244
[506]	valid_0's binary_logloss: 0.399253
[507]	valid_0's binary_logloss: 0.399265
[508]	valid_0's binary_logloss: 0.39926
[509]	valid_0's binary_logloss: 0.399255
[510]	valid_0's binary_logloss: 0.399262
[511]	valid_0's binary_logloss: 0.399245
[512]	valid_0's binary_logloss: 0.399231
[513]	valid_0's binary_logloss: 0.39922
[514]	valid_0's binary_logloss: 0.399205
[515]	valid_0's binary_logloss: 0.399201
[516]	valid_0's binary_logloss: 0.399208
[517]	valid_0's binary_logloss: 0.399203
[518]	valid_0's binary_logloss: 0.39921
[519]	valid_0's binary_logloss: 0.399195
[520]	valid_0's binary_logloss: 0.3992
[521]	valid_0's binary_logloss: 0.399201
[522]	valid_0's binary_logloss: 0.399192
[523]	valid_0's binary_logloss: 0.399185
[524]	valid_0's binary_logloss: 0.399179
[525]	valid_0's binary_logloss: 0.399182
[526]	valid_0's binary_logloss: 0.39918
[527]	valid_0's binary_logloss: 0.399178
[528]	valid_0's binary_logloss: 0.399169
[529]	valid_0's binary_logloss: 0.399163
[530]	valid_0's binary_logloss: 0.39916
[531]	valid_0's binary_logloss: 0.399161
[532]	valid_0's binary_logloss: 0.399166
[533]	valid_0's binary_logloss: 0.399174
[534]	valid_0's binary_logloss: 0.399173
[535]	valid_0's binary_logloss: 0.399175
[536]	valid_0's binary_logloss: 0.399172
[537]	valid_0's binary_logloss: 0.399151
[538]	valid_0's binary_logloss: 0.399147
[539]	valid_0's binary_logloss: 0.399147
[540]	valid_0's binary_logloss: 0.399139
[541]	valid_0's binary_logloss: 0.39914
[542]	valid_0's binary_logloss: 0.399143
[543]	valid_0's binary_logloss: 0.399144
[544]	valid_0's binary_logloss: 0.399142
[545]	valid_0's binary_logloss: 0.399142
[546]	valid_0's binary_logloss: 0.399129
[547]	valid_0's binary_logloss: 0.399131
[548]	valid_0's binary_logloss: 0.39913
[549]	valid_0's binary_logloss: 0.399132
[550]	valid_0's binary_logloss: 0.39913
[551]	valid_0's binary_logloss: 0.399128
[552]	valid_0's binary_logloss: 0.39914
[553]	valid_0's binary_logloss: 0.399134
[554]	valid_0's binary_logloss: 0.399136
[555]	valid_0's binary_logloss: 0.399146
[556]	valid_0's binary_logloss: 0.399139
[557]	valid_0's binary_logloss: 0.399146
[558]	valid_0's binary_logloss: 0.399146
[559]	valid_0's binary_logloss: 0.399121
[560]	valid_0's binary_logloss: 0.399108
[561]	valid_0's binary_logloss: 0.399101
[562]	valid_0's binary_logloss: 0.399096
[563]	valid_0's binary_logloss: 0.3991
[564]	valid_0's binary_logloss: 0.399104
[565]	valid_0's binary_logloss: 0.399096
[566]	valid_0's binary_logloss: 0.399098
[567]	valid_0's binary_logloss: 0.399099
[568]	valid_0's binary_logloss: 0.399089
[569]	valid_0's binary_logloss: 0.399083
[570]	valid_0's binary_logloss: 0.399077
[571]	valid_0's binary_logloss: 0.399067
[572]	valid_0's binary_logloss: 0.39907
[573]	valid_0's binary_logloss: 0.399065
[574]	valid_0's binary_logloss: 0.399066
[575]	valid_0's binary_logloss: 0.399066
[576]	valid_0's binary_logloss: 0.399062
[577]	valid_0's binary_logloss: 0.399057
[578]	valid_0's binary_logloss: 0.399044
[579]	valid_0's binary_logloss: 0.399054
[580]	valid_0's binary_logloss: 0.39905
[581]	valid_0's binary_logloss: 0.399044
[582]	valid_0's binary_logloss: 0.399043
[583]	valid_0's binary_logloss: 0.399034
[584]	valid_0's binary_logloss: 0.399036
[585]	valid_0's binary_logloss: 0.399034
[586]	valid_0's binary_logloss: 0.39903
[587]	valid_0's binary_logloss: 0.399028
[588]	valid_0's binary_logloss: 0.399028
[589]	valid_0's binary_logloss: 0.399026
[590]	valid_0's binary_logloss: 0.399031
[591]	valid_0's binary_logloss: 0.399034
[592]	valid_0's binary_logloss: 0.399023
[593]	valid_0's binary_logloss: 0.39902
[594]	valid_0's binary_logloss: 0.399021
[595]	valid_0's binary_logloss: 0.399034
[596]	valid_0's binary_logloss: 0.399024
[597]	valid_0's binary_logloss: 0.399017
[598]	valid_0's binary_logloss: 0.399021
[599]	valid_0's binary_logloss: 0.39901
[600]	valid_0's binary_logloss: 0.399012
[601]	valid_0's binary_logloss: 0.399014
[602]	valid_0's binary_logloss: 0.399025
[603]	valid_0's binary_logloss: 0.399016
[604]	valid_0's binary_logloss: 0.399012
[605]	valid_0's binary_logloss: 0.399023
[606]	valid_0's binary_logloss: 0.399024
[607]	valid_0's binary_logloss: 0.399021
[608]	valid_0's binary_logloss: 0.39903
[609]	valid_0's binary_logloss: 0.399039
[610]	valid_0's binary_logloss: 0.399038
[611]	valid_0's binary_logloss: 0.399036
[612]	valid_0's binary_logloss: 0.39904
[613]	valid_0's binary_logloss: 0.399036
[614]	valid_0's binary_logloss: 0.399034
[615]	valid_0's binary_logloss: 0.39903
[616]	valid_0's binary_logloss: 0.399029
[617]	valid_0's binary_logloss: 0.399033
[618]	valid_0's binary_logloss: 0.399025
[619]	valid_0's binary_logloss: 0.399011
[620]	valid_0's binary_logloss: 0.399008
[621]	valid_0's binary_logloss: 0.399002
[622]	valid_0's binary_logloss: 0.399007
[623]	valid_0's binary_logloss: 0.398985
[624]	valid_0's binary_logloss: 0.398971
[625]	valid_0's binary_logloss: 0.398954
[626]	valid_0's binary_logloss: 0.398954
[627]	valid_0's binary_logloss: 0.398953
[628]	valid_0's binary_logloss: 0.39895
[629]	valid_0's binary_logloss: 0.398961
[630]	valid_0's binary_logloss: 0.398958
[631]	valid_0's binary_logloss: 0.398962
[632]	valid_0's binary_logloss: 0.398965
[633]	valid_0's binary_logloss: 0.398962
[634]	valid_0's binary_logloss: 0.398959
[635]	valid_0's binary_logloss: 0.398966
[636]	valid_0's binary_logloss: 0.398965
[637]	valid_0's binary_logloss: 0.398967
[638]	valid_0's binary_logloss: 0.398974
[639]	valid_0's binary_logloss: 0.398978
[640]	valid_0's binary_logloss: 0.398979
[641]	valid_0's binary_logloss: 0.398978
[642]	valid_0's binary_logloss: 0.398991
[643]	valid_0's binary_logloss: 0.398996
[644]	valid_0's binary_logloss: 0.398992
[645]	valid_0's binary_logloss: 0.398996
[646]	valid_0's binary_logloss: 0.398988
[647]	valid_0's binary_logloss: 0.398987
[648]	valid_0's binary_logloss: 0.398995
[649]	valid_0's binary_logloss: 0.398984
[650]	valid_0's binary_logloss: 0.398985
[651]	valid_0's binary_logloss: 0.39898
[652]	valid_0's binary_logloss: 0.39898
[653]	valid_0's binary_logloss: 0.398975
[654]	valid_0's binary_logloss: 0.398971
[655]	valid_0's binary_logloss: 0.398962
[656]	valid_0's binary_logloss: 0.398961
[657]	valid_0's binary_logloss: 0.398963
[658]	valid_0's binary_logloss: 0.398969
[659]	valid_0's binary_logloss: 0.398979
[660]	valid_0's binary_logloss: 0.398977
[661]	valid_0's binary_logloss: 0.39898
[662]	valid_0's binary_logloss: 0.398973
[663]	valid_0's binary_logloss: 0.398973
[664]	valid_0's binary_logloss: 0.398966
[665]	valid_0's binary_logloss: 0.398963
[666]	valid_0's binary_logloss: 0.398955
[667]	valid_0's binary_logloss: 0.398965
[668]	valid_0's binary_logloss: 0.398971
[669]	valid_0's binary_logloss: 0.398971
[670]	valid_0's binary_logloss: 0.398972
[671]	valid_0's binary_logloss: 0.398974
[672]	valid_0's binary_logloss: 0.398974
[673]	valid_0's binary_logloss: 0.398956
[674]	valid_0's binary_logloss: 0.398961
[675]	valid_0's binary_logloss: 0.398966
[676]	valid_0's binary_logloss: 0.398963
[677]	valid_0's binary_logloss: 0.398963
[678]	valid_0's binary_logloss: 0.398964
[679]	valid_0's binary_logloss: 0.398955
[680]	valid_0's binary_logloss: 0.398955
[681]	valid_0's binary_logloss: 0.398954
[682]	valid_0's binary_logloss: 0.39896
[683]	valid_0's binary_logloss: 0.398948
[684]	valid_0's binary_logloss: 0.398941
[685]	valid_0's binary_logloss: 0.398941
[686]	valid_0's binary_logloss: 0.398935
[687]	valid_0's binary_logloss: 0.398934
[688]	valid_0's binary_logloss: 0.398942
[689]	valid_0's binary_logloss: 0.398937
[690]	valid_0's binary_logloss: 0.39894
[691]	valid_0's binary_logloss: 0.39894
[692]	valid_0's binary_logloss: 0.398936
[693]	valid_0's binary_logloss: 0.398942
[694]	valid_0's binary_logloss: 0.398941
[695]	valid_0's binary_logloss: 0.398938
[696]	valid_0's binary_logloss: 0.398936
[697]	valid_0's binary_logloss: 0.398936
[698]	valid_0's binary_logloss: 0.398941
[699]	valid_0's binary_logloss: 0.398934
[700]	valid_0's binary_logloss: 0.398929
[701]	valid_0's binary_logloss: 0.398922
[702]	valid_0's binary_logloss: 0.398929
[703]	valid_0's binary_logloss: 0.398926
[704]	valid_0's binary_logloss: 0.398929
[705]	valid_0's binary_logloss: 0.398932
[706]	valid_0's binary_logloss: 0.398928
[707]	valid_0's binary_logloss: 0.398926
[708]	valid_0's binary_logloss: 0.398928
[709]	valid_0's binary_logloss: 0.398931
[710]	valid_0's binary_logloss: 0.398942
[711]	valid_0's binary_logloss: 0.398938
[712]	valid_0's binary_logloss: 0.398936
[713]	valid_0's binary_logloss: 0.398935
[714]	valid_0's binary_logloss: 0.398918
[715]	valid_0's binary_logloss: 0.398918
[716]	valid_0's binary_logloss: 0.39891
[717]	valid_0's binary_logloss: 0.398914
[718]	valid_0's binary_logloss: 0.398907
[719]	valid_0's binary_logloss: 0.398909
[720]	valid_0's binary_logloss: 0.3989
[721]	valid_0's binary_logloss: 0.398903
[722]	valid_0's binary_logloss: 0.398891
[723]	valid_0's binary_logloss: 0.398889
[724]	valid_0's binary_logloss: 0.398878
[725]	valid_0's binary_logloss: 0.398878
[726]	valid_0's binary_logloss: 0.398882
[727]	valid_0's binary_logloss: 0.398878
[728]	valid_0's binary_logloss: 0.398885
[729]	valid_0's binary_logloss: 0.398884
[730]	valid_0's binary_logloss: 0.398877
[731]	valid_0's binary_logloss: 0.398883
[732]	valid_0's binary_logloss: 0.398878
[733]	valid_0's binary_logloss: 0.398877
[734]	valid_0's binary_logloss: 0.39888
[735]	valid_0's binary_logloss: 0.398886
[736]	valid_0's binary_logloss: 0.398876
[737]	valid_0's binary_logloss: 0.398865
[738]	valid_0's binary_logloss: 0.398862
[739]	valid_0's binary_logloss: 0.398864
[740]	valid_0's binary_logloss: 0.398862
[741]	valid_0's binary_logloss: 0.398854
[742]	valid_0's binary_logloss: 0.398852
[743]	valid_0's binary_logloss: 0.398853
[744]	valid_0's binary_logloss: 0.39885
[745]	valid_0's binary_logloss: 0.398848
[746]	valid_0's binary_logloss: 0.398845
[747]	valid_0's binary_logloss: 0.398838
[748]	valid_0's binary_logloss: 0.398841
[749]	valid_0's binary_logloss: 0.398845
[750]	valid_0's binary_logloss: 0.398841
[751]	valid_0's binary_logloss: 0.398834
[752]	valid_0's binary_logloss: 0.398822
[753]	valid_0's binary_logloss: 0.398808
[754]	valid_0's binary_logloss: 0.398815
[755]	valid_0's binary_logloss: 0.398824
[756]	valid_0's binary_logloss: 0.39884
[757]	valid_0's binary_logloss: 0.398835
[758]	valid_0's binary_logloss: 0.398837
[759]	valid_0's binary_logloss: 0.398839
[760]	valid_0's binary_logloss: 0.398843
[761]	valid_0's binary_logloss: 0.398833
[762]	valid_0's binary_logloss: 0.39883
[763]	valid_0's binary_logloss: 0.398829
[764]	valid_0's binary_logloss: 0.398821
[765]	valid_0's binary_logloss: 0.398825
[766]	valid_0's binary_logloss: 0.398827
[767]	valid_0's binary_logloss: 0.398824
[768]	valid_0's binary_logloss: 0.398825
[769]	valid_0's binary_logloss: 0.398828
[770]	valid_0's binary_logloss: 0.398831
[771]	valid_0's binary_logloss: 0.398838
[772]	valid_0's binary_logloss: 0.39883
[773]	valid_0's binary_logloss: 0.398831
[774]	valid_0's binary_logloss: 0.398832
[775]	valid_0's binary_logloss: 0.398839
[776]	valid_0's binary_logloss: 0.39884
[777]	valid_0's binary_logloss: 0.398839
[778]	valid_0's binary_logloss: 0.398838
[779]	valid_0's binary_logloss: 0.398841
[780]	valid_0's binary_logloss: 0.398842
[781]	valid_0's binary_logloss: 0.398833
[782]	valid_0's binary_logloss: 0.398823
[783]	valid_0's binary_logloss: 0.398816
[784]	valid_0's binary_logloss: 0.398816
[785]	valid_0's binary_logloss: 0.398818
[786]	valid_0's binary_logloss: 0.39881
[787]	valid_0's binary_logloss: 0.398812
[788]	valid_0's binary_logloss: 0.398819
[789]	valid_0's binary_logloss: 0.398817
[790]	valid_0's binary_logloss: 0.398801
[791]	valid_0's binary_logloss: 0.398793
[792]	valid_0's binary_logloss: 0.398803
[793]	valid_0's binary_logloss: 0.398808
[794]	valid_0's binary_logloss: 0.398812
[795]	valid_0's binary_logloss: 0.398819
[796]	valid_0's binary_logloss: 0.398814
[797]	valid_0's binary_logloss: 0.398809
[798]	valid_0's binary_logloss: 0.398811
[799]	valid_0's binary_logloss: 0.398814
[800]	valid_0's binary_logloss: 0.398811
[801]	valid_0's binary_logloss: 0.398818
[802]	valid_0's binary_logloss: 0.398833
[803]	valid_0's binary_logloss: 0.398843
[804]	valid_0's binary_logloss: 0.398855
[805]	valid_0's binary_logloss: 0.398854
[806]	valid_0's binary_logloss: 0.398853
[807]	valid_0's binary_logloss: 0.398864
[808]	valid_0's binary_logloss: 0.398861
[809]	valid_0's binary_logloss: 0.398861
[810]	valid_0's binary_logloss: 0.398854
[811]	valid_0's binary_logloss: 0.398849
[812]	valid_0's binary_logloss: 0.398854
[813]	valid_0's binary_logloss: 0.398851
[814]	valid_0's binary_logloss: 0.398843
[815]	valid_0's binary_logloss: 0.398849
[816]	valid_0's binary_logloss: 0.398845
[817]	valid_0's binary_logloss: 0.398855
[818]	valid_0's binary_logloss: 0.398856
[819]	valid_0's binary_logloss: 0.39885
[820]	valid_0's binary_logloss: 0.398858
[821]	valid_0's binary_logloss: 0.398869
[822]	valid_0's binary_logloss: 0.398861
[823]	valid_0's binary_logloss: 0.398869
[824]	valid_0's binary_logloss: 0.398868
[825]	valid_0's binary_logloss: 0.398866
[826]	valid_0's binary_logloss: 0.398859
[827]	valid_0's binary_logloss: 0.398857
[828]	valid_0's binary_logloss: 0.398858
[829]	valid_0's binary_logloss: 0.398858
[830]	valid_0's binary_logloss: 0.398864
[831]	valid_0's binary_logloss: 0.398863
[832]	valid_0's binary_logloss: 0.398861
[833]	valid_0's binary_logloss: 0.398873
[834]	valid_0's binary_logloss: 0.398871
[835]	valid_0's binary_logloss: 0.39885
[836]	valid_0's binary_logloss: 0.39885
[837]	valid_0's binary_logloss: 0.398836
[838]	valid_0's binary_logloss: 0.398835
[839]	valid_0's binary_logloss: 0.39883
[840]	valid_0's binary_logloss: 0.398823
[841]	valid_0's binary_logloss: 0.398827
[842]	valid_0's binary_logloss: 0.398823
[843]	valid_0's binary_logloss: 0.398818
[844]	valid_0's binary_logloss: 0.398816
[845]	valid_0's binary_logloss: 0.398823
[846]	valid_0's binary_logloss: 0.39882
[847]	valid_0's binary_logloss: 0.398821
[848]	valid_0's binary_logloss: 0.398823
[849]	valid_0's binary_logloss: 0.398829
[850]	valid_0's binary_logloss: 0.398824
[851]	valid_0's binary_logloss: 0.398819
[852]	valid_0's binary_logloss: 0.398811
[853]	valid_0's binary_logloss: 0.39881
[854]	valid_0's binary_logloss: 0.398809
[855]	valid_0's binary_logloss: 0.398808
[856]	valid_0's binary_logloss: 0.398813
[857]	valid_0's binary_logloss: 0.398806
[858]	valid_0's binary_logloss: 0.398805
[859]	valid_0's binary_logloss: 0.398806
[860]	valid_0's binary_logloss: 0.398819
[861]	valid_0's binary_logloss: 0.398815
[862]	valid_0's binary_logloss: 0.398826
[863]	valid_0's binary_logloss: 0.398833
[864]	valid_0's binary_logloss: 0.398832
[865]	valid_0's binary_logloss: 0.39883
[866]	valid_0's binary_logloss: 0.398831
[867]	valid_0's binary_logloss: 0.398828
[868]	valid_0's binary_logloss: 0.398837
[869]	valid_0's binary_logloss: 0.398827
[870]	valid_0's binary_logloss: 0.398831
[871]	valid_0's binary_logloss: 0.39884
[872]	valid_0's binary_logloss: 0.398849
[873]	valid_0's binary_logloss: 0.398847
[874]	valid_0's binary_logloss: 0.398841
[875]	valid_0's binary_logloss: 0.398855
[876]	valid_0's binary_logloss: 0.398848
[877]	valid_0's binary_logloss: 0.398851
[878]	valid_0's binary_logloss: 0.398848
[879]	valid_0's binary_logloss: 0.398848
[880]	valid_0's binary_logloss: 0.398836
[881]	valid_0's binary_logloss: 0.398833
[882]	valid_0's binary_logloss: 0.398835
[883]	valid_0's binary_logloss: 0.398834
[884]	valid_0's binary_logloss: 0.398836
[885]	valid_0's binary_logloss: 0.398825
[886]	valid_0's binary_logloss: 0.398831
[887]	valid_0's binary_logloss: 0.398837
[888]	valid_0's binary_logloss: 0.398842
[889]	valid_0's binary_logloss: 0.398858
[890]	valid_0's binary_logloss: 0.398856
[891]	valid_0's binary_logloss: 0.398853
[892]	valid_0's binary_logloss: 0.39885
[893]	valid_0's binary_logloss: 0.398854
[894]	valid_0's binary_logloss: 0.398849
[895]	valid_0's binary_logloss: 0.398855
[896]	valid_0's binary_logloss: 0.398862
[897]	valid_0's binary_logloss: 0.398865
[898]	valid_0's binary_logloss: 0.398874
[899]	valid_0's binary_logloss: 0.398861
[900]	valid_0's binary_logloss: 0.398843
[901]	valid_0's binary_logloss: 0.398851
[902]	valid_0's binary_logloss: 0.398855
[903]	valid_0's binary_logloss: 0.398855
[904]	valid_0's binary_logloss: 0.398859
[905]	valid_0's binary_logloss: 0.398867
[906]	valid_0's binary_logloss: 0.398873
[907]	valid_0's binary_logloss: 0.398876
[908]	valid_0's binary_logloss: 0.398878
[909]	valid_0's binary_logloss: 0.398878
[910]	valid_0's binary_logloss: 0.398889
[911]	valid_0's binary_logloss: 0.398892
[912]	valid_0's binary_logloss: 0.398892
[913]	valid_0's binary_logloss: 0.398895
[914]	valid_0's binary_logloss: 0.3989
[915]	valid_0's binary_logloss: 0.398911
[916]	valid_0's binary_logloss: 0.398915
[917]	valid_0's binary_logloss: 0.398913
[918]	valid_0's binary_logloss: 0.398922
[919]	valid_0's binary_logloss: 0.398926
[920]	valid_0's binary_logloss: 0.398927
[921]	valid_0's binary_logloss: 0.398934
[922]	valid_0's binary_logloss: 0.398938
[923]	valid_0's binary_logloss: 0.398942
[924]	valid_0's binary_logloss: 0.398947
[925]	valid_0's binary_logloss: 0.39895
[926]	valid_0's binary_logloss: 0.398948
[927]	valid_0's binary_logloss: 0.398948
[928]	valid_0's binary_logloss: 0.398956
[929]	valid_0's binary_logloss: 0.398965
[930]	valid_0's binary_logloss: 0.398966
[931]	valid_0's binary_logloss: 0.398955
[932]	valid_0's binary_logloss: 0.39895
[933]	valid_0's binary_logloss: 0.398955
[934]	valid_0's binary_logloss: 0.398952
[935]	valid_0's binary_logloss: 0.398949
[936]	valid_0's binary_logloss: 0.398943
[937]	valid_0's binary_logloss: 0.39894
[938]	valid_0's binary_logloss: 0.398936
[939]	valid_0's binary_logloss: 0.398926
[940]	valid_0's binary_logloss: 0.398926
[941]	valid_0's binary_logloss: 0.398921
[942]	valid_0's binary_logloss: 0.398922
[943]	valid_0's binary_logloss: 0.398916
[944]	valid_0's binary_logloss: 0.39891
[945]	valid_0's binary_logloss: 0.398912
[946]	valid_0's binary_logloss: 0.398912
[947]	valid_0's binary_logloss: 0.398905
[948]	valid_0's binary_logloss: 0.398898
[949]	valid_0's binary_logloss: 0.398888
[950]	valid_0's binary_logloss: 0.398882
[951]	valid_0's binary_logloss: 0.398875
[952]	valid_0's binary_logloss: 0.398874
[953]	valid_0's binary_logloss: 0.39888
[954]	valid_0's binary_logloss: 0.398876
[955]	valid_0's binary_logloss: 0.398879
[956]	valid_0's binary_logloss: 0.39888
[957]	valid_0's binary_logloss: 0.39888
[958]	valid_0's binary_logloss: 0.398877
[959]	valid_0's binary_logloss: 0.398876
[960]	valid_0's binary_logloss: 0.398883
[961]	valid_0's binary_logloss: 0.398881
[962]	valid_0's binary_logloss: 0.398883
[963]	valid_0's binary_logloss: 0.39889
[964]	valid_0's binary_logloss: 0.398894
[965]	valid_0's binary_logloss: 0.398906
[966]	valid_0's binary_logloss: 0.39892
[967]	valid_0's binary_logloss: 0.398911
[968]	valid_0's binary_logloss: 0.39891
[969]	valid_0's binary_logloss: 0.398916
[970]	valid_0's binary_logloss: 0.398914
[971]	valid_0's binary_logloss: 0.398912
[972]	valid_0's binary_logloss: 0.398909
[973]	valid_0's binary_logloss: 0.398905
[974]	valid_0's binary_logloss: 0.398902
[975]	valid_0's binary_logloss: 0.398904
[976]	valid_0's binary_logloss: 0.39891
[977]	valid_0's binary_logloss: 0.398913
[978]	valid_0's binary_logloss: 0.398911
[979]	valid_0's binary_logloss: 0.398919
[980]	valid_0's binary_logloss: 0.398903
[981]	valid_0's binary_logloss: 0.398897
[982]	valid_0's binary_logloss: 0.398891
[983]	valid_0's binary_logloss: 0.398903
[984]	valid_0's binary_logloss: 0.398895
[985]	valid_0's binary_logloss: 0.398896
[986]	valid_0's binary_logloss: 0.39889
[987]	valid_0's binary_logloss: 0.398894
[988]	valid_0's binary_logloss: 0.398892
[989]	valid_0's binary_logloss: 0.398898
[990]	valid_0's binary_logloss: 0.398905
[991]	valid_0's binary_logloss: 0.398896
[992]	valid_0's binary_logloss: 0.398898
[993]	valid_0's binary_logloss: 0.398897
[994]	valid_0's binary_logloss: 0.398889
[995]	valid_0's binary_logloss: 0.398892
[996]	valid_0's binary_logloss: 0.398897
[997]	valid_0's binary_logloss: 0.398888
[998]	valid_0's binary_logloss: 0.398887
[999]	valid_0's binary_logloss: 0.398887
[1000]	valid_0's binary_logloss: 0.398876
[1001]	valid_0's binary_logloss: 0.39888
[1002]	valid_0's binary_logloss: 0.398885
[1003]	valid_0's binary_logloss: 0.39889
[1004]	valid_0's binary_logloss: 0.398886
[1005]	valid_0's binary_logloss: 0.398891
[1006]	valid_0's binary_logloss: 0.398895
[1007]	valid_0's binary_logloss: 0.398891
[1008]	valid_0's binary_logloss: 0.398884
[1009]	valid_0's binary_logloss: 0.398893
[1010]	valid_0's binary_logloss: 0.398889
[1011]	valid_0's binary_logloss: 0.398882
[1012]	valid_0's binary_logloss: 0.398873
[1013]	valid_0's binary_logloss: 0.398867
[1014]	valid_0's binary_logloss: 0.398868
[1015]	valid_0's binary_logloss: 0.398859
[1016]	valid_0's binary_logloss: 0.398868
[1017]	valid_0's binary_logloss: 0.398863
[1018]	valid_0's binary_logloss: 0.398869
[1019]	valid_0's binary_logloss: 0.398862
[1020]	valid_0's binary_logloss: 0.398872
[1021]	valid_0's binary_logloss: 0.398878
[1022]	valid_0's binary_logloss: 0.398882
[1023]	valid_0's binary_logloss: 0.398885
[1024]	valid_0's binary_logloss: 0.398889
[1025]	valid_0's binary_logloss: 0.398895
[1026]	valid_0's binary_logloss: 0.398897
[1027]	valid_0's binary_logloss: 0.398899
[1028]	valid_0's binary_logloss: 0.398904
[1029]	valid_0's binary_logloss: 0.398921
[1030]	valid_0's binary_logloss: 0.398921
[1031]	valid_0's binary_logloss: 0.398926
[1032]	valid_0's binary_logloss: 0.398921
[1033]	valid_0's binary_logloss: 0.398913
[1034]	valid_0's binary_logloss: 0.398926
[1035]	valid_0's binary_logloss: 0.398936
[1036]	valid_0's binary_logloss: 0.398928
[1037]	valid_0's binary_logloss: 0.398934
[1038]	valid_0's binary_logloss: 0.398933
[1039]	valid_0's binary_logloss: 0.398937
[1040]	valid_0's binary_logloss: 0.398933
[1041]	valid_0's binary_logloss: 0.398943
[1042]	valid_0's binary_logloss: 0.398949
[1043]	valid_0's binary_logloss: 0.398945
[1044]	valid_0's binary_logloss: 0.398942
[1045]	valid_0's binary_logloss: 0.39895
[1046]	valid_0's binary_logloss: 0.398956
[1047]	valid_0's binary_logloss: 0.398948
[1048]	valid_0's binary_logloss: 0.398939
[1049]	valid_0's binary_logloss: 0.398952
[1050]	valid_0's binary_logloss: 0.398957
[1051]	valid_0's binary_logloss: 0.398966
[1052]	valid_0's binary_logloss: 0.398967
[1053]	valid_0's binary_logloss: 0.398964
[1054]	valid_0's binary_logloss: 0.398971
[1055]	valid_0's binary_logloss: 0.398971
[1056]	valid_0's binary_logloss: 0.398967
[1057]	valid_0's binary_logloss: 0.398964
[1058]	valid_0's binary_logloss: 0.398967
[1059]	valid_0's binary_logloss: 0.398974
[1060]	valid_0's binary_logloss: 0.398971
[1061]	valid_0's binary_logloss: 0.398972
[1062]	valid_0's binary_logloss: 0.398969
[1063]	valid_0's binary_logloss: 0.398963
[1064]	valid_0's binary_logloss: 0.398968
[1065]	valid_0's binary_logloss: 0.398964
[1066]	valid_0's binary_logloss: 0.398964
[1067]	valid_0's binary_logloss: 0.398956
[1068]	valid_0's binary_logloss: 0.398954
[1069]	valid_0's binary_logloss: 0.39895
[1070]	valid_0's binary_logloss: 0.39895
[1071]	valid_0's binary_logloss: 0.398951
[1072]	valid_0's binary_logloss: 0.398957
[1073]	valid_0's binary_logloss: 0.398946
[1074]	valid_0's binary_logloss: 0.398948
[1075]	valid_0's binary_logloss: 0.398955
[1076]	valid_0's binary_logloss: 0.398957
[1077]	valid_0's binary_logloss: 0.398958
[1078]	valid_0's binary_logloss: 0.398959
[1079]	valid_0's binary_logloss: 0.398951
[1080]	valid_0's binary_logloss: 0.398953
[1081]	valid_0's binary_logloss: 0.398949
[1082]	valid_0's binary_logloss: 0.39895
[1083]	valid_0's binary_logloss: 0.398947
[1084]	valid_0's binary_logloss: 0.398948
[1085]	valid_0's binary_logloss: 0.398949
[1086]	valid_0's binary_logloss: 0.398949
[1087]	valid_0's binary_logloss: 0.398948
[1088]	valid_0's binary_logloss: 0.398952
[1089]	valid_0's binary_logloss: 0.398956
[1090]	valid_0's binary_logloss: 0.398948
[1091]	valid_0's binary_logloss: 0.39895
[1092]	valid_0's binary_logloss: 0.398944
[1093]	valid_0's binary_logloss: 0.398944
[1094]	valid_0's binary_logloss: 0.398947
[1095]	valid_0's binary_logloss: 0.398948
[1096]	valid_0's binary_logloss: 0.398943
[1097]	valid_0's binary_logloss: 0.398929
[1098]	valid_0's binary_logloss: 0.398931
[1099]	valid_0's binary_logloss: 0.398931
[1100]	valid_0's binary_logloss: 0.39893
[1101]	valid_0's binary_logloss: 0.398933
[1102]	valid_0's binary_logloss: 0.398939
[1103]	valid_0's binary_logloss: 0.398942
[1104]	valid_0's binary_logloss: 0.398934
[1105]	valid_0's binary_logloss: 0.398949
[1106]	valid_0's binary_logloss: 0.398951
[1107]	valid_0's binary_logloss: 0.39895
[1108]	valid_0's binary_logloss: 0.398956
[1109]	valid_0's binary_logloss: 0.39895
[1110]	valid_0's binary_logloss: 0.398954
[1111]	valid_0's binary_logloss: 0.398945
[1112]	valid_0's binary_logloss: 0.398939
[1113]	valid_0's binary_logloss: 0.398942
[1114]	valid_0's binary_logloss: 0.398947
[1115]	valid_0's binary_logloss: 0.398948
[1116]	valid_0's binary_logloss: 0.398951
[1117]	valid_0's binary_logloss: 0.398952
[1118]	valid_0's binary_logloss: 0.398954
[1119]	valid_0's binary_logloss: 0.398958
[1120]	valid_0's binary_logloss: 0.398952
[1121]	valid_0's binary_logloss: 0.398947
[1122]	valid_0's binary_logloss: 0.39894
[1123]	valid_0's binary_logloss: 0.398943
[1124]	valid_0's binary_logloss: 0.398939
[1125]	valid_0's binary_logloss: 0.398948
[1126]	valid_0's binary_logloss: 0.398935
[1127]	valid_0's binary_logloss: 0.398925
[1128]	valid_0's binary_logloss: 0.398936
[1129]	valid_0's binary_logloss: 0.398931
[1130]	valid_0's binary_logloss: 0.398938
[1131]	valid_0's binary_logloss: 0.398941
[1132]	valid_0's binary_logloss: 0.398946
[1133]	valid_0's binary_logloss: 0.398951
[1134]	valid_0's binary_logloss: 0.398962
[1135]	valid_0's binary_logloss: 0.398969
[1136]	valid_0's binary_logloss: 0.398973
[1137]	valid_0's binary_logloss: 0.398984
[1138]	valid_0's binary_logloss: 0.398987
[1139]	valid_0's binary_logloss: 0.398999
[1140]	valid_0's binary_logloss: 0.399004
[1141]	valid_0's binary_logloss: 0.399004
[1142]	valid_0's binary_logloss: 0.399005
[1143]	valid_0's binary_logloss: 0.399012
[1144]	valid_0's binary_logloss: 0.399013
[1145]	valid_0's binary_logloss: 0.39901
[1146]	valid_0's binary_logloss: 0.39901
[1147]	valid_0's binary_logloss: 0.398999
[1148]	valid_0's binary_logloss: 0.398991
[1149]	valid_0's binary_logloss: 0.398988
[1150]	valid_0's binary_logloss: 0.398995
[1151]	valid_0's binary_logloss: 0.398992
[1152]	valid_0's binary_logloss: 0.398994
[1153]	valid_0's binary_logloss: 0.399003
[1154]	valid_0's binary_logloss: 0.399005
[1155]	valid_0's binary_logloss: 0.399014
[1156]	valid_0's binary_logloss: 0.399015
[1157]	valid_0's binary_logloss: 0.399013
[1158]	valid_0's binary_logloss: 0.399015
[1159]	valid_0's binary_logloss: 0.399008
[1160]	valid_0's binary_logloss: 0.399005
[1161]	valid_0's binary_logloss: 0.399007
[1162]	valid_0's binary_logloss: 0.399009
[1163]	valid_0's binary_logloss: 0.399003
[1164]	valid_0's binary_logloss: 0.399009
[1165]	valid_0's binary_logloss: 0.399003
[1166]	valid_0's binary_logloss: 0.399003
[1167]	valid_0's binary_logloss: 0.399016
[1168]	valid_0's binary_logloss: 0.399014
[1169]	valid_0's binary_logloss: 0.399006
[1170]	valid_0's binary_logloss: 0.399009
[1171]	valid_0's binary_logloss: 0.399002
[1172]	valid_0's binary_logloss: 0.398994
[1173]	valid_0's binary_logloss: 0.399002
[1174]	valid_0's binary_logloss: 0.398999
[1175]	valid_0's binary_logloss: 0.398999
[1176]	valid_0's binary_logloss: 0.399001
[1177]	valid_0's binary_logloss: 0.399003
[1178]	valid_0's binary_logloss: 0.399007
[1179]	valid_0's binary_logloss: 0.399001
[1180]	valid_0's binary_logloss: 0.399004
[1181]	valid_0's binary_logloss: 0.398998
[1182]	valid_0's binary_logloss: 0.399
[1183]	valid_0's binary_logloss: 0.399
[1184]	valid_0's binary_logloss: 0.398997
[1185]	valid_0's binary_logloss: 0.398989
[1186]	valid_0's binary_logloss: 0.398991
[1187]	valid_0's binary_logloss: 0.398987
[1188]	valid_0's binary_logloss: 0.399001
[1189]	valid_0's binary_logloss: 0.399001
[1190]	valid_0's binary_logloss: 0.398998
[1191]	valid_0's binary_logloss: 0.398999
[1192]	valid_0's binary_logloss: 0.399006
[1193]	valid_0's binary_logloss: 0.399013
[1194]	valid_0's binary_logloss: 0.399024
[1195]	valid_0's binary_logloss: 0.399026
[1196]	valid_0's binary_logloss: 0.399021
[1197]	valid_0's binary_logloss: 0.39902
[1198]	valid_0's binary_logloss: 0.399027
[1199]	valid_0's binary_logloss: 0.399031
[1200]	valid_0's binary_logloss: 0.399034
[1201]	valid_0's binary_logloss: 0.399027
[1202]	valid_0's binary_logloss: 0.399025
[1203]	valid_0's binary_logloss: 0.399031
[1204]	valid_0's binary_logloss: 0.399037
[1205]	valid_0's binary_logloss: 0.399026
[1206]	valid_0's binary_logloss: 0.39904
[1207]	valid_0's binary_logloss: 0.399058
[1208]	valid_0's binary_logloss: 0.399062
[1209]	valid_0's binary_logloss: 0.399066
[1210]	valid_0's binary_logloss: 0.399072
[1211]	valid_0's binary_logloss: 0.399065
[1212]	valid_0's binary_logloss: 0.399069
[1213]	valid_0's binary_logloss: 0.399074
[1214]	valid_0's binary_logloss: 0.399075
[1215]	valid_0's binary_logloss: 0.399077
[1216]	valid_0's binary_logloss: 0.399077
[1217]	valid_0's binary_logloss: 0.399069
[1218]	valid_0's binary_logloss: 0.399067
[1219]	valid_0's binary_logloss: 0.399058
[1220]	valid_0's binary_logloss: 0.399059
[1221]	valid_0's binary_logloss: 0.399064
[1222]	valid_0's binary_logloss: 0.399063
[1223]	valid_0's binary_logloss: 0.399061
[1224]	valid_0's binary_logloss: 0.399066
[1225]	valid_0's binary_logloss: 0.399054
[1226]	valid_0's binary_logloss: 0.399052
[1227]	valid_0's binary_logloss: 0.399051
[1228]	valid_0's binary_logloss: 0.399054
[1229]	valid_0's binary_logloss: 0.399064
[1230]	valid_0's binary_logloss: 0.39907
[1231]	valid_0's binary_logloss: 0.39908
[1232]	valid_0's binary_logloss: 0.399084
[1233]	valid_0's binary_logloss: 0.399088
[1234]	valid_0's binary_logloss: 0.399089
[1235]	valid_0's binary_logloss: 0.399097
[1236]	valid_0's binary_logloss: 0.399099
[1237]	valid_0's binary_logloss: 0.399091
[1238]	valid_0's binary_logloss: 0.399088
[1239]	valid_0's binary_logloss: 0.399089
[1240]	valid_0's binary_logloss: 0.399088
[1241]	valid_0's binary_logloss: 0.399079
[1242]	valid_0's binary_logloss: 0.399083
[1243]	valid_0's binary_logloss: 0.399079
[1244]	valid_0's binary_logloss: 0.399089
[1245]	valid_0's binary_logloss: 0.399085
[1246]	valid_0's binary_logloss: 0.399083
[1247]	valid_0's binary_logloss: 0.399076
[1248]	valid_0's binary_logloss: 0.399075
[1249]	valid_0's binary_logloss: 0.399081
[1250]	valid_0's binary_logloss: 0.399078
[1251]	valid_0's binary_logloss: 0.399076
[1252]	valid_0's binary_logloss: 0.399074
[1253]	valid_0's binary_logloss: 0.39907
[1254]	valid_0's binary_logloss: 0.39907
[1255]	valid_0's binary_logloss: 0.399074
[1256]	valid_0's binary_logloss: 0.399072
[1257]	valid_0's binary_logloss: 0.399083
[1258]	valid_0's binary_logloss: 0.399094
[1259]	valid_0's binary_logloss: 0.399092
[1260]	valid_0's binary_logloss: 0.399084
[1261]	valid_0's binary_logloss: 0.399085
[1262]	valid_0's binary_logloss: 0.399091
[1263]	valid_0's binary_logloss: 0.399093
[1264]	valid_0's binary_logloss: 0.399086
[1265]	valid_0's binary_logloss: 0.399091
[1266]	valid_0's binary_logloss: 0.399082
[1267]	valid_0's binary_logloss: 0.399081
[1268]	valid_0's binary_logloss: 0.399087
[1269]	valid_0's binary_logloss: 0.399079
[1270]	valid_0's binary_logloss: 0.399074
[1271]	valid_0's binary_logloss: 0.399072
[1272]	valid_0's binary_logloss: 0.39907
[1273]	valid_0's binary_logloss: 0.399071
[1274]	valid_0's binary_logloss: 0.399072
[1275]	valid_0's binary_logloss: 0.399068
[1276]	valid_0's binary_logloss: 0.399069
[1277]	valid_0's binary_logloss: 0.399076
[1278]	valid_0's binary_logloss: 0.399085
[1279]	valid_0's binary_logloss: 0.39909
[1280]	valid_0's binary_logloss: 0.399088
[1281]	valid_0's binary_logloss: 0.399093
[1282]	valid_0's binary_logloss: 0.399091
[1283]	valid_0's binary_logloss: 0.399092
[1284]	valid_0's binary_logloss: 0.399094
[1285]	valid_0's binary_logloss: 0.399099
[1286]	valid_0's binary_logloss: 0.399097
[1287]	valid_0's binary_logloss: 0.399104
[1288]	valid_0's binary_logloss: 0.399102
[1289]	valid_0's binary_logloss: 0.399098
[1290]	valid_0's binary_logloss: 0.399105
[1291]	valid_0's binary_logloss: 0.399111
Early stopping, best iteration is:
[791]	valid_0's binary_logloss: 0.398793

In [40]:

print(gbm.best_score)
print(gbm.best_iteration)

defaultdict(, {'valid_0': {'binary_logloss': 0.39879332755938357}})
791

XGBoost 模型

XGBoost是boosting算法的其中一种。Boosting算法的思想是将许多弱分类器集成在一起形成一个强分类器。因为XGBoost是一种提升树模型，所以它是将许多树模型集成在一起，形成一个很强的分类器。

In [41]:

def run_default_test(train, test, features, target, random_state=0):
    eta = 0.1
    max_depth = 5
    subsample = 0.8
    colsample_bytree = 0.8
    params = {
        "objective": "binary:logistic",
        "booster" : "gbtree",
        "eval_metric": "logloss",
        "eta": eta,
        "max_depth": max_depth,
        "subsample": subsample,
        "colsample_bytree": colsample_bytree,
        "silent": 1,
        "seed": random_state
    }
    num_boost_round = 260
    early_stopping_rounds = 20
    test_size = 0.2

    X_train, X_valid = train_test_split(train, test_size=test_size, random_state=random_state)
    y_train = X_train[target]
    y_valid = X_valid[target]
    dtrain = xgb.DMatrix(X_train[features], y_train)
    dvalid = xgb.DMatrix(X_valid[features], y_valid)
    watchlist = [(dtrain, 'train'), (dvalid, 'eval')]
    gbm = xgb.train(params, dtrain, num_boost_round, evals=watchlist, early_stopping_rounds=early_stopping_rounds, verbose_eval=True)

In [42]:

features = ['C1', 'banner_pos', 'device_type', 'device_conn_type', 'C14',
       'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'time',
       'site_id_int', 'site_domain_int', 'site_category_int', 'app_id_int',
       'app_domain_int', 'app_category_int', 'device_id_int', 'device_ip_int',
       'device_model_int', 'day_of_week_int']
run_default_test(train, y_target, features, 'click')

d:\ProgramData\Anaconda3\lib\site-packages\xgboost\core.py:587: FutureWarning: Series.base is deprecated and will be removed in a future version
  if getattr(data, 'base', None) is not None and \
d:\ProgramData\Anaconda3\lib\site-packages\xgboost\core.py:588: FutureWarning: Series.base is deprecated and will be removed in a future version
  data.base is not None and isinstance(data, np.ndarray) \

[0]	train-logloss:0.648202	eval-logloss:0.648482
Multiple eval metrics have been passed: 'eval-logloss' will be used for early stopping.

Will train until eval-logloss hasn't improved in 20 rounds.
[1]	train-logloss:0.611812	eval-logloss:0.612024
[2]	train-logloss:0.581816	eval-logloss:0.582063
[3]	train-logloss:0.556549	eval-logloss:0.557064
[4]	train-logloss:0.535814	eval-logloss:0.536118
[5]	train-logloss:0.517958	eval-logloss:0.518558
[6]	train-logloss:0.502994	eval-logloss:0.50364
[7]	train-logloss:0.490596	eval-logloss:0.491261
[8]	train-logloss:0.479941	eval-logloss:0.480669
[9]	train-logloss:0.470785	eval-logloss:0.471492
[10]	train-logloss:0.463122	eval-logloss:0.463869
[11]	train-logloss:0.45646	eval-logloss:0.457213
[12]	train-logloss:0.450839	eval-logloss:0.451555
[13]	train-logloss:0.446007	eval-logloss:0.446811
[14]	train-logloss:0.441838	eval-logloss:0.442608
[15]	train-logloss:0.438173	eval-logloss:0.43901
[16]	train-logloss:0.435234	eval-logloss:0.436098
[17]	train-logloss:0.432811	eval-logloss:0.433679
[18]	train-logloss:0.430664	eval-logloss:0.431539
[19]	train-logloss:0.428844	eval-logloss:0.429709
[20]	train-logloss:0.427242	eval-logloss:0.428106
[21]	train-logloss:0.425909	eval-logloss:0.426819
[22]	train-logloss:0.424743	eval-logloss:0.425673
[23]	train-logloss:0.423717	eval-logloss:0.424661
[24]	train-logloss:0.422768	eval-logloss:0.423749
[25]	train-logloss:0.421947	eval-logloss:0.422912
[26]	train-logloss:0.421313	eval-logloss:0.422268
[27]	train-logloss:0.420647	eval-logloss:0.421645
[28]	train-logloss:0.41985	eval-logloss:0.420835
[29]	train-logloss:0.419281	eval-logloss:0.420271
[30]	train-logloss:0.418854	eval-logloss:0.41988
[31]	train-logloss:0.418329	eval-logloss:0.419392
[32]	train-logloss:0.417988	eval-logloss:0.419038
[33]	train-logloss:0.417455	eval-logloss:0.418506
[34]	train-logloss:0.417182	eval-logloss:0.418232
[35]	train-logloss:0.416796	eval-logloss:0.417869
[36]	train-logloss:0.416541	eval-logloss:0.417632
[37]	train-logloss:0.416249	eval-logloss:0.417374
[38]	train-logloss:0.415979	eval-logloss:0.417077
[39]	train-logloss:0.415615	eval-logloss:0.416748
[40]	train-logloss:0.415409	eval-logloss:0.416551
[41]	train-logloss:0.415238	eval-logloss:0.416376
[42]	train-logloss:0.414928	eval-logloss:0.416093
[43]	train-logloss:0.414788	eval-logloss:0.415934
[44]	train-logloss:0.414658	eval-logloss:0.415817
[45]	train-logloss:0.414423	eval-logloss:0.415599
[46]	train-logloss:0.414296	eval-logloss:0.415471
[47]	train-logloss:0.413996	eval-logloss:0.415189
[48]	train-logloss:0.413684	eval-logloss:0.414891
[49]	train-logloss:0.413493	eval-logloss:0.414722
[50]	train-logloss:0.413347	eval-logloss:0.414585
[51]	train-logloss:0.413164	eval-logloss:0.414415
[52]	train-logloss:0.412888	eval-logloss:0.414135
[53]	train-logloss:0.412761	eval-logloss:0.41403
[54]	train-logloss:0.412538	eval-logloss:0.413815
[55]	train-logloss:0.412267	eval-logloss:0.413574
[56]	train-logloss:0.412141	eval-logloss:0.41345
[57]	train-logloss:0.411992	eval-logloss:0.413313
[58]	train-logloss:0.411802	eval-logloss:0.413127
[59]	train-logloss:0.411595	eval-logloss:0.412914
[60]	train-logloss:0.411502	eval-logloss:0.412833
[61]	train-logloss:0.41133	eval-logloss:0.412657
[62]	train-logloss:0.411182	eval-logloss:0.412536
[63]	train-logloss:0.410978	eval-logloss:0.412339
[64]	train-logloss:0.410857	eval-logloss:0.412238
[65]	train-logloss:0.410672	eval-logloss:0.41206
[66]	train-logloss:0.410547	eval-logloss:0.411965
[67]	train-logloss:0.410384	eval-logloss:0.411808
[68]	train-logloss:0.410266	eval-logloss:0.411698
[69]	train-logloss:0.409953	eval-logloss:0.411408
[70]	train-logloss:0.409885	eval-logloss:0.411342
[71]	train-logloss:0.409768	eval-logloss:0.411262
[72]	train-logloss:0.409633	eval-logloss:0.411144
[73]	train-logloss:0.409546	eval-logloss:0.411063
[74]	train-logloss:0.409468	eval-logloss:0.411003
[75]	train-logloss:0.409314	eval-logloss:0.410868
[76]	train-logloss:0.409136	eval-logloss:0.410702
[77]	train-logloss:0.409036	eval-logloss:0.410592
[78]	train-logloss:0.408706	eval-logloss:0.410316
[79]	train-logloss:0.408608	eval-logloss:0.410226
[80]	train-logloss:0.408517	eval-logloss:0.41015
[81]	train-logloss:0.408446	eval-logloss:0.41008
[82]	train-logloss:0.408164	eval-logloss:0.409835
[83]	train-logloss:0.408062	eval-logloss:0.409746
[84]	train-logloss:0.407919	eval-logloss:0.409628
[85]	train-logloss:0.407757	eval-logloss:0.409457
[86]	train-logloss:0.407614	eval-logloss:0.409318
[87]	train-logloss:0.407421	eval-logloss:0.40916
[88]	train-logloss:0.40737	eval-logloss:0.409113
[89]	train-logloss:0.407301	eval-logloss:0.409064
[90]	train-logloss:0.407224	eval-logloss:0.408999
[91]	train-logloss:0.407149	eval-logloss:0.408935
[92]	train-logloss:0.406966	eval-logloss:0.408762
[93]	train-logloss:0.406783	eval-logloss:0.408604
[94]	train-logloss:0.406727	eval-logloss:0.408548
[95]	train-logloss:0.40661	eval-logloss:0.408455
[96]	train-logloss:0.406514	eval-logloss:0.408377
[97]	train-logloss:0.406396	eval-logloss:0.408263
[98]	train-logloss:0.406261	eval-logloss:0.408159
[99]	train-logloss:0.40613	eval-logloss:0.408039
[100]	train-logloss:0.406038	eval-logloss:0.407954
[101]	train-logloss:0.405962	eval-logloss:0.407872
[102]	train-logloss:0.405841	eval-logloss:0.407774
[103]	train-logloss:0.405767	eval-logloss:0.407721
[104]	train-logloss:0.405717	eval-logloss:0.407674
[105]	train-logloss:0.405654	eval-logloss:0.407619
[106]	train-logloss:0.405615	eval-logloss:0.407582
[107]	train-logloss:0.405513	eval-logloss:0.407487
[108]	train-logloss:0.405457	eval-logloss:0.407445
[109]	train-logloss:0.405383	eval-logloss:0.407379
[110]	train-logloss:0.405321	eval-logloss:0.407327
[111]	train-logloss:0.40527	eval-logloss:0.407289
[112]	train-logloss:0.405204	eval-logloss:0.407231
[113]	train-logloss:0.4051	eval-logloss:0.407138
[114]	train-logloss:0.405065	eval-logloss:0.407108
[115]	train-logloss:0.405014	eval-logloss:0.40708
[116]	train-logloss:0.404949	eval-logloss:0.407036
[117]	train-logloss:0.404917	eval-logloss:0.407019
[118]	train-logloss:0.404852	eval-logloss:0.406968
[119]	train-logloss:0.404778	eval-logloss:0.406898
[120]	train-logloss:0.404709	eval-logloss:0.406843
[121]	train-logloss:0.404631	eval-logloss:0.40679
[122]	train-logloss:0.404534	eval-logloss:0.406706
[123]	train-logloss:0.404472	eval-logloss:0.406667
[124]	train-logloss:0.40441	eval-logloss:0.406604
[125]	train-logloss:0.404349	eval-logloss:0.406563
[126]	train-logloss:0.404262	eval-logloss:0.406484
[127]	train-logloss:0.404168	eval-logloss:0.406419
[128]	train-logloss:0.404136	eval-logloss:0.406397
[129]	train-logloss:0.404097	eval-logloss:0.406368
[130]	train-logloss:0.404016	eval-logloss:0.406302
[131]	train-logloss:0.403964	eval-logloss:0.406259
[132]	train-logloss:0.403933	eval-logloss:0.406248
[133]	train-logloss:0.403899	eval-logloss:0.406226
[134]	train-logloss:0.403845	eval-logloss:0.406191
[135]	train-logloss:0.403785	eval-logloss:0.406131
[136]	train-logloss:0.403698	eval-logloss:0.406073
[137]	train-logloss:0.40364	eval-logloss:0.40602
[138]	train-logloss:0.403559	eval-logloss:0.405959
[139]	train-logloss:0.403499	eval-logloss:0.405906
[140]	train-logloss:0.40341	eval-logloss:0.405839
[141]	train-logloss:0.403348	eval-logloss:0.405797
[142]	train-logloss:0.403319	eval-logloss:0.405781
[143]	train-logloss:0.403228	eval-logloss:0.405703
[144]	train-logloss:0.403186	eval-logloss:0.405684
[145]	train-logloss:0.403176	eval-logloss:0.405673
[146]	train-logloss:0.403143	eval-logloss:0.405651
[147]	train-logloss:0.403072	eval-logloss:0.405584
[148]	train-logloss:0.403037	eval-logloss:0.405568
[149]	train-logloss:0.402974	eval-logloss:0.405524
[150]	train-logloss:0.402884	eval-logloss:0.405443
[151]	train-logloss:0.402859	eval-logloss:0.405435
[152]	train-logloss:0.402766	eval-logloss:0.405351
[153]	train-logloss:0.402674	eval-logloss:0.405268
[154]	train-logloss:0.402614	eval-logloss:0.40522
[155]	train-logloss:0.402594	eval-logloss:0.405206
[156]	train-logloss:0.402566	eval-logloss:0.405193
[157]	train-logloss:0.402496	eval-logloss:0.405131
[158]	train-logloss:0.40247	eval-logloss:0.405121
[159]	train-logloss:0.402423	eval-logloss:0.405089
[160]	train-logloss:0.402399	eval-logloss:0.405077
[161]	train-logloss:0.402341	eval-logloss:0.405034
[162]	train-logloss:0.402307	eval-logloss:0.405011
[163]	train-logloss:0.402286	eval-logloss:0.405
[164]	train-logloss:0.402218	eval-logloss:0.404934
[165]	train-logloss:0.402188	eval-logloss:0.404908
[166]	train-logloss:0.402158	eval-logloss:0.404883
[167]	train-logloss:0.402128	eval-logloss:0.40486
[168]	train-logloss:0.402092	eval-logloss:0.40485
[169]	train-logloss:0.402039	eval-logloss:0.404823
[170]	train-logloss:0.401968	eval-logloss:0.404758
[171]	train-logloss:0.401911	eval-logloss:0.404704
[172]	train-logloss:0.401889	eval-logloss:0.404691
[173]	train-logloss:0.401846	eval-logloss:0.404652
[174]	train-logloss:0.401816	eval-logloss:0.404642
[175]	train-logloss:0.401726	eval-logloss:0.404589
[176]	train-logloss:0.401669	eval-logloss:0.404541
[177]	train-logloss:0.401629	eval-logloss:0.404521
[178]	train-logloss:0.401597	eval-logloss:0.404497
[179]	train-logloss:0.401561	eval-logloss:0.404473
[180]	train-logloss:0.40151	eval-logloss:0.404441
[181]	train-logloss:0.401445	eval-logloss:0.40438
[182]	train-logloss:0.401411	eval-logloss:0.404359
[183]	train-logloss:0.401394	eval-logloss:0.404348
[184]	train-logloss:0.401359	eval-logloss:0.404326
[185]	train-logloss:0.401315	eval-logloss:0.404285
[186]	train-logloss:0.401253	eval-logloss:0.404246
[187]	train-logloss:0.401216	eval-logloss:0.404226
[188]	train-logloss:0.401192	eval-logloss:0.40421
[189]	train-logloss:0.401159	eval-logloss:0.404186
[190]	train-logloss:0.401137	eval-logloss:0.404173
[191]	train-logloss:0.401077	eval-logloss:0.404115
[192]	train-logloss:0.40104	eval-logloss:0.404071
[193]	train-logloss:0.401018	eval-logloss:0.404056
[194]	train-logloss:0.400969	eval-logloss:0.404018
[195]	train-logloss:0.400946	eval-logloss:0.404011
[196]	train-logloss:0.400911	eval-logloss:0.404001
[197]	train-logloss:0.400883	eval-logloss:0.404
[198]	train-logloss:0.400866	eval-logloss:0.403999
[199]	train-logloss:0.400786	eval-logloss:0.403922
[200]	train-logloss:0.400774	eval-logloss:0.403923
[201]	train-logloss:0.40073	eval-logloss:0.403899
[202]	train-logloss:0.400705	eval-logloss:0.403885
[203]	train-logloss:0.400691	eval-logloss:0.40388
[204]	train-logloss:0.400662	eval-logloss:0.403872
[205]	train-logloss:0.400647	eval-logloss:0.403873
[206]	train-logloss:0.400601	eval-logloss:0.403841
[207]	train-logloss:0.400592	eval-logloss:0.403837
[208]	train-logloss:0.40057	eval-logloss:0.403832
[209]	train-logloss:0.400553	eval-logloss:0.403826
[210]	train-logloss:0.400513	eval-logloss:0.403798
[211]	train-logloss:0.400478	eval-logloss:0.403777
[212]	train-logloss:0.400455	eval-logloss:0.403779
[213]	train-logloss:0.400441	eval-logloss:0.403776
[214]	train-logloss:0.400386	eval-logloss:0.40373
[215]	train-logloss:0.400331	eval-logloss:0.403681
[216]	train-logloss:0.400291	eval-logloss:0.403653
[217]	train-logloss:0.400215	eval-logloss:0.403595
[218]	train-logloss:0.400184	eval-logloss:0.403579
[219]	train-logloss:0.400157	eval-logloss:0.403566
[220]	train-logloss:0.400145	eval-logloss:0.403566
[221]	train-logloss:0.400109	eval-logloss:0.403548
[222]	train-logloss:0.400079	eval-logloss:0.40353
[223]	train-logloss:0.400037	eval-logloss:0.403498
[224]	train-logloss:0.400003	eval-logloss:0.403484
[225]	train-logloss:0.399988	eval-logloss:0.403473
[226]	train-logloss:0.399955	eval-logloss:0.403446
[227]	train-logloss:0.399902	eval-logloss:0.403409
[228]	train-logloss:0.399886	eval-logloss:0.403403
[229]	train-logloss:0.39986	eval-logloss:0.403393
[230]	train-logloss:0.399848	eval-logloss:0.403391
[231]	train-logloss:0.399806	eval-logloss:0.403367
[232]	train-logloss:0.399772	eval-logloss:0.403355
[233]	train-logloss:0.399746	eval-logloss:0.40334
[234]	train-logloss:0.39971	eval-logloss:0.403309
[235]	train-logloss:0.399688	eval-logloss:0.403297
[236]	train-logloss:0.39964	eval-logloss:0.403262
[237]	train-logloss:0.399575	eval-logloss:0.40321
[238]	train-logloss:0.399535	eval-logloss:0.403181
[239]	train-logloss:0.399517	eval-logloss:0.403169
[240]	train-logloss:0.399477	eval-logloss:0.403151
[241]	train-logloss:0.399443	eval-logloss:0.403127
[242]	train-logloss:0.399422	eval-logloss:0.403124
[243]	train-logloss:0.399406	eval-logloss:0.403121
[244]	train-logloss:0.399389	eval-logloss:0.40312
[245]	train-logloss:0.399346	eval-logloss:0.403092
[246]	train-logloss:0.399303	eval-logloss:0.403066
[247]	train-logloss:0.399254	eval-logloss:0.403028
[248]	train-logloss:0.399227	eval-logloss:0.403013
[249]	train-logloss:0.399195	eval-logloss:0.402997
[250]	train-logloss:0.399152	eval-logloss:0.402965
[251]	train-logloss:0.399126	eval-logloss:0.402949
[252]	train-logloss:0.399116	eval-logloss:0.402944
[253]	train-logloss:0.399079	eval-logloss:0.402915
[254]	train-logloss:0.399065	eval-logloss:0.402905
[255]	train-logloss:0.399038	eval-logloss:0.402892
[256]	train-logloss:0.399019	eval-logloss:0.402887
[257]	train-logloss:0.399006	eval-logloss:0.402883
[258]	train-logloss:0.398993	eval-logloss:0.402881
[259]	train-logloss:0.398973	eval-logloss:0.402872

机器学习机器学习实例

人工智能机器学习：Avazu移动互联在线广告点击预测

广告的点击率 = 广告点击量/广告的展现量

数据

特征工程

对Hour的特征工程

按星期特征工程

匿名特征变量C1

banner_pos

device_type

app features

C14 - C21 features

建模

has简介

LightGBM 模型

XGBoost 模型

相关

[ML]机器学习中我未见过的概念

[机器学习笔记(一)] TensorFLow安装

机器学习 - 线性回归模型实战 02

机器学习-支持向量机SVM

TensorFlow——机器学习编程框架

机器学习（三、神经网络）

吴恩达机器学习作业2- 逻辑回归与正则化作业（python实现）

[ 机器学习 - 吴恩达 ] | 1-2 What is machine learning

《神经网络与机器学习》第8章泛化与正则化

【机器学习】逻辑回归（Logistic Regression）

机器学习--决策树算法(CART)

机器学习--决策树算法(CART)

标签