Kaggle项目——房价预测

第一部分：https://zhuanlan.zhihu.com/p/42406829

第二部分：https://zhuanlan.zhihu.com/p/42409652

github地址：wtchen77/Kaggle-House-Prices

1. 问题描述

项目地址：House Prices: Advanced Regression Techniques

• 基于项目提供的爱荷华州埃姆斯的房屋历史成交数据，预测新的房屋销售价格
• 这是一个回归问题
• 项目的评分标准是均方根误差(RMSE)，预测价格和实际价格取对数计算均方根误差

# 导入类库
import numpy as np
import pandas as pd
import scipy.stats as stats

import matplotlib.pyplot as plt
import seaborn as sns

import warnings
warnings.filterwarnings('ignore')
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import RobustScaler

from sklearn.decomposition import PCA
from sklearn.model_selection import cross_val_score, GridSearchCV, KFold

from sklearn.base import BaseEstimator, TransformerMixin, RegressorMixin
from sklearn.base import clone
from sklearn.linear_model import Lasso
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Ridge
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor, ExtraTreesRegressor
from sklearn.svm import SVR, LinearSVR
from sklearn.linear_model import ElasticNet, SGDRegressor, BayesianRidge
from sklearn.kernel_ridge import KernelRidge
from xgboost import XGBRegressor
# 显示中文
plt.rcParams['font.sans-serif'] = ['SimHei']
plt.rcParams['axes.unicode_minus'] = False

2. 数据理解

2.1 数据概览

# 导入数据
train_df = pd.read_csv('./data/train.csv')
test_df = pd.read_csv('./data/test.csv')
# 查看前几行数据
train_df.head()

print('训练集维度：%s，测试集维度：%s' % (train_df.shape, test_df.shape))
训练集维度：(1460, 81)，测试集维度：(1459, 80)
# 查看数据基本信息
train_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 1460 entries, 0 to 1459 Data columns (total 81 columns): Id 1460 non-null int64 MSSubClass 1460 non-null int64 MSZoning 1460 non-null object LotFrontage 1201 non-null float64 LotArea 1460 non-null int64 Street 1460 non-null object Alley 91 non-null object LotShape 1460 non-null object LandContour 1460 non-null object Utilities 1460 non-null object LotConfig 1460 non-null object LandSlope 1460 non-null object Neighborhood 1460 non-null object Condition1 1460 non-null object Condition2 1460 non-null object BldgType 1460 non-null object HouseStyle 1460 non-null object OverallQual 1460 non-null int64 OverallCond 1460 non-null int64 YearBuilt 1460 non-null int64 YearRemodAdd 1460 non-null int64 RoofStyle 1460 non-null object RoofMatl 1460 non-null object Exterior1st 1460 non-null object Exterior2nd 1460 non-null object MasVnrType 1452 non-null object MasVnrArea 1452 non-null float64 ExterQual 1460 non-null object ExterCond 1460 non-null object Foundation 1460 non-null object BsmtQual 1423 non-null object BsmtCond 1423 non-null object BsmtExposure 1422 non-null object BsmtFinType1 1423 non-null object BsmtFinSF1 1460 non-null int64 BsmtFinType2 1422 non-null object BsmtFinSF2 1460 non-null int64 BsmtUnfSF 1460 non-null int64 TotalBsmtSF 1460 non-null int64 Heating 1460 non-null object HeatingQC 1460 non-null object CentralAir 1460 non-null object Electrical 1459 non-null object 1stFlrSF 1460 non-null int64 2ndFlrSF 1460 non-null int64 LowQualFinSF 1460 non-null int64 GrLivArea 1460 non-null int64 BsmtFullBath 1460 non-null int64 BsmtHalfBath 1460 non-null int64 FullBath 1460 non-null int64 HalfBath 1460 non-null int64 BedroomAbvGr 1460 non-null int64 KitchenAbvGr 1460 non-null int64 KitchenQual 1460 non-null object TotRmsAbvGrd 1460 non-null int64 Functional 1460 non-null object Fireplaces 1460 non-null int64 FireplaceQu 770 non-null object GarageType 1379 non-null object GarageYrBlt 1379 non-null float64 GarageFinish 1379 non-null object GarageCars 1460 non-null int64 GarageArea 1460 non-null int64 GarageQual 1379 non-null object GarageCond 1379 non-null object PavedDrive 1460 non-null object WoodDeckSF 1460 non-null int64 OpenPorchSF 1460 non-null int64 EnclosedPorch 1460 non-null int64 3SsnPorch 1460 non-null int64 ScreenPorch 1460 non-null int64 PoolArea 1460 non-null int64 PoolQC 7 non-null object Fence 281 non-null object MiscFeature 54 non-null object MiscVal 1460 non-null int64 MoSold 1460 non-null int64 YrSold 1460 non-null int64 SaleType 1460 non-null object SaleCondition 1460 non-null object SalePrice 1460 non-null int64 dtypes: float64(3), int64(35), object(43) memory usage: 924.0+ KB  # 查看数据统计信息 train_df.describe() 

数据基本信息

• 训练集维度：(1460, 81)，测试集维度：(1459, 80)
• 特征变量79个(不包括’Id’)，目标变量为’SalePrice’
• 特征变量类型：float64(3), int64(33), object(43)

数据集变量解释

• – SalePrice: 房产销售价格，以美元计价。所要预测的目标变量
• – MSSubClass: Identifies the type of dwelling involved in the sale 住所类型
• – MSZoning: The general zoning classification 区域分类
• – LotFrontage: Linear feet of street connected to property 房子同街道之间的距离
• – LotArea: Lot size in square feet 建筑面积
• – Street: Type of road access 主路的路面类型
• – Alley: Type of alley access 小道的路面类型
• – LotShape: General shape of property 房屋外形
• – LandContour: Flatness of the property 平整度
• – Utilities: Type of utilities available 配套公用设施类型
• – LotConfig: Lot configuration 配置
• – LandSlope: Slope of property 土地坡度
• – Neighborhood: Physical locations within Ames city limits 房屋在埃姆斯市的位置
• – Condition1: Proximity to main road or railroad 附近交通情况
• – Condition2: Proximity to main road or railroad (if a second is present) 附近交通情况（如果同时满足两种情况）
• – BldgType: Type of dwelling 住宅类型
• – HouseStyle: Style of dwelling 房屋的层数
• – OverallQual: Overall material and finish quality 完工质量和材料
• – OverallCond: Overall condition rating 整体条件等级
• – YearBuilt: Original construction date 建造年份
• – YearRemodAdd: Remodel date 翻修年份
• – RoofStyle: Type of roof 屋顶类型
• – RoofMatl: Roof material 屋顶材料
• – Exterior1st: Exterior covering on house 外立面材料
• – Exterior2nd: Exterior covering on house (if more than one material) 外立面材料2
• – MasVnrType: Masonry veneer type 装饰石材类型
• – MasVnrArea: Masonry veneer area in square feet 装饰石材面积
• – ExterQual: Exterior material quality 外立面材料质量
• – ExterCond: Present condition of the material on the exterior 外立面材料外观情况
• – Foundation: Type of foundation 房屋结构类型
• – BsmtQual: Height of the basement 评估地下室层高情况
• – BsmtCond: General condition of the basement 地下室总体情况
• – BsmtExposure: Walkout or garden level basement walls 地下室出口或者花园层的墙面
• – BsmtFinType1: Quality of basement finished area 地下室区域质量
• – BsmtFinSF1: Type 1 finished square feet Type 1完工面积
• – BsmtFinType2: Quality of second finished area (if present) 二次完工面积质量（如果有）
• – BsmtFinSF2: Type 2 finished square feet Type 2完工面积
• – BsmtUnfSF: Unfinished square feet of basement area 地下室区域未完工面积
• – TotalBsmtSF: Total square feet of basement area 地下室总体面积
• – Heating: Type of heating 采暖类型
• – HeatingQC: Heating quality and condition 采暖质量和条件
• – CentralAir: Central air conditioning 中央空调系统
• – Electrical: Electrical system 电力系统
• – 1stFlrSF: First Floor square feet 第一层面积
• – 2ndFlrSF: Second floor square feet 第二层面积
• – LowQualFinSF: Low quality finished square feet (all floors) 低质量完工面积
• – GrLivArea: Above grade (ground) living area square feet 地面以上部分起居面积
• – BsmtFullBath: Basement full bathrooms 地下室全浴室数量
• – BsmtHalfBath: Basement half bathrooms 地下室半浴室数量
• – FullBath: Full bathrooms above grade 地面以上全浴室数量
• – HalfBath: Half baths above grade 地面以上半浴室数量
• – Bedroom: Number of bedrooms above basement level 地面以上卧室数量
• – KitchenAbvGr: Number of kitchens 厨房数量
• – KitchenQual: Kitchen quality 厨房质量
• – TotRmsAbvGrd: Total rooms above grade (does not include bathrooms) 总房间数（不含浴室和地下部分）
• – Functional: Home functionality rating 功能性评级
• – Fireplaces: Number of fireplaces 壁炉数量
• – FireplaceQu: Fireplace quality 壁炉质量
• – GarageType: Garage location 车库位置
• – GarageYrBlt: Year garage was built 车库建造时间
• – GarageFinish: Interior finish of the garage 车库内饰
• – GarageCars: Size of garage in car capacity 车壳大小以停车数量表示
• – GarageArea: Size of garage in square feet 车库面积
• – GarageQual: Garage quality 车库质量
• – GarageCond: Garage condition 车库条件
• – PavedDrive: Paved driveway 车道铺砌情况
• – WoodDeckSF: Wood deck area in square feet 实木地板面积
• – OpenPorchSF: Open porch area in square feet 开放式门廊面积
• – EnclosedPorch: Enclosed porch area in square feet 封闭式门廊面积
• – 3SsnPorch: Three season porch area in square feet 时令门廊面积
• – ScreenPorch: Screen porch area in square feet 屏风门廊面积
• – PoolArea: Pool area in square feet 游泳池面积
• – PoolQC: Pool quality 游泳池质量
• – Fence: Fence quality 围栏质量
• – MiscFeature: Miscellaneous feature not covered in other categories 其它条件中未包含部分的特性
• – MiscVal: $Value of miscellaneous feature 杂项部分价值
• – MoSold: Month Sold 卖出月份
• – YrSold: Year Sold 卖出年份
• – SaleType: Type of sale 出售类型
• – SaleCondition: Condition of sale 出售条件

2.2 数据探索分析

2.2.1 分析目标变量(SalePrice)

# SalePrice统计信息
train_df['SalePrice'].describe()
count      1460.000000
mean     180921.195890
std       79442.502883
min       34900.000000
25%      129975.000000
50%      163000.000000
75%      214000.000000
max      755000.000000
Name: SalePrice, dtype: float64
# SalePrice直方图
# seaborn作直方图默认画出密度曲线
plt.figure()
sns.distplot(train_df['SalePrice'])
plt.title('SalePrice分布')
plt.show()

通过图中可以看出房价不完全服从正态分布。我们可以查看峰度（Kurtosis）和 偏度（Skewness）两个统计量。

峰度（Kurtosis）是描述某变量所有取值分布形态陡缓程度的统计量
计算公式：β = M_4 /σ^4 偏度

• – Kurtosis=0 与正态分布的陡缓程度相同
• – Kurtosis>0 比正态分布的高峰陡峭——尖顶峰
• – Kurtosis<0 比正态分布的高峰平缓——平顶峰

偏度（Skewness）是描述某变量取值分布对称性的统计量
计算公式： S= (X^ – M_0)/δ

• – Skewness=0 分布形态与正态分布偏度相同
• – Skewness>0 正偏差数值较大，为正偏或右偏。长尾巴拖在右边
• – Skewness<0 负偏差数值较大，为负偏或左偏。长尾巴拖在左边

# 计算峰度和偏度
print('峰度(Kurtosis): ', train_df['SalePrice'].kurt())
print('偏度(Skewness): ', train_df['SalePrice'].skew())
峰度(Kurtosis):  6.536281860064529
偏度(Skewness):  1.8828757597682129

可以看出，SalePrice高峰陡峭，右偏长尾分布。可以对变量做对数变换，使其呈现正常的正态分布。

plt.figure()
sns.distplot(np.log(train_df['SalePrice']))
plt.title('SalePrice对数变换')
plt.show()

2.2.2 分析特征变量

2.2.2.1 特征变量分类

特征变量按照数据类型分成定量变量和定性变量

quantitative = [feature for feature in train_df.columns if train_df.dtypes[feature] != 'object'] # 定量变量
quantitative.remove('Id')
quantitative.remove('SalePrice')
qualitative = [feature for feature in train_df.columns if train_df.dtypes[feature] == 'object'] # 定性变量

2.2.2.2 定量变量分析

# 定量变量直方图
m_cont = pd.melt(train_df, value_vars=quantitative)
g = sns.FacetGrid(m_cont, col='variable', col_wrap=4, sharex=False, sharey=False)
g.map(sns.distplot, 'value')
<seaborn.axisgrid.FacetGrid at 0x1da87ce1550>

上图中有些变量类似正态分布，可以做对数变换提升数据质量，适合的对象有LotFrontage, TotalBsmtSF, 1stFlrSF，LotArea, GrLivArea

# 定量变量与房价关系图
m_cont = pd.melt(train_df, id_vars='SalePrice', value_vars=quantitative)
g = sns.FacetGrid(m_cont, col='variable', col_wrap=4, sharex=False, sharey=True)
g.map(plt.scatter, 'value', 'SalePrice')
<seaborn.axisgrid.FacetGrid at 0x1da8971cd68>

2.2.2.3 定性变量分析

# 定性变量频数统计图
m_disc = pd.melt(train_df, value_vars=qualitative)
g = sns.FacetGrid(m_disc, col='variable', col_wrap=4, sharex=False, sharey=False)
g.map(sns.countplot, 'value')

# 定性变量与房价关系图
m_disc = pd.melt(train_df, id_vars='SalePrice', value_vars=qualitative)
g = sns.FacetGrid(m_disc, col='variable', col_wrap=4, sharex=False, sharey=True)
g.map(sns.boxplot, 'value', 'SalePrice')

对定性变量进行单因素方差分析，分析定性变量对SalePrice的影响

方差分析（Analysis of Variance，简称ANOVA）又称“变异数分析”或“F检验”，是由R.A.Fister发明的，用于对两个及两个以上的样本集合的统计特性：平均数差别的显著性检验 。

单因素ANOVA也就是单因素方差分析，是用来研究一个控制变量的不同水平是否对观测变量产生了显著影响。通俗地讲就是分析变量x的变化对变量y的影响的显著性，所以一般变量之间存在某种影响关系的，验证一种变量的变化对另一种变量的影响显著性的检验。

Python 实现单因素方差分析使用scipy.stats.f_oneway()方法，传入多个数组，返回F，P

def anova(frame, qualitative):
    anv = pd.DataFrame()
    anv['feature'] = qualitative
    p_vals = []
    for fea in qualitative:
        samples = []
        cls = frame[fea].unique() # 变量的类别值
        for c in cls:
            c_array = frame[frame[fea]==c]['SalePrice'].values
            samples.append(c_array)
        p_val = stats.f_oneway(*samples)[1] # 获得p值,p值越小，对SalePrice的显著性影响越大
        p_vals.append(p_val)
    anv['pval'] = p_vals
    return anv.sort_values('pval')
a = anova(train_df, qualitative)
a['disparity'] = np.log(1./a['pval'].values) # 对SalePrice的影响悬殊度
plt.figure(figsize=(8, 6))
sns.barplot(x='feature', y='disparity', data=a)
plt.xticks(rotation=90)
plt.show()

对定性变量进行数值化编码。变量每个类别值对应的SalePrice均值排序，相应的对每个类别值赋值1，2，3…

def encode(frame, feature):
    ordering = pd.DataFrame()
    ordering['val'] = frame[feature].unique()
    ordering.index = ordering['val']
    ordering['spmean'] = frame[[feature, 'SalePrice']].groupby(feature)['SalePrice'].mean()
    ordering = ordering.sort_values('spmean')
    ordering['ordering'] = np.arange(1, ordering.shape[0]+1)
    ordering = ordering['ordering'].to_dict() # 返回的数据样例{category1:1, category2:2, ...}

    # 对frame[feature]编码
    for category, code_value in ordering.items():
        frame.loc[frame[feature]==category, feature+'_E'] = code_value
qual_encoded = []
for qual in qualitative:
    encode(train_df, qual)
    qual_encoded.append(qual+'_E')
print(qual_encoded)
['MSZoning_E', 'Street_E', 'Alley_E', 'LotShape_E', 'LandContour_E', 'Utilities_E', 'LotConfig_E', 'LandSlope_E', 'Neighborhood_E', 'Condition1_E', 'Condition2_E', 'BldgType_E', 'HouseStyle_E', 'RoofStyle_E', 'RoofMatl_E', 'Exterior1st_E', 'Exterior2nd_E', 'MasVnrType_E', 'ExterQual_E', 'ExterCond_E', 'Foundation_E', 'BsmtQual_E', 'BsmtCond_E', 'BsmtExposure_E', 'BsmtFinType1_E', 'BsmtFinType2_E', 'Heating_E', 'HeatingQC_E', 'CentralAir_E', 'Electrical_E', 'KitchenQual_E', 'Functional_E', 'FireplaceQu_E', 'GarageType_E', 'GarageFinish_E', 'GarageQual_E', 'GarageCond_E', 'PavedDrive_E', 'PoolQC_E', 'Fence_E', 'MiscFeature_E', 'SaleType_E', 'SaleCondition_E']

2.2.2.4 相关性

# 计算特征变量与房价的spearman相关系数
def spearman(frame, features):
    spr =  pd.DataFrame()
    spr['feature'] = features
    spr['spearman'] = [frame[f].corr(frame['SalePrice'], 'spearman') for f in features]
    spr = spr.sort_values('spearman')

    plt.figure(figsize=(6, 0.25*len(features)))
    sns.barplot(x='spearman', y='feature', data=spr)
spearman(train_df, quantitative+qual_encoded)

数据集变量间存在非线性关系时，因此使用spearman相关系数评价变量之间的相关性。可以看出OverallQual,Neighborhood与房价间有较大的正相关性，EnclosedPorch与房价间有较大的负相关性。

# 分析定量变量之间、定性变量之间、定量变量与定性变量之间的相关性
plt.figure(1, figsize=(12,9))
corr = train_df[quantitative+['SalePrice']].corr()
sns.heatmap(corr, vmin=-1, vmax=1)
plt.title('定量变量相关性')

plt.figure(2, figsize=(12,9))
corr = train_df[qual_encoded+['SalePrice']].corr()
sns.heatmap(corr, vmin=-1, vmax=1)
plt.title('定性变量相关性')

plt.figure(3, figsize=(12,9))
corr = pd.DataFrame(np.zeros((len(quantitative)+1, len(qual_encoded)+1)), index=quantitative+['SalePrice'], columns=qual_encoded+['SalePrice'])
for q1 in quantitative+['SalePrice']:
    for q2 in qual_encoded+['SalePrice']:
        corr.loc[q1, q2] = train_df[q1].corr(train_df[q2])
sns.heatmap(corr, vmin=-1, vmax=1)
plt.title('定量变量和定性变量相关性')
Text(0.5,1,'定量变量和定性变量相关性')

2.2.2.5 房价分段

房价分成高价格、标准价格两个等级，分析定量变量与不同价格段的关系

standard = train_df[train_df['SalePrice'] < 200000]
pricey = train_df[train_df['SalePrice'] >= 200000]

features = quantitative

diff = pd.DataFrame()
diff['feature'] = features
diff['difference'] = [(pricey[f].mean()-standard[f].mean())/standard[f].mean() for f in features]

plt.figure(figsize=(8, 6))
sns.barplot(x='feature', y='difference', data=diff)
plt.xticks(rotation=90)
plt.show()

房价以200000为分界点，价格高的房屋有泳池(PoolArea)，装饰石材(MasVnrArea)，开放门廊(OpenPorchSF)

3. 数据清洗

train = pd.read_csv('./data/train.csv')
test = pd.read_csv('./data/test.csv')

3.1 离群点处理

从前面观察分析定量变量与房价关系图看出，GrLivArea存在2个离群点，居住面积大，但是房价低，应该删除。TotalBsmtSF,1stFlrSF,BsmtFinSF1各存在1个离群点，与GrLivArea其中的一个离群点相同。

plt.figure(figsize=(12,6))
plt.scatter(train['GrLivArea'], train['SalePrice'])
plt.xlabel('GrLivArea')
plt.ylabel('SalePrice')
plt.title('GrLivArea与SalePrice散点图，观察异常值')
plt.grid(b=True, ls=':')
plt.show()

# 删除两个离群点
train.drop(train[(train.GrLivArea>4000) & (train.SalePrice<200000)].index, inplace=True)
# 合并训练集和测试集，便于同时对两个数据集进行数据清洗和特征工程
full = pd.concat([train, test], axis=0, ignore_index=True)
train.shape, test.shape, full.shape
((1458, 81), (1459, 80), (2917, 81))

3.2 缺失值处理

# 查看缺失值
missing_count = full.isnull().sum()
missing_count = missing_count[missing_count > 0].sort_values(ascending=False)
missing_rate = missing_count / len(full)

missing = pd.DataFrame()
missing['count'] = missing_count
missing['rate'] = missing_rate

(1) LotFrontage缺失值处理

train_df[quantitative+qual_encoded].corr('spearman')['LotFrontage'].sort_values(ascending=False)

可以看出LotFrontage与LotArea,Neighborhood相关性较大，不过LotArea是连续变量，这里对其区间分段，最后使用每个分组的中位数填补

full['LotAreaCut'] = pd.qcut(full['LotArea'], 10)
full.groupby(['LotAreaCut', 'Neighborhood'])['LotFrontage'].agg(['count', 'mean', 'median'])
full['LotFrontage'] = full.groupby(['LotAreaCut', 'Neighborhood'])['LotFrontage'].transform(lambda x: x.fillna(x.median()))
# 由于某些分组没有数据，因此未填补的缺失值单独利用LotAreaCut填充
full['LotFrontage'] = full.groupby(['LotAreaCut'])['LotFrontage'].transform(lambda x: x.fillna(x.median()))

(2) 某些类别变量有缺失值是因为没有这个属性，填补’None’ 比如PoolQC(泳池质量)，缺失值是因为这个房子没有泳池

cols = ["PoolQC" , "MiscFeature", "Alley", "Fence", "FireplaceQu", "GarageQual", "GarageCond", "GarageFinish", "GarageYrBlt", "GarageType", "BsmtExposure", "BsmtCond", "BsmtQual", "BsmtFinType2", "BsmtFinType1", "MasVnrType"]
for col in cols:
    full[col] = full[col].fillna('None')

(3)某些数值变量有缺失值是因为没有这个属性，填补0 比如GarageArea(车库面积)，缺失值是因为这个房子没有车库

cols2 = ["MasVnrArea", "BsmtUnfSF", "TotalBsmtSF", "GarageCars", "BsmtFinSF2", "BsmtFinSF1", "GarageArea"]
for col in cols2:
    full[col] = full[col].fillna(0)

(4)其他缺失值变量都是离散变量，缺失值数目较少，用众数填补

cols3 = ["MSZoning", "BsmtFullBath", "BsmtHalfBath", "Utilities", "Functional", "Electrical", "KitchenQual", "SaleType","Exterior1st", "Exterior2nd"]
for col in cols3:
    full[col] = full[col].fillna(full[col].mode()[0])

(5)检查有无缺失值

full.isnull().sum()[full.isnull().sum()>0] # 缺失值填补完毕
SalePrice    1459
dtype: int64
full.drop('LotAreaCut', axis=1, inplace=True)
full.shape
(2917, 81)