将key相对分散,并且数据量小的表放在join的左边,这样可以有效减少内存溢出错误发生的几率;再进一步,可以使用Group让小的维度表(1000条以下的记录条数)先进内存。在map端完成reduce。
实际测试发现:新版的hive已经对小表JOIN大表和大表JOIN小表进行了优化。小表放在左边和右边已经没有明显区别。
案例实操
测试大表JOIN小表和小表JOIN大表的效率
1、建大表、小表和JOIN后表的语句
hive (default)> create table bigtable(id bigint, time bigint, uid string,
keyword string,url_rank int, click_num int, click_url string)
row format delimited fields
terminated by '\t';
hive (default)> create table smalltable(id bigint, time bigint, uid string,
keyword string,url_rank int, click_num int, click_url string)
row format delimited fields
terminated by '\t';
hive (default)> create table jointable(id bigint, time bigint, uid string,
keyword string,url_rank int, click_num int, click_url string)
row format delimited fields
terminated by '\t';
2、分别向大表和小表中导入数据
hive (default)> load data local inpath '/opt/module/datas/bigtable'
into table bigtable;
hive (default)>load data local inpath '/opt/module/datas/smalltable'
into table smalltable;
3、关闭mapjoin功能(默认是打开的)
hive (default)> set hive.auto.convert.join = false;
4、执行小表JOIN大表语句
hive (default)> insert overwrite table jointable
select b.id, b.time, b.uid, b.keyword, b.url_rank, b.click_num, b.click_url
from smalltable s
left join bigtable b
on b.id = s.id;
Time taken: 35.921 seconds
5、执行大表JOIN小表语句
hive (default)> insert overwrite table jointable
select b.id, b.time, b.uid, b.keyword, b.url_rank, b.click_num, b.click_url
from bigtable b
left join smalltable s
on s.id = b.id;
Time taken: 34.196 seconds