Hive实验5：查看Hql执行计划及关键步骤说明

2024年3月29日 154次阅读 来源: 英雄史诗

1、查看执行计划方法

语法：explain [extended] Hiveql;

/*例子：*/
explain select count(distinct mobilename) from testtab_small;

2、执行计划基本要素

  1. 主要步骤及依赖关系，从上到下
  2. 各主要步骤关键信息，包括：
关键信息关键字说明
Map或reduce操作Map Operator Tree、Reduce Operator Treemap、reduce阶段
扫描表TableScan要查询的表
表数据量统计Statistics包括行数、数据大小
查询算子Select Operator要检索的字段
分区算子Group By Operator聚合如count()等需要
排序sort order是否排序，+表排序，空不排序
是否本地任务Local Work、Local Tables、Map Local Operator Tree见于Map端的连接，有小表参与连接、且auto.convert.join=true
连接算子Join Operator连接
连接条件condition map连接条件：Left Outer Join0 to 1

2、explain select count(distinct mobilename) from testtab_small

完整的执行计划示例。之后的主要体现差异，不是全部，比如删除Stage0部分。

/*整体步骤，从上往下看*/
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Map Operator Tree:
      	  /*扫描的表*/
          TableScan
            alias: testtab_small
            /*表数据量统计，13条，629字节？*/
            Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
            Select Operator
              /*检索的字段是mobilename*/
              expressions: mobilename (type: string)
              outputColumnNames: mobilename
              Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
              Group By Operator
                /*聚合操作*/
                aggregations: count(DISTINCT mobilename)
                keys: mobilename (type: string)
                mode: hash
                outputColumnNames: _col0, _col1
                Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
                Reduce Output Operator
                  key expressions: _col0 (type: string)
                  /*+ 需要排序*/
                  sort order: +
                  Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
      Reduce Operator Tree:
        Group By Operator
          aggregations: count(DISTINCT KEY._col0:0._col0)
          mode: mergepartial
          outputColumnNames: _col0
          Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
          File Output Operator
            compressed: false
            Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
            table:
                input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

  Stage: Stage-0
    Fetch Operator
      limit: -1
      Processor Tree:
        ListSink

3、explain select count(mobilename) from testtab_small;

  Stage: Stage-1
    Map Reduce
      Map Operator Tree:
          TableScan
            alias: testtab_small
            Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
            Select Operator
              expressions: mobilename (type: string)
              outputColumnNames: mobilename
              Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
              Group By Operator
                aggregations: count(mobilename)
                mode: hash
                outputColumnNames: _col0
                Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
                Reduce Output Operator
                  sort order:
                  Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
                  value expressions: _col0 (type: bigint)
      Reduce Operator Tree:
        Group By Operator
          aggregations: count(VALUE._col0)
          mode: mergepartial
          outputColumnNames: _col0
          Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
          File Output Operator
            compressed: false
            Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
            table:
                input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

4、全局排序

explain select * from testtab_small order by mobilename;
  1. 无聚合算子
  2. 在Map Operator Tree下的Reduce Output Operator下就有排序了(sort order: +)
  Stage: Stage-1
    Map Reduce
      Map Operator Tree:
          TableScan
            alias: testtab_small
            Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
            Select Operator
              expressions: mobilename (type: string), testrecordid (type: string)
              outputColumnNames: _col0, _col1
              Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
              Reduce Output Operator
                key expressions: _col0 (type: string)
                sort order: +
                Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
                value expressions: _col1 (type: string)
      Reduce Operator Tree:
        Select Operator
          expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string)
          outputColumnNames: _col0, _col1
          Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
          File Output Operator
            compressed: false
            Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
            table:
                input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

5、关联小表，Map端连接

hive.auto.convert.join=true /默认值/

hive> explain select a.testrecordid,a.mobilename,b.mobilename from testtab_small a left join testtab_small2 b on a.testrecordid=b.testrecordid;
  1. 总步骤增至3个
  2. 多了本地任务，Map Reduce Local Work。即hadoop的分布式缓存技术
  3. Map Join Operator
STAGE DEPENDENCIES:
  Stage-4 is a root stage
  Stage-3 depends on stages: Stage-4
  Stage-0 depends on stages: Stage-3

STAGE PLANS:
  Stage: Stage-4
    Map Reduce Local Work
      Alias -> Map Local Tables:
        $hdt$_1:b
          Fetch Operator
            limit: -1
      Alias -> Map Local Operator Tree:
        $hdt$_1:b
          TableScan
            alias: b
            Statistics: Num rows: 13 Data size: 655 Basic stats: COMPLETE Column stats: NONE
            Select Operator
              expressions: mobilename (type: string), testrecordid (type: string)
              outputColumnNames: _col0, _col1
              Statistics: Num rows: 13 Data size: 655 Basic stats: COMPLETE Column stats: NONE
              HashTable Sink Operator
                keys:
                  0 _col1 (type: string)
                  1 _col1 (type: string)

  Stage: Stage-3
    Map Reduce
      Map Operator Tree:
          TableScan
            alias: a
            Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
            Select Operator
              expressions: mobilename (type: string), testrecordid (type: string)
              outputColumnNames: _col0, _col1
              Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
              Map Join Operator
                condition map:
                     Left Outer Join0 to 1
                keys:
                  0 _col1 (type: string)
                  1 _col1 (type: string)
                outputColumnNames: _col0, _col1, _col2
                Statistics: Num rows: 14 Data size: 691 Basic stats: COMPLETE Column stats: NONE
                Select Operator
                  expressions: _col1 (type: string), _col0 (type: string), _col2 (type: string)
                  outputColumnNames: _col0, _col1, _col2
                  Statistics: Num rows: 14 Data size: 691 Basic stats: COMPLETE Column stats: NONE
                  File Output Operator
                    compressed: false
                    Statistics: Num rows: 14 Data size: 691 Basic stats: COMPLETE Column stats: NONE
                    table:
                        input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                        output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                        serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
      Local Work:
        Map Reduce Local Work

6、Reduce端连接

hive.auto.convert.join=false
与5相同的语句，Reduce端连接执行计划：

  1. Map Operator Tree里有2个评级的TableScan，对应Mapper多个数据来源
  2. Reduce Operator Tree下Join Operator：reduce端连接。
STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Map Operator Tree:
          TableScan
            alias: a
            Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
            Select Operator
              expressions: mobilename (type: string), testrecordid (type: string)
              outputColumnNames: _col0, _col1
              Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
              Reduce Output Operator
                key expressions: _col1 (type: string)
                sort order: +
                Map-reduce partition columns: _col1 (type: string)
                Statistics: Num rows: 13 Data size: 629 Basic stats: COMPLETE Column stats: NONE
                value expressions: _col0 (type: string)
          TableScan
            alias: b
            Statistics: Num rows: 13 Data size: 655 Basic stats: COMPLETE Column stats: NONE
            Select Operator
              expressions: mobilename (type: string), testrecordid (type: string)
              outputColumnNames: _col0, _col1
              Statistics: Num rows: 13 Data size: 655 Basic stats: COMPLETE Column stats: NONE
              Reduce Output Operator
                key expressions: _col1 (type: string)
                sort order: +
                Map-reduce partition columns: _col1 (type: string)
                Statistics: Num rows: 13 Data size: 655 Basic stats: COMPLETE Column stats: NONE
                value expressions: _col0 (type: string)
      Reduce Operator Tree:
        Join Operator
          condition map:
               Left Outer Join0 to 1
          keys:
            0 _col1 (type: string)
            1 _col1 (type: string)
          outputColumnNames: _col0, _col1, _col2
          Statistics: Num rows: 14 Data size: 691 Basic stats: COMPLETE Column stats: NONE
          Select Operator
            expressions: _col1 (type: string), _col0 (type: string), _col2 (type: string)
            outputColumnNames: _col0, _col1, _col2
            Statistics: Num rows: 14 Data size: 691 Basic stats: COMPLETE Column stats: NONE
            File Output Operator
              compressed: false
              Statistics: Num rows: 14 Data size: 691 Basic stats: COMPLETE Column stats: NONE
              table:
                  input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
    原文作者：英雄史诗
    原文地址: https://blog.csdn.net/heroicpoem/article/details/100535316
    本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系博主进行删除。
点赞