CDH 使用 Hive schematool

翻译: https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hive_schema_tool.html
版本: 5.14.2

使用Hive命令行 schematool 为非托管集群升级或验证Hive Metastore数据库schema 。

注意:

如果您正在使用Cloudera Manager来管理群集,则schematool 也可以在Hive服务页面中验证或升级Metastore:

  1. 从Cloudera Manager管理控制台中,选择Hive服务。
    • 要验证架构,请在Hive服务页面上单击Actions ,然后选择Validate Hive Metastore Schema。
    • 升级模式:
      1. 在Hive服务页面上,单击Actions ,然后选择 停止服务。
      2. 仍然在Hive服务页面上,单击Actions ,然后选择升级Hive数据库Metastore架构。
      3. 升级完成后,重新启动服务。

继续阅读:

Schema版本Verification 和Validation

Hive将schema 版本记录在Metastore数据库中,并验证Metastore schema版本是否与即将访问Metastore的Hive二进制文件兼容。隐式创建或更改现有schema的Hive配置属性在默认情况下处于禁用状态。因此,Hive不会尝试隐式更改Metastore schema 。当您对schema 未初始化或 旧schema的Metastore执行Hive查询时,它无法访问Metastore,并且错误日志中会显示与以下示例类似的条目:

...
Caused by: MetaException(message:Version information not found in metastore. )
        at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:5638)
...

使用Hive schematool 通过初始化schema或升级它来修复导致此错误的条件。

使用schematool

使用Hive schematool 为当前Hive版本初始化Metastore schema 或从旧版本升级架构。如果该工具在该处可用,该工具将尝试从Metastore中查找当前schema。

schematool 确定初始化或升级schema所需的SQL脚本,然后在Metastore数据库执行这些脚本。Metabase数据库连接信息(如JDBC URL,JDBC驱动程序和数据库凭证)从Hive配置中提取。如果需要,您可以提供备用数据库凭据。

以下选项是schematool 其中的一部分:

$ schematool -help
usage: schemaTool
 -dbType <databaseType>             Metastore database type                 
 -dryRun                                  List SQL scripts (no execute)                                       
 -help                                    Print this message
 -info                                    Show config and schema details
 -initSchema                              Schema initialization
 -initSchemaTo <initTo>             Schema initialization to a version
 -passWord <password>               Override config file password
 -upgradeSchema                           Schema upgrade
 -upgradeSchemaFrom <upgradeFrom>   Schema upgrade from a version
 -userName <user>                   Override config file user name
 -validate                                Validate the database
 -verbose                                 Only print SQL statements

dbType 选项必须始终指定,并且可以是以下之一:

derby|mysql|postgres|oracle

先决条件配置

在你使用schematool之前 ,您必须将以下属性添加到/etc/hive/conf/hive-site.xml 文件:

  • javax.jdo.option.ConnectionURL
  • javax.jdo.option.ConnectionDriverName

例如,以下 hive-site.xml, 如果您使用MySQL数据库作为您的Hive Metastore,hive1 是数据库用户名:

<property>
   <name>javax.jdo.option.ConnectionURL</name>
   <value>jdbc:mysql://my_cluster.com:3306/hive1?useUnicode=true&amp;characterEncoding=UTF-8</value>
</property>
<property>
   <name>javax.jdo.option.ConnectionDriverName</name>
   <value>com.mysql.jdbc.Driver</value>
</property>

用法示例

使用 schematool 命令行工具,导航到它所在的目录:

  • 如果您使用parcel安装CDH,schematool 通常位于:/opt/cloudera/parcels/CDH/lib/hive/bin/schematool

  • 如果您使用软件包安装CDH, schematool 通常位于:/usr/lib/hive/bin/schematool

找到可执行文件后,可以使用 schematool 执行以下操作:

  • 使用initSchema 将您的Metastore初始化为新的Hive。
$ schematool -dbType mysql -initSchema -passWord <db_user_pswd> -userName
  <db_user_name>
Metastore connection URL:
jdbc:mysql://<cluster_address>:3306/<user_name>?useUnicode=true&characterEncoding=UTF-8
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: <user_name>
Starting metastore schema initialization to <new_version>
Initialization script hive-schema-<new_version_number>.mysql.sql
Initialization script completed
schemaTool completed
  • 使用info 获取模式信息。
$ schematool -dbType mysql -info -passWord <db_user_pswd> -userName
  <db_user_name>
Metastore connection URL:
jdbc:mysql://<cluster_address>:3306/<user_name>?useUnicode=true&characterEncoding=UTF-8
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: <user_name>
Hive distribution version:       <new_version>
Required schema version:         <new_version>
Metastore schema version:        <new_version>
schemaTool completed
  • 如果您尝试从未存储版本信息的旧版metastore获取架构信息,或者架构未初始化,则该工具将按如下所示报告错误。
$ schematool -dbType mysql -info -passWord <db_user_pswd> -userName
  <db_user_name>
Metastore connection URL:
jdbc:mysql://<cluster_address>:3306/<user_name>?useUnicode=true&characterEncoding=UTF-8
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: <user_name>
Hive distribution version:       <new_version>
Required schema version:         <new_version>
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version,
Cause:<cause_description>
*** schemaTool failed ***
  • 您可以通过指定特定发行版来升级架构 -upgradeSchemaFrom 。-upgradeSchemaFrom选项需要Hive版本而不是CDH版本。有关每个CDH版本附带哪个Hive版本的信息,请参阅CDH 5 Packaging and Tarball信息。以下示例显示了如何从CDH 5.2 / Hive 0.13.1升级:
$ schematool -dbType mysql -passWord <db_user_pswd> -upgradeSchemaFrom
  0.13.1 -userName <db_user_name>
Metastore connection URL:
jdbc:mysql://<cluster_address>:3306/<user_name>?useUnicode=true&characterEncoding=UTF-8
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: <user_name>
Starting upgrade metastore schema from version 0.13.1 to <new_version>
Upgrade script upgrade-0.13.1-to-<new_version>.mysql.sql
Completed pre-0-upgrade-0.13.1-to-<new_version>.mysql.sql
Completed upgrade-0.13.1-to-<new_version>.mysql.sql
schemaTool completed
  • 使用 -validate 选项来验证Metastore schema。以下示例显示schematool 与此选项一起使用时对Metastore模式执行的验证类型:
$ schematool -dbType mysql -passWord <db_user_pswd> -userName
               <db_user_name> -validate
Starting metastore validation

Validating schema version
Succeeded in schema version validation.
[SUCCESS]

Validating sequence number for SEQUENCE_TABLE
Succeeded in sequence number validation for SEQUENCE_TABLE
[SUCCESS]

Validating metastore schema tables
Succeeded in schema table validation.
[SUCCESS]

Validating database/table/partition locations
Succeeded in database/table/partition location validation
[SUCCESS]

Validating columns for incorrect NULL values
Succeeded in column validation for incorrect NULL values
[SUCCESS]

Done with metastore validation: [SUCCESS]
schemaTool completed
  • 如果您想查找schema升级所需的所有脚本,请使用 dryRun 选项。
$ schematool -dbType mysql -upgradeSchemaFrom 0.10.0 -dryRun -passWord
              <db_user_pswd> -userName <db_user_name>
Metastore connection URL:
jdbc:mysql://<cluster_address>:3306/<user_name>?useUnicode=true&characterEncoding=UTF-8
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: <user_name>
Starting upgrade metastore schema from version 0.10.0 to <new_version>
Upgrade script upgrade-0.10.0-to-0.11.0.mysql.sql
Upgrade script upgrade-0.11.0-to-0.12.0.mysql.sql
Upgrade script upgrade-0.12.0-to-0.13.0.mysql.sql
Upgrade script upgrade-0.13.0-to-0.14.0.mysql.sql
Upgrade script upgrade-0.14.0-to-1.1.0.mysql.sql
Upgrade script upgrade-1.1.0-to-<new_version>.mysql.sql
schemaTool completed
    原文作者:金刚_30bf
    原文地址: https://www.jianshu.com/p/a6b8591573b1
    本文转自网络文章,转载此文章仅为分享知识,如有侵权,请联系博主进行删除。
点赞