Mysql数据导出到Hbase

2023年1月18日 184次阅读 来源: gofun

Mysql表center_pub_merchant数据导出为center_pub_merchant.tsv

mysql -h 172.xx.x.175 -P 3307 -uUSER -pPASSWORD -e "SELECT concat('merchant:',id), parentId, id, cascade_label AS cascadeLabel, cascade_level AS cascadeLevel, account_id AS accountId, merchant_name AS merchantName,merchant_type AS merchantType,merchant_project AS merchantProject, full_name AS fullName, telephone, email, thumb, industry, user_id AS userId, province,city, county FROM center_pub_merchant" awifi_dc > /opt/gofunTest/center_pub_merchant.tsv

hdfs dfs -put center_pub_merchant.tsv /tmp/center_pub_merchant.tsv

hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=HBASE_ROW_KEY,base_data:parentId,base_data:id,base_data:cascadeLabel,base_data:cascadeLevel,base_data:accountId,base_data:merchantName,base_data:merchantType,base_data:merchantProject,base_data:fullName,base_data:telephone,base_data:email,base_data:thumb,base_data:industry,base_data:userId,base_data:province,base_data:city,base_data:county tj_base_merchant /tmp/center_pub_merchant.tsv

Mysql表center_pub_device数据导出为center_pub_device.tsv

mysql -h 172.XX.X.175 -P 3307 -uUSER -pPASSWORD -e “SELECT concat(‘device:’,d.device_id),d.province,d.city,d.county,e.entity_type as entityType,d.device_id as deviceId,d.project_id as projectId,d.belongto FROM center_pub_device d INNER JOIN center_pub_entity e ON d.OUT_ID = e.ID WHERE d.OUT_TYPE_ID = ’00′” awifi_dc > /opt/gofunTest/center_pub_device.tsv

hdfs dfs -put center_pub_device.tsv /tmp/center_pub_device.tsv

hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=HBASE_ROW_KEY,base_data:province,base_data:city,base_data:county,base_data:entityType,base_data:deviceId,base_data:projectId,base_data:belongto tj_base_device /tmp/center_pub_device.tsv

注意：

  • 若Permission denied: user=hbase, access=WRITE, inode=”/user”:hdfs:supergroup:drwxr-xr-x
    修改 hadoop fs -chmod 777 /user
  • 若报min.user.id。。。的错
    找到这个文件/etc/hadoop/conf.cloudera.yarn/container-executor.cfg
    chmod 755 container-executor.cfg
    将 min.user.id=900改小
    chmod 400 container-executor.cfg
  • 若报banned.users。。。的错
    chmod 755 container-executor.cfg
    则将banned.users=hdfs,yarn,mapred,bin中hdfs删掉
    chmod 400 container-executor.cfg
    原文作者：gofun
    原文地址: https://www.jianshu.com/p/510d21603998
    本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系博主进行删除。
点赞