Cloudera Hadoop 分层存储实战

2023年12月31日 213次阅读 来源: shaopi0211

HDFS分层存储

why

为何要使用分层存储(充分利用资源,降低成本)
http://developer.51cto.com/art/201504/470660.htm

HDFS从Hadoop2.3开始支持分层存储

archive节点设置

为了表明这个本地目录属于归档层，该本地目录配置中会带有[ARCHIVE]的前缀

例:dfs.datanode.data.dir 参数设置为 [ARCHIVE]/hdfs01/dfs/dn

查看分层存储命令

# 基本命令
[dengsc@nfjd-hadoop-test01 ~]$ hdfs storagepolicies
Usage: bin/hdfs storagepolicies [COMMAND]
          [-listPolicies]
          [-setStoragePolicy -path <path> -policy <policy>]
          [-getStoragePolicy -path <path>]
          [-help <command-name>]

# 分层策略查看
[dengsc@nfjd-hadoop-test01 ~]$ hdfs storagepolicies -listPolicies
Block Storage Policies:
    BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}
    BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE], replicationFallbacks=[DISK, ARCHIVE]}
    BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
    BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD, DISK]}
    BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]}
    BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK], replicationFallbacks=[DISK]}
    
# 策略副本存放
Hot - for both storage and compute. The data that is popular and still being used for processing will stay in this policy. When a block is hot, all replicas are stored in DISK.
Cold - only for storage with limited compute. The data that is no longer being used, or data that needs to be archived is moved from hot storage to cold storage. When a block is cold, all replicas are stored in ARCHIVE.
Warm - partially hot and partially cold. When a block is warm, some of its replicas are stored in DISK and the remaining replicas are stored in ARCHIVE.
All_SSD - for storing all replicas in SSD.
One_SSD - for storing one of the replicas in SSD. The remaining replicas are stored in DISK.
Lazy_Persist - for writing blocks with single replica in memory. The replica is first written in RAM_DISK and then it is lazily persisted in DISK.

使用示例

设置文件夹为cold类型

[dengsc@nfjd-hadoop-test01 ~]$ hdfs storagepolicies -setStoragePolicy -path /user/dengsc/cold -policy COLD
Set storage policy COLD on /user/dengsc/cold

查看文件夹分层属性

[dengsc@nfjd-hadoop-test01 ~]$ hdfs storagepolicies -getStoragePolicy -path /user/dengsc/cold
The storage policy of /user/dengsc/cold:
BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}

put文件后查看文件存放位置(block全在archive节点,此集群副本冗余为2,所以只能看到两个block)

[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -put a.txt /user/dengsc/cold
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -ls /user/dengsc/cold/a.txt
-rw-r--r--   2 dengsc supergroup         21 2017-09-20 20:34 /user/dengsc/cold/a.txt
[dengsc@nfjd-hadoop-test01 ~]$ hdfs fsck /user/dengsc/cold/a.txt -files -blocks -locations
Connecting to namenode via http://nfjd-hadoop02-node179.jpushoa.com:50070
FSCK started by dengsc (auth:SIMPLE) from /192.168.250.40 for path /user/dengsc/cold/a.txt at Wed Sep 20 20:35:19 CST 2017
/user/dengsc/cold/a.txt 21 bytes, 1 block(s):  OK
0. BP-535292861-192.168.250.41-1481889113054:blk_1108602669_34871883 len=21 Live_repl=2 [DatanodeInfoWithStorage[192.168.254.228:50010,DS-c87cf985-e8dd-4b97-908c-873318c243d9,ARCHIVE], DatanodeInfoWithStorage[192.168.254.227:50010,DS-876ae32f-c8a3-4c25-8047-1ebd5c59a407,ARCHIVE]]

mv文件至cold文件夹(block还是在disk节点,mv文件需要启动mover来重新放置block)

[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -put a.txt /user/dengsc/b.txt
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -ls /user/dengsc/b.txt
-rw-r--r--   2 dengsc supergroup         21 2017-09-20 20:40 /user/dengsc/b.txt
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -mv /user/dengsc/b.txt /user/dengsc/cold
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -ls /user/dengsc/cold/b.txt
-rw-r--r--   2 dengsc supergroup         21 2017-09-20 20:40 /user/dengsc/cold/b.txt
[dengsc@nfjd-hadoop-test01 ~]$ hdfs fsck /user/dengsc/cold/b.txt -files -blocks -locations
Connecting to namenode via http://nfjd-hadoop02-node179.jpushoa.com:50070
FSCK started by dengsc (auth:SIMPLE) from /192.168.250.40 for path /user/dengsc/cold/b.txt at Wed Sep 20 20:41:01 CST 2017
/user/dengsc/cold/b.txt 21 bytes, 1 block(s):  OK
0. BP-535292861-192.168.250.41-1481889113054:blk_1108602676_34871890 len=21 Live_repl=2 [DatanodeInfoWithStorage[192.168.250.40:50010,DS-67f618a4-e49e-42b2-91c2-69550fd7a769,DISK], DatanodeInfoWithStorage[192.168.250.29:50010,DS-19c6fcd2-528a-4193-a0e2-c76d001ee5ff,DISK]]

启动hdfs mover(日志显示block从disk节点移动到archive节点)

-bash-4.2$ hdfs mover -p /user/dengsc/cold/
17/09/20 20:43:17 INFO mover.Mover: namenodes = {hdfs://nameservice-test1=[/user/dengsc/cold]}
17/09/20 20:43:19 INFO net.NetworkTopology: Adding a new node: /default/192.168.250.40:50010
17/09/20 20:43:19 INFO net.NetworkTopology: Adding a new node: /default/192.168.254.228:50010
17/09/20 20:43:19 INFO net.NetworkTopology: Adding a new node: /default/192.168.250.41:50010
17/09/20 20:43:19 INFO net.NetworkTopology: Adding a new node: /default/192.168.254.227:50010
17/09/20 20:43:19 INFO net.NetworkTopology: Adding a new node: /default/192.168.250.29:50010
17/09/20 20:43:19 INFO balancer.Dispatcher: Successfully moved blk_1108602676_34871890 with size=21 from 192.168.250.40:50010:DISK to 192.168.254.228:50010:ARCHIVE through 192.168.250.40:50010
17/09/20 20:43:58 INFO balancer.Dispatcher: Successfully moved blk_1108602676_34871890 with size=21 from 192.168.250.29:50010:DISK to 192.168.254.227:50010:ARCHIVE through 192.168.254.228:50010
Sep 20, 2017 8:44:37 PM  Mover took 1mins, 21sec

再次查看block分布(block已经分布在archive节点)

-bash-4.2$ hdfs fsck /user/dengsc/cold/b.txt -files -blocks -locations
Connecting to namenode via http://nfjd-hadoop02-node179.jpushoa.com:50070
FSCK started by hdfs (auth:SIMPLE) from /192.168.250.40 for path /user/dengsc/cold/b.txt at Wed Sep 20 20:46:05 CST 2017
/user/dengsc/cold/b.txt 21 bytes, 1 block(s):  OK
0. BP-535292861-192.168.250.41-1481889113054:blk_1108602676_34871890 len=21 Live_repl=2 [DatanodeInfoWithStorage[192.168.254.228:50010,DS-39cb2b6f-c410-4c6d-9e31-b1aa845fb44d,ARCHIVE], DatanodeInfoWithStorage[192.168.254.227:50010,DS-7be78d22-fbbd-4e4c-b5a8-5768e77621f7,ARCHIVE]]

注：直接向cold属性文件夹cp或者put文件，文件类型会继承文件夹分层属性，但是mv操作不会继承分层属性。

warm类型操作类似cold。

官方文档
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html

    原文作者：shaopi0211
    原文地址: https://www.jianshu.com/p/cc62606fcb40
    本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系博主进行删除。
点赞