基于Docker的Hadoop集群构建
0. 绪论
使用Docker搭建Hadoop技术平台，包括安装Docker、Java、Scala、Hadoop、 Hbase、Spark。
集群共有5台机器，主机名分别为 h01、h02、h03、h04、h05。其中 h01 为 master，其他的为 slave。
- JDK 1.8
- Scala 2.11.6
- Hadoop 3.2.0
- Hbase 2.1.3
- Spark 2.4.0
1. Docker
1.1 安装Docker
1.1.1 Ubuntu 16.04 安装 Docker
在 Ubuntu 下对 Docker 的操作都需要加上
sudo，如果已经是 root 账号了，则不需要。如果不加
sudo，Docker 相关命令会无法执行。
在 Ubuntu 下安装 Docker 的时候需在管理员的账号下操作。
dhu719@dhu719:~$ wget -qO- https://get.docker.com/ | sh安装完成之后，以 sudo 启动 Docker 服务。
dhu719@dhu719:~$ sudo service docker start显示 Docker 中所有正在运行的容器，由于 Docker 才安装，我们没有运行任何容器，所以显示结果如下所示。
dhu719@dhu719:~$ sudo docker ps
[sudo] password for dhu719:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dhu719@dhu719:~$1.1.2 MacOS 安装 Docker
MacOS 既可以使用 Homebrew 安装 Docker 也可以使用 .dmg 镜像安装，但是 Homebrew 不换源，下载速度会非常慢。
如果 Homebrew 未安装，使用脚本安装 Homebrew 的时候会非常慢，可以使用下载 .dmg 文件的安装方式。如果 Homebrew 已安装好推荐使用
brew命令安装。
使用 Homebrew 安装 Docker，命令如下
qigangdeMacBook-Pro:~ qigang$ brew cask install docker下载 .dmg 的地址如下，下载后安装
https://download.docker.com/mac/stable/Docker.dmgDocker 安装完成后，在应用中找到 Docker 的图标，点击运行 Docker，状态栏中会显示一个小鲸鱼的图标。
打开终端，显示Docker中的容器。
qigangdeMacBook-Pro:~ qigang$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
qigangdeMacBook-Pro:~ qigang$1.2 使用Docker
现在的 Docker 网络能够提供 DNS 解析功能，我们可以使用如下命令为接下来的 Hadoop 集群单独构建一个虚拟的网络。
dhu719@dhu719:~$ sudo docker network create --driver=bridge hadoop以上命令创建了一个名为 Hadoop 的虚拟桥接网络，该虚拟网络内部提供了自动的DNS解析服务。
使用下面这个命令查看 Docker 中的网络，可以看到刚刚创建的名为 hadoop 的虚拟桥接网络。
dhu719@dhu719:~$ sudo docker network ls
NETWORK ID NAME DRIVER SCOPE
06548c9440f8 bridge bridge local
b21dba8dc351 hadoop bridge local
eb48a64969d1 host host local
3e8c9d771ec8 none null local
dhu719@dhu719:~$查找 ubuntu 容器
dhu719@dhu719:~$ sudo docker search ubuntu
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
ubuntu Ubuntu is a Debian-based Linux operating sys… 9326 [OK]
dorowu/ubuntu-desktop-lxde-vnc Docker image to provide HTML5 VNC interface … 281 [OK]
rastasheep/ubuntu-sshd Dockerized SSH service, built on top of offi… 209 [OK]
consol/ubuntu-xfce-vnc Ubuntu container with "headless" VNC session… 161 [OK]
ubuntu-upstart Upstart is an event-based replacement for th… 97 [OK]
ansible/ubuntu14.04-ansible Ubuntu 14.04 LTS with ansible 96 [OK]
neurodebian NeuroDebian provides neuroscience research s… 56 [OK]
1and1internet/ubuntu-16-nginx-php-phpmyadmin-mysql-5 ubuntu-16-nginx-php-phpmyadmin-mysql-5 49 [OK]
ubuntu-debootstrap debootstrap --variant=minbase --components=m… 40 [OK]
nuagebec/ubuntu Simple always updated Ubuntu docker images w… 23 [OK]
tutum/ubuntu Simple Ubuntu docker images with SSH access 19下载 ubuntu 16.04 版本的镜像文件
dhu719@dhu719:~$ sudo docker pull ubuntu:16.04
16.04: Pulling from library/ubuntu
34667c7e4631: Pull complete
d18d76a881a4: Pull complete
119c7358fbfc: Pull complete
2aaf13f3eff0: Pull complete
Digest: sha256:58d0da8bc2f434983c6ca4713b08be00ff5586eb5cdff47bcde4b2e88fd40f88
Status: Downloaded newer image for ubuntu:16.04
dhu719@dhu719:~$查看已经下载的镜像
dhu719@dhu719:~$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> ccac37c7045c 4 days ago 1.85GB
ubuntu 16.04 9361ce633ff1 7 days ago 118MB
dhu719@dhu719:~$根据镜像启动一个容器，可以看出 shell 已经是容器的 shell 了
dhu719@dhu719:~$ sudo docker run -it ubuntu:16.04 /bin/bash
root@fab4da838c2f:/#输入 exit 可以退出容器
root@fab4da838c2f:/# exit
exit
dhu719@dhu719:~$查看本机上所有的容器
dhu719@dhu719:~$ sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fab4da838c2f ubuntu:16.04 "/bin/bash" 2 minutes ago Exited (0) 49 seconds ago nifty_pascal
dhu719@dhu719:~$启动一个状态为退出的容器，最后一个参数为容器 ID
dhu719@dhu719:~$ sudo docker start fab4da838c2f
fab4da838c2f
dhu719@dhu719:~$进入一个容器
并不是所有容器都可以这么干
dhu719@dhu719:~$ sudo docker exec -it fab4da838c2f /bin/bash
root@fab4da838c2f:/#查看正在运行的容器
dhu719@dhu719:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fab4da838c2f ubuntu:16.04 "/bin/bash" 7 minutes ago Up 3 minutes nifty_pascal
dhu719@dhu719:~$关闭一个容器
dhu719@dhu719:~$ sudo docker stop fab4da838c2f
fab4da838c2f
dhu719@dhu719:~$以上，Docker 的基础使用，在下面会用到，大部分的操作都会在容器里完成，比用虚拟机安装好多了。
2. 安装集群
主要是安装 JDK 1.8 的环境，因为 Spark 要 Scala，Scala 要 JDK 1.8，以及 Hadoop，以此来构建基础镜像。
2.1 安装 Java 与 Scala
进入之前的 Ubuntu 容器
先更换 apt 的源
2.1.1 修改 apt 源
备份源
root@fab4da838c2f:/# cp /etc/apt/sources.list /etc/apt/sources_init.list
root@fab4da838c2f:/#先删除就源文件，这个时候没有 vim 工具..
root@fab4da838c2f:/# rm /etc/apt/sources.list使用 echo 命令将源写入新文件
root@fab4da838c2f:/# echo "deb http://mirrors.aliyun.com/ubuntu/ xenial main
> deb-src http://mirrors.aliyun.com/ubuntu/ xenial main
>
> deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main
> deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates main
>
> deb http://mirrors.aliyun.com/ubuntu/ xenial universe
> deb-src http://mirrors.aliyun.com/ubuntu/ xenial universe
> deb http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
> deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
>
> deb http://mirrors.aliyun.com/ubuntu/ xenial-security main
> deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security main
> deb http://mirrors.aliyun.com/ubuntu/ xenial-security universe
> deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security universe" > /etc/apt/sources.list
root@fab4da838c2f:#阿里源如下
deb http://mirrors.aliyun.com/ubuntu/ xenial main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial main
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates main
deb http://mirrors.aliyun.com/ubuntu/ xenial universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial universe
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
deb http://mirrors.aliyun.com/ubuntu/ xenial-security main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security main
deb http://mirrors.aliyun.com/ubuntu/ xenial-security universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security universe再使用 apt update 来更新
2.1.2 安装 Scala 与 Java
直接输入命令
root@fab4da838c2f:/# apt install openjdk-8-jdk来安装 jdk 1.8
测试一下安装结果
root@fab4da838c2f:/# java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.16.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
root@fab4da838c2f:/#再输入
root@fab4da838c2f:/# apt install scala直接安装 Scala
测试一下安装结果
MacOS 下可以直接按
Ctrl + D退出 Scala 的交互模式
root@fab4da838c2f:/# scala
Welcome to Scala version 2.11.6 (OpenJDK 64-Bit Server VM, Java 1.8.0_191).
Type in expressions to have them evaluated.
Type :help for more information.
scala>2.2 安装 Hadoop
- 在当前容器中将配置配好
- 导入出为镜像
- 以此镜像为基础创建五个容器，并赋予 hostname
- 进入 h01 容器，启动 Hadoop
2.2.1 安装 Vim 与 网络工具包
安装 vim，用来编辑文件
root@fab4da838c2f:/# apt install vim安装 net-tools
root@fab4da838c2f:/# apt install net-tools2.2.2 安装 SSH
安装 SSH，并配置免密登录，由于后面的容器之间是由一个镜像启动的，就像同一个磨具出来的 5 把锁与钥匙，可以互相开锁。所以在当前容器里配置 SSH 自身免密登录就 OK 了。
安装 SSH
root@fab4da838c2f:/# apt-get install openssh-server安装 SSH 的客户端
root@fab4da838c2f:/# apt-get install openssh-client进入当前用户的用户根目录
root@fab4da838c2f:/# cd ~
root@fab4da838c2f:~#生成密钥，不用输入，一直回车就行，生成的密钥在当前用户根目录下的 .ssh 文件夹中
以
.开头的文件与文件夹
ls是看不懂的，需要
ls -al才能查看。
root@fab4da838c2f:~# ssh-keygen -t rsa -P ""将公钥追加到 authorized_keys 文件中
root@fab4da838c2f:~# cat .ssh/id_rsa.pub >> .ssh/authorized_keys
root@fab4da838c2f:~#启动 SSH 服务
root@fab4da838c2f:~# service ssh start
* Starting OpenBSD Secure Shell server sshd [ OK ]
root@fab4da838c2f:~#免密登录自己
root@fab4da838c2f:~# ssh 127.0.0.1
Welcome to Ubuntu 16.04.6 LTS (GNU/Linux 4.15.0-45-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
Last login: Tue Mar 19 07:46:14 2019 from 127.0.0.1
root@fab4da838c2f:~#修改 .bashrc 文件，启动 shell 的时候，自动启动 SSH 服务
用 vim 打开 .bashrc 文件
root@fab4da838c2f:~# vim ~/.bashrc按一下 i 键，使得 vim 进入插入模式，此时终端的左下角会显示为 -- INSERT --，将光标移动到最后面，添加一行
service ssh start添加完的结果为，只显示最后几行
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
#if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
# . /etc/bash_completion
#fi
service ssh start按一下 Esc 键，使得 vim 退出插入模式
再输入英文模式下的冒号 :，此时终端的左下方会有一个冒号 : 显示出来
再输入三个字符 wq!，这是一个组合命令
w是保存的意思q是退出的意思!是强制的意思
再输入回车，退出 vim。
此时，SSH 免密登录已经完全配置好。
2.2.3 安装 Hadoop
下载 Hadoop 的安装文件
root@fab4da838c2f:~# wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz解压到 /usr/local 目录下面并重命名文件夹
root@fab4da838c2f:~# tar -zxvf hadoop-3.2.0.tar.gz -C /usr/local/
root@fab4da838c2f:~# cd /usr/local/
root@fab4da838c2f:/usr/local# mv hadoop-3.2.0 hadoop
root@fab4da838c2f:/usr/local#修改 /etc/profile 文件，添加一下环境变量到文件中
先用 vim 打开 /etc/profile
vim /etc/profile追加以下内容
JAVA_HOME 为 JDK 安装路径，使用 apt 安装就是这个，用
update-alternatives --config java可查看
#java
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
#hadoop
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_CONF_DIR=$HADOOP_HOME
export HADOOP_LIBEXEC_DIR=$HADOOP_HOME/libexec
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
export HDFS_DATANODE_USER=root
export HDFS_DATANODE_SECURE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_NAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root使环境变量生效
root@fab4da838c2f:/usr/local# source /etc/profile
root@fab4da838c2f:/usr/local#在目录 /usr/local/hadoop/etc/hadoop 下
修改 hadoop-env.sh 文件，在文件末尾添加一下信息
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root修改 core-site.xml，修改为
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://h01:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop3/hadoop/tmp</value>
</property>
</configuration>修改 hdfs-site.xml，修改为
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop3/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.namenode.data.dir</name>
<value>/home/hadoop3/hadoop/hdfs/data</value>
</property>
</configuration>修改 mapred-site.xml，修改为
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>
/usr/local/hadoop/etc/hadoop,
/usr/local/hadoop/share/hadoop/common/*,
/usr/local/hadoop/share/hadoop/common/lib/*,
/usr/local/hadoop/share/hadoop/hdfs/*,
/usr/local/hadoop/share/hadoop/hdfs/lib/*,
/usr/local/hadoop/share/hadoop/mapreduce/*,
/usr/local/hadoop/share/hadoop/mapreduce/lib/*,
/usr/local/hadoop/share/hadoop/yarn/*,
/usr/local/hadoop/share/hadoop/yarn/lib/*
</value>
</property>
</configuration>修改 yarn-site.xml，修改为
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>h01</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>修改 workser 为
h01
h02
h03
h04
h05此时，hadoop已经配置好了
2.2.4 在 Docker 中启动集群
先将当前容器导出为镜像，并查看当前镜像
dhu719@dhu719:~$ sudo docker commit -m "haddop" -a "hadoop" fab4da838c2f newuhadoop
sha256:648d8e082a231919faeaa14e09f5ce369b20879544576c03ef94074daf978823
dhu719@dhu719:~$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
newuhadoop latest 648d8e082a23 7 seconds ago 1.82GB
<none> <none> ccac37c7045c 4 days ago 1.85GB
ubuntu 16.04 9361ce633ff1 7 days ago 118MB
dhu719@dhu719:~$启动 5 个终端，分别执行这几个命令
第一条命令启动的是 h01 是做 master 节点的，所以暴露了端口，以供访问 web 页面
–network hadoop 参数是将当前容器加入到名为
hadoop的虚拟桥接网络中，此网站提供自动的 DNS 解析功能
dhu719@dhu719:~$ sudo docker run -it --network hadoop -h "h01" --name "h01" -p 9870:9870 -p 8088:8088 newuhadoop /bin/bash
* Starting OpenBSD Secure Shell server sshd [ OK ]
root@h01:/#其余的四条命令就是几乎一样的了
dhu719@dhu719:~$ sudo docker run -it --network hadoop -h "h02" --name "h02" newuhadoop /bin/bash
[sudo] password for dhu719:
* Starting OpenBSD Secure Shell server sshd [ OK ]
root@h02:/#
dhu719@dhu719:~$ sudo docker run -it --network hadoop -h "h03" --name "h03" newuhadoop /bin/bash
[sudo] password for dhu719:
* Starting OpenBSD Secure Shell server sshd [ OK ]
root@h03:/#
dhu719@dhu719:~$ sudo docker run -it --network hadoop -h "h04" --name "h04" newuhadoop /bin/bash
[sudo] password for dhu719:
* Starting OpenBSD Secure Shell server sshd [ OK ]
root@h04:/#
dhu719@dhu719:~$ sudo docker run -it --network hadoop -h "h05" --name "h05" newuhadoop /bin/bash
[sudo] password for dhu719:
* Starting OpenBSD Secure Shell server sshd [ OK ]
root@h05:/#接下来，在 h01 主机中，启动 Haddop 集群
先进行格式化操作
不格式化操作，hdfs会起不来
root@h01:/usr/local/hadoop/bin# ./hadoop namenode -format进入 hadoop 的 sbin 目录
root@h01:/# cd /usr/local/hadoop/sbin/
root@h01:/usr/local/hadoop/sbin#启动
root@h01:/usr/local/hadoop/sbin# ./start-all.sh
Starting namenodes on [h01]
h01: Warning: Permanently added 'h01,172.18.0.7' (ECDSA) to the list of known hosts.
Starting datanodes
h05: Warning: Permanently added 'h05,172.18.0.11' (ECDSA) to the list of known hosts.
h02: Warning: Permanently added 'h02,172.18.0.8' (ECDSA) to the list of known hosts.
h03: Warning: Permanently added 'h03,172.18.0.9' (ECDSA) to the list of known hosts.
h04: Warning: Permanently added 'h04,172.18.0.10' (ECDSA) to the list of known hosts.
h03: WARNING: /usr/local/hadoop/logs does not exist. Creating.
h05: WARNING: /usr/local/hadoop/logs does not exist. Creating.
h02: WARNING: /usr/local/hadoop/logs does not exist. Creating.
h04: WARNING: /usr/local/hadoop/logs does not exist. Creating.
Starting secondary namenodes [h01]
Starting resourcemanager
Starting nodemanagers
root@h01:/usr/local/hadoop/sbin#访问本机的 8088 与 9870 端口就可以看到监控信息了
使用命令 ./hadoop dfsadmin -report 可查看分布式文件系统的状态
root@h01:/usr/local/hadoop/bin# ./hadoop dfsadmin -report
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.
Configured Capacity: 5893065379840 (5.36 TB)
Present Capacity: 5237598752768 (4.76 TB)
DFS Remaining: 5237598629888 (4.76 TB)
DFS Used: 122880 (120 KB)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (5):
Name: 172.18.0.10:9866 (h03.hadoop)
Hostname: h03
Decommission Status : Normal
Configured Capacity: 1178613075968 (1.07 TB)
DFS Used: 24576 (24 KB)
Non DFS Used: 71199543296 (66.31 GB)
DFS Remaining: 1047519793152 (975.58 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.88%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Tue Mar 19 09:12:13 UTC 2019
Last Block Report: Tue Mar 19 09:10:46 UTC 2019
Num of Blocks: 0
Name: 172.18.0.11:9866 (h02.hadoop)
Hostname: h02
Decommission Status : Normal
Configured Capacity: 1178613075968 (1.07 TB)
DFS Used: 24576 (24 KB)
Non DFS Used: 71199625216 (66.31 GB)
DFS Remaining: 1047519711232 (975.58 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.88%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Tue Mar 19 09:12:13 UTC 2019
Last Block Report: Tue Mar 19 09:10:46 UTC 2019
Num of Blocks: 0
Name: 172.18.0.7:9866 (h01)
Hostname: h01
Decommission Status : Normal
Configured Capacity: 1178613075968 (1.07 TB)
DFS Used: 24576 (24 KB)
Non DFS Used: 71199633408 (66.31 GB)
DFS Remaining: 1047519703040 (975.58 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.88%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Tue Mar 19 09:12:13 UTC 2019
Last Block Report: Tue Mar 19 09:10:46 UTC 2019
Num of Blocks: 0
Name: 172.18.0.8:9866 (h05.hadoop)
Hostname: h05
Decommission Status : Normal
Configured Capacity: 1178613075968 (1.07 TB)
DFS Used: 24576 (24 KB)
Non DFS Used: 71199625216 (66.31 GB)
DFS Remaining: 1047519711232 (975.58 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.88%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Tue Mar 19 09:12:13 UTC 2019
Last Block Report: Tue Mar 19 09:10:46 UTC 2019
Num of Blocks: 0
Name: 172.18.0.9:9866 (h04.hadoop)
Hostname: h04
Decommission Status : Normal
Configured Capacity: 1178613075968 (1.07 TB)
DFS Used: 24576 (24 KB)
Non DFS Used: 71199625216 (66.31 GB)
DFS Remaining: 1047519711232 (975.58 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.88%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Tue Mar 19 09:12:13 UTC 2019
Last Block Report: Tue Mar 19 09:10:46 UTC 2019
Num of Blocks: 0
root@h01:/usr/local/hadoop/bin#Hadoop 集群已经构建好了
2.2.5 运行内置WordCount例子
把license作为需要统计的文件
root@h01:/usr/local/hadoop# cat LICENSE.txt > file1.txt
root@h01:/usr/local/hadoop# ls在 HDFS 中创建 input 文件夹
root@h01:/usr/local/hadoop/bin# ./hadoop fs -mkdir /input
root@h01:/usr/local/hadoop/bin#上传 file1.txt 文件到 HDFS 中
root@h01:/usr/local/hadoop/bin# ./hadoop fs -put ../file1.txt /input
root@h01:/usr/local/hadoop/bin#查看 HDFS 中 input 文件夹里的内容
root@h01:/usr/local/hadoop/bin# ./hadoop fs -ls /input
Found 1 items
-rw-r--r-- 2 root supergroup 150569 2019-03-19 11:13 /input/file1.txt
root@h01:/usr/local/hadoop/bin#运作 wordcount 例子程序
root@h01:/usr/local/hadoop/bin# ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount /input /output输出如下
2019-03-19 11:18:23,953 INFO client.RMProxy: Connecting to ResourceManager at h01/172.18.0.7:8032
2019-03-19 11:18:24,381 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1552986653954_0001
2019-03-19 11:18:24,659 INFO input.FileInputFormat: Total input files to process : 1
2019-03-19 11:18:25,095 INFO mapreduce.JobSubmitter: number of splits:1
2019-03-19 11:18:25,129 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2019-03-19 11:18:25,208 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1552986653954_0001
2019-03-19 11:18:25,210 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-03-19 11:18:25,368 INFO conf.Configuration: resource-types.xml not found
2019-03-19 11:18:25,368 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2019-03-19 11:18:25,797 INFO impl.YarnClientImpl: Submitted application application_1552986653954_0001
2019-03-19 11:18:25,836 INFO mapreduce.Job: The url to track the job: http://h01:8088/proxy/application_1552986653954_0001/
2019-03-19 11:18:25,837 INFO mapreduce.Job: Running job: job_1552986653954_0001
2019-03-19 11:18:33,990 INFO mapreduce.Job: Job job_1552986653954_0001 running in uber mode : false
2019-03-19 11:18:33,991 INFO mapreduce.Job: map 0% reduce 0%
2019-03-19 11:18:39,067 INFO mapreduce.Job: map 100% reduce 0%
2019-03-19 11:18:45,106 INFO mapreduce.Job: map 100% reduce 100%
2019-03-19 11:18:46,124 INFO mapreduce.Job: Job job_1552986653954_0001 completed successfully
2019-03-19 11:18:46,227 INFO mapreduce.Job: Counters: 54
File System Counters
FILE: Number of bytes read=46852
FILE: Number of bytes written=537641
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=150665
HDFS: Number of bytes written=35324
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3129
Total time spent by all reduces in occupied slots (ms)=3171
Total time spent by all map tasks (ms)=3129
Total time spent by all reduce tasks (ms)=3171
Total vcore-milliseconds taken by all map tasks=3129
Total vcore-milliseconds taken by all reduce tasks=3171
Total megabyte-milliseconds taken by all map tasks=3204096
Total megabyte-milliseconds taken by all reduce tasks=3247104
Map-Reduce Framework
Map input records=2814
Map output records=21904
Map output bytes=234035
Map output materialized bytes=46852
Input split bytes=96
Combine input records=21904
Combine output records=2981
Reduce input groups=2981
Reduce shuffle bytes=46852
Reduce input records=2981
Reduce output records=2981
Spilled Records=5962
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=111
CPU time spent (ms)=2340
Physical memory (bytes) snapshot=651853824
Virtual memory (bytes) snapshot=5483622400
Total committed heap usage (bytes)=1197998080
Peak Map Physical memory (bytes)=340348928
Peak Map Virtual memory (bytes)=2737307648
Peak Reduce Physical memory (bytes)=311504896
Peak Reduce Virtual memory (bytes)=2746314752
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=150569
File Output Format Counters
Bytes Written=35324查看 HDFS 中的 /output 文件夹的内容
root@h01:/usr/local/hadoop/bin# ./hadoop fs -ls /output
Found 2 items
-rw-r--r-- 2 root supergroup 0 2019-03-19 11:18 /output/_SUCCESS
-rw-r--r-- 2 root supergroup 35324 2019-03-19 11:18 /output/part-r-00000查看 part-r-00000 文件的内容
root@h01:/usr/local/hadoop/bin# ./hadoop fs -cat /output/part-r-00000Hadoop 部分结束了
2.3 安装 Hbase
在 Hadoop 集群的基础上安装 Hbase
下载 Hbase 2.1.3
root@h01:~# wget https://www-eu.apache.org/dist/hbase/2.1.3/hbase-2.1.3-bin.tar.gz解压到 /usr/local 目录下面
root@h01:~# tar -zxvf hbase-2.1.3-bin.tar.gz -C /usr/local/修改 /etc/profile 环境变量文件，添加 Hbase 的环境变量，追加下述代码
export HBASE_HOME=/usr/local/hbase-2.1.3
export PATH=$PATH:$HBASE_HOME/bin使环境变量配置文件生效
root@h01:/usr/local# source /etc/profile
root@h01:/usr/local#使用 ssh h02 可进入其他四个容器，依次修改。
即是每个容器都要在 /etc/profile 文件后追加那两行环境变量
在目录 /usr/local/hbase-2.1.3/conf 修改配置
修改 hbase-env.sh，追加
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HBASE_MANAGES_ZK=true修改 hbase-site.xml 为
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://h01:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.master</name>
<value>h01:60000</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>h01,h02,h03,h04,h05</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/hadoop/zoodata</value>
</property>
</configuration>修改 regionservers 文件为
h01
h02
h03
h04
h05使用 scp 命令将配置好的 Hbase 复制到其他 4 个容器中
root@h01:~# scp -r /usr/local/hbase-2.1.3 root@h02:/usr/local/
root@h01:~# scp -r /usr/local/hbase-2.1.3 root@h03:/usr/local/
root@h01:~# scp -r /usr/local/hbase-2.1.3 root@h04:/usr/local/
root@h01:~# scp -r /usr/local/hbase-2.1.3 root@h05:/usr/local/启动 Hbase
root@h01:/usr/local/hbase-2.1.3/bin# ./start-hbase.sh
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hbase-2.1.3/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hbase-2.1.3/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
h04: running zookeeper, logging to /usr/local/hbase-2.1.3/bin/../logs/hbase-root-zookeeper-h04.out
h01: running zookeeper, logging to /usr/local/hbase-2.1.3/bin/../logs/hbase-root-zookeeper-h01.out
h05: running zookeeper, logging to /usr/local/hbase-2.1.3/bin/../logs/hbase-root-zookeeper-h05.out
h03: running zookeeper, logging to /usr/local/hbase-2.1.3/bin/../logs/hbase-root-zookeeper-h03.out
h02: running zookeeper, logging to /usr/local/hbase-2.1.3/bin/../logs/hbase-root-zookeeper-h02.out
running master, logging to /usr/local/hbase-2.1.3/logs/hbase--master-h01.out
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hbase-2.1.3/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
h03: running regionserver, logging to /usr/local/hbase-2.1.3/bin/../logs/hbase-root-regionserver-h03.out
h01: running regionserver, logging to /usr/local/hbase-2.1.3/bin/../logs/hbase-root-regionserver-h01.out
h04: running regionserver, logging to /usr/local/hbase-2.1.3/bin/../logs/hbase-root-regionserver-h04.out
h05: running regionserver, logging to /usr/local/hbase-2.1.3/bin/../logs/hbase-root-regionserver-h05.out
h02: running regionserver, logging to /usr/local/hbase-2.1.3/bin/../logs/hbase-root-regionserver-h02.out打开 Hbase 的 shell
root@h01:/usr/local/hbase-2.1.3/bin# hbase shell
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hbase-2.1.3/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.1.3, rda5ec9e4c06c537213883cca8f3cc9a7c19daf67, Mon Feb 11 15:45:33 CST 2019
Took 0.0033 seconds
hbase(main):001:0>
hbase(main):002:0> whoami
root (auth:SIMPLE)
groups: root
Took 0.0134 seconds2.4 安装 Spark
在 Hadoop 的基础上安装 Spark
下载 Spark 2.4.0
root@h01:~# http://mirrors.shu.edu.cn/apache/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz解压到 /usr/local 目录下面
root@h01:~# tar -zxvf spark-2.4.0-bin-hadoop2.7.tgz -C /usr/local/修改文件夹的名字
root@h01:~# cd /usr/local/
root@h01:/usr/local# mv spark-2.4.0-bin-hadoop2.7 spark-2.4.0修改 /etc/profile 环境变量文件，添加 Hbase 的环境变量，追加下述代码
export SPARK_HOME=/usr/local/spark-2.4.0
export PATH=$PATH:$SPARK_HOME/bin使环境变量配置文件生效
root@h01:/usr/local# source /etc/profile
root@h01:/usr/local#使用 ssh h02 可进入其他四个容器，依次修改。
即是每个容器都要在 /etc/profile 文件后追加那两行环境变量
在目录 /usr/local/spark-2.4.0/conf 修改配置
修改文件名
root@h01:/usr/local/spark-2.4.0/conf# mv spark-env.sh.template spark-env.sh
root@h01:/usr/local/spark-2.4.0/conf#修改 spark-env.sh，追加
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export SCALA_HOME=/usr/share/scala
export SPARK_MASTER_HOST=h01
export SPARK_MASTER_IP=h01
export SPARK_WORKER_MEMORY=4g修改文件名
root@h01:/usr/local/spark-2.4.0/conf# mv slaves.template slaves
root@h01:/usr/local/spark-2.4.0/conf#修改 slaves 如下
h01
h02
h03
h04
h05使用 scp 命令将配置好的 Hbase 复制到其他 4 个容器中
root@h01:/usr/local# scp -r /usr/local/spark-2.4.0 root@h02:/usr/local/
root@h01:/usr/local# scp -r /usr/local/spark-2.4.0 root@h03:/usr/local/
root@h01:/usr/local# scp -r /usr/local/spark-2.4.0 root@h04:/usr/local/
root@h01:/usr/local# scp -r /usr/local/spark-2.4.0 root@h05:/usr/local/启动 Spark
root@h01:/usr/local/spark-2.4.0/sbin# ./start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-2.4.0/logs/spark--org.apache.spark.deploy.master.Master-1-h01.out
h04: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.4.0/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-h04.out
h05: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.4.0/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-h05.out
h01: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.4.0/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-h01.out
h03: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.4.0/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-h03.out
h02: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.4.0/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-h02.out
root@h01:/usr/local/spark-2.4.0/sbin#3 其他
3.1 HDFS 重格式化问题
参考
https://blog.csdn.net/gis_101/article/details/52821946
- 重新格式化意味着集群的数据会被全部删除，格式化前需考虑数据备份或转移问题；
- 先删除主节点（即namenode节点），Hadoop的临时存储目录tmp、namenode存储永久性元数据目录dfs/name、Hadoop系统日志文件目录log 中的内容 （注意是删除目录下的内容不是目录）；
- 删除所有数据节点(即datanode节点) ，Hadoop的临时存储目录tmp、namenode存储永久性元数据目录dfs/name、Hadoop系统日志文件目录log 中的内容；
- 格式化一个新的分布式文件系统：
root@h01:/usr/local/hadoop/bin# ./hadoop namenode -format注意事项
- Hadoop的临时存储目录tmp（即core-site.xml配置文件中的hadoop.tmp.dir属性，默认值是/tmp/hadoop-${user.name}），如果没有配置hadoop.tmp.dir属性，那么hadoop格式化时将会在/tmp目录下创建一个目录，例如在cloud用户下安装配置hadoop，那么Hadoop的临时存储目录就位于/tmp/hadoop-cloud目录下
- Hadoop的namenode元数据目录（即hdfs-site.xml配置文件中的dfs.namenode.name.dir属性，默认值是${hadoop.tmp.dir}/dfs/name），同样如果没有配置该属性，那么hadoop在格式化时将自行创建。必须注意的是在格式化前必须清楚所有子节点（即DataNode节点）dfs/name下的内容，否则在启动hadoop时子节点的守护进程会启动失败。这是由于，每一次format主节点namenode，dfs/name/current目录下的VERSION文件会产生新的clusterID、namespaceID。但是如果子节点的dfs/name/current仍存在，hadoop格式化时就不会重建该目录，因此形成子节点的clusterID、namespaceID与主节点（即namenode节点）的clusterID、namespaceID不一致。最终导致hadoop启动失败。
