爱尚实训 2018-01-25 10:03
目前在练习hadoop的时候发现每次都要在准备基本环境的时候浪费很多时间，俗话说：“工欲善其事,必先利其器”，干脆横下一条心来，花了两周的时间，整理好了各种集群模板，需要哪项技术就直接克隆模板就好，目前主备的模板大概4套：
1、最基本的集群（4台主机）
2、基本集群+hive（5台主机）
3、 hadoop集群的的HA模板（7台主机）
4、hadoop集群的HA+Hbase模板（7台主机）
如下图所示：
大数据确实很锻炼人，要求不光有编写代码的能力、理解底层的能力、理解业务的能力，还要具有运维的能力，否则没有“实操”，那些高大尚的能力根本没机会得到锻炼，很多初学者都会被大数据的环境所难倒，但是不过这一关，是学不好大数据的，在这上面耽误点时间也是值得的，话不多说，将自己整理的hadoop的基础环境安装文档分享一下。
一、主机规划和环境：
机器名 域名IP作用
hdp-01192.168.33.31NameNode、SecondaryNameNode、ResourceManager
hdp-02192.168.33.32DataNode、NodeManager
hdp-03192.168.33.33DataNode、NodeManager
hdp-04192.168.33.34DataNode、NodeManager
机器名
域名
IP
作用
hdp-01
192.168.33.31
NameNode、SecondaryNameNode、ResourceManager
hdp-02
192.168.33.32
DataNode、NodeManager
hdp-03
192.168.33.33
DataNode、NodeManager
hdp-04
192.168.33.34
DataNode、NodeManager
OS: linux-CentOS6.7 64位
hadoop: hadoop-2.6.4
二、每台机器准备工作
1. 通过root登录
创建hadoop用户
Useradd hadoo
Passwd hadoo
2. 配置好每一台服务器的主机名、网络IP、域名列表
修改主机名称：vi /etc/sysconfig/network
修改IP：vi /etc/sysconfig/network-scripts/ifcfg-eth0
修改主机名和IP的映射关系：vi /etc/host
如：192.168.33.31 hdp-01
3. 配置好yum源 每一台机器需要安装scp（yum install -y openssh-clients）、jdk
4. 关闭防火墙
#查看防火墙状态：service iptables statu
#关闭防火墙开机启动：chkconfig iptables off
5. 配置hadoop用户的sudo权限vi /etc/sudoer
6. 使用刚创建的hadoop用户登录
三、安装Hadoop集群
1. 先上传hadoop的安装包到服务器上去/home/hadoop/
Alt+p：开启一个上传的bash
ftp> lcd D:/大数据资料/5.hadoop/ sftp> put cenos-6.5-hadoop-2.6.4.tar.gz
ftp> lcd D:/大数据资料/5.hadoop/
ftp> put cenos-6.5-hadoop-2.6.4.tar.gz
在另一个bash中，运行ll，看到上传成功
[hadoop@hdp-01 ~]$ ll 总用量 176580 -rw-rw-r–. 1 hadoop hadoop 180813065 9月 22 09:23 cenos-6.5-hadoop-2.6.4.tar.gz
[hadoop@hdp-01 ~]$ ll
总用量 176580
-rw-rw-r–. 1 hadoop hadoop 180813065 9月 22 09:23 cenos-6.5-hadoop-2.6.4.tar.gz
2. 创建在/home/hadoop中创建目录apps/，以后应用都放到这里
[hadoop@hdp-01 ~]$ mkdir apps/
[hadoop@hdp-01 ~]$ mkdir apps/
3. 解压hadoop到apps/中
[hadoop@hdp-01 ~]$ tar -zxvf cenos-6.5-hadoop-2.6.4.tar.gz -C apps/
[hadoop@hdp-01 ~]$ tar -zxvf cenos-6.5-hadoop-2.6.4.tar.gz -C apps/
4. 进到hadoop的安装目录，找到etc目录
[hadoop@hdp-01 ~]$ cd /home/hadoop/apps/hadoop-2.6.4 [hadoop@hdp-01 hadoop-2.6.4]$ ll 总用量 52 drwxrwxr-x. 2 hadoop hadoop 4096 3月 8 2016 bin drwxrwxr-x. 3 hadoop hadoop 4096 3月 8 2016 etc drwxrwxr-x. 2 hadoop hadoop 4096 3月 8 2016 include drwxrwxr-x. 3 hadoop hadoop 4096 3月 8 2016 lib drwxrwxr-x. 2 hadoop hadoop 4096 3月 8 2016 libexec -rw-r–r–. 1 hadoop hadoop 15429 3月 8 2016 LICENSE.txt -rw-r–r–. 1 hadoop hadoop 101 3月 8 2016 NOTICE.txt -rw-r–r–. 1 hadoop hadoop 1366 3月 8 2016 README.txt drwxrwxr-x. 2 hadoop hadoop 4096 3月 8 2016 sbin drwxrwxr-x. 4 hadoop hadoop 4096 3月 8 2016 share -rw-r–r–. 1 hadoop hadoop 4436 3月 8 2016 capacity-scheduler.xml -rw-r–r–. 1 hadoop hadoop 1335 3月 8 2016 configuration.xsl -rw-r–r–. 1 hadoop hadoop 318 3月 8 2016 container-executor.cfg -rw-r–r–. 1 hadoop hadoop 774 3月 8 2016 core-site.xml -rw-r–r–. 1 hadoop hadoop 3670 3月 8 2016 hadoop-env.cmd -rw-r–r–. 1 hadoop hadoop 4224 3月 8 2016 hadoop-env.sh -rw-r–r–. 1 hadoop hadoop 2598 3月 8 2016 hadoop-metrics2.properties -rw-r–r–. 1 hadoop hadoop 2490 3月 8 2016 hadoop-metrics.properties -rw-r–r–. 1 hadoop hadoop 9683 3月 8 2016 hadoop-policy.xml -rw-r–r–. 1 hadoop hadoop 775 3月 8 2016 hdfs-site.xml -rw-r–r–. 1 hadoop hadoop 1449 3月 8 2016 httpfs-env.sh -rw-r–r–. 1 hadoop hadoop 1657 3月 8 2016 httpfs-log4j.properties -rw-r–r–. 1 hadoop hadoop 21 3月 8 2016 httpfs-signature.secret -rw-r–r–. 1 hadoop hadoop 620 3月 8 2016 httpfs-site.xml -rw-r–r–. 1 hadoop hadoop 3523 3月 8 2016 kms-acls.xml -rw-r–r–. 1 hadoop hadoop 1325 3月 8 2016 kms-env.sh -rw-r–r–. 1 hadoop hadoop 1631 3月 8 2016 kms-log4j.properties -rw-r–r–. 1 hadoop hadoop 5511 3月 8 2016 kms-site.xml -rw-r–r–. 1 hadoop hadoop 11291 3月 8 2016 log4j.properties -rw-r–r–. 1 hadoop hadoop 938 3月 8 2016 mapred-env.cmd -rw-r–r–. 1 hadoop hadoop 1383 3月 8 2016 mapred-env.sh -rw-r–r–. 1 hadoop hadoop 4113 3月 8 2016 mapred-queues.xml.template -rw-r–r–. 1 hadoop hadoop 758 3月 8 2016 mapred-site.xml.template -rw-r–r–. 1 hadoop hadoop 10 3月 8 2016 slaves -rw-r–r–. 1 hadoop hadoop 2316 3月 8 2016 ssl-client.xml.example -rw-r–r–. 1 hadoop hadoop 2268 3月 8 2016 ssl-server.xml.example -rw-r–r–. 1 hadoop hadoop 2237 3月 8 2016 yarn-env.cmd -rw-r–r–. 1 hadoop hadoop 4567 3月 8 2016 yarn-env.sh -rw-r–r–. 1 hadoop hadoop 690 3月 8 2016 yarn-site.xml
[hadoop@hdp-01 ~]$ cd /home/hadoop/apps/hadoop-2.6.4
[hadoop@hdp-01 hadoop-2.6.4]$ ll
总用量 52
drwxrwxr-x. 2 hadoop hadoop 4096 3月 8 2016 bi
drwxrwxr-x. 2 hadoop hadoop 4096 3月 8 2016 include
drwxrwxr-x. 3 hadoop hadoop 4096 3月 8 2016 li
drwxrwxr-x. 2 hadoop hadoop 4096 3月 8 2016 libexec
-rw-r–r–. 1 hadoop hadoop 15429 3月 8 2016 LICENSE.txt
-rw-r–r–. 1 hadoop hadoop 101 3月 8 2016 NOTICE.txt
-rw-r–r–. 1 hadoop hadoop 1366 3月 8 2016 README.txt
drwxrwxr-x. 2 hadoop hadoop 4096 3月 8 2016 sbi
drwxrwxr-x. 4 hadoop hadoop 4096 3月 8 2016 share
-rw-r–r–. 1 hadoop hadoop 4436 3月 8 2016 capacity-scheduler.xml
-rw-r–r–. 1 hadoop hadoop 1335 3月 8 2016 configuration.xsl
-rw-r–r–. 1 hadoop hadoop 318 3月 8 2016 container-executor.cfg
-rw-r–r–. 1 hadoop hadoop 3670 3月 8 2016 hadoop-env.cmd
-rw-r–r–. 1 hadoop hadoop 2598 3月 8 2016 hadoop-metrics2.propertie
-rw-r–r–. 1 hadoop hadoop 2490 3月 8 2016 hadoop-metrics.propertie
-rw-r–r–. 1 hadoop hadoop 9683 3月 8 2016 hadoop-policy.xml
-rw-r–r–. 1 hadoop hadoop 1449 3月 8 2016 httpfs-env.sh
-rw-r–r–. 1 hadoop hadoop 1657 3月 8 2016 httpfs-log4j.propertie
-rw-r–r–. 1 hadoop hadoop 21 3月 8 2016 httpfs-signature.secret
-rw-r–r–. 1 hadoop hadoop 620 3月 8 2016 httpfs-site.xml
-rw-r–r–. 1 hadoop hadoop 3523 3月 8 2016 kms-acls.xml
-rw-r–r–. 1 hadoop hadoop 1325 3月 8 2016 kms-env.sh
-rw-r–r–. 1 hadoop hadoop 1631 3月 8 2016 kms-log4j.propertie
-rw-r–r–. 1 hadoop hadoop 5511 3月 8 2016 kms-site.xml
-rw-r–r–. 1 hadoop hadoop 11291 3月 8 2016 log4j.propertie
-rw-r–r–. 1 hadoop hadoop 938 3月 8 2016 mapred-env.cmd
-rw-r–r–. 1 hadoop hadoop 1383 3月 8 2016 mapred-env.sh
-rw-r–r–. 1 hadoop hadoop 4113 3月 8 2016 mapred-queues.xml.template
-rw-r–r–. 1 hadoop hadoop 10 3月 8 2016 slave
-rw-r–r–. 1 hadoop hadoop 2316 3月 8 2016 ssl-client.xml.example
-rw-r–r–. 1 hadoop hadoop 2268 3月 8 2016 ssl-server.xml.example
-rw-r–r–. 1 hadoop hadoop 2237 3月 8 2016 yarn-env.cmd
-rw-r–r–. 1 hadoop hadoop 4567 3月 8 2016 yarn-env.sh
5. 进入etc/hadoop目录，修改5个配置文件
v 第一个：修改hadoop-env.sh
先查看JAVA_HOME的路径
[hadoop@hdp-01 hadoop]$ echo $JAVA_HOME /usr/local/jdk1.7.0_45
[hadoop@hdp-01 hadoop]$ echo $JAVA_HOME
/usr/local/jdk1.7.0_45
修改hadoop-env.sh中的部分内容
[hadoop@hdp-01 hadoop]$ vi hadoop-env.sh
[hadoop@hdp-01 hadoop]$ vi hadoop-env.sh
将 export JAVA_HOME=${JAVA_HOME}
修改成真正的JAVA_HOME地址 ：export JAVA_HOME=/usr/local/jdk1.7.0_45
第二个：修改core-site.xml
fs.defaultFS hdfs://hdp-01:9000 hadoop.tmp.dir /home/hadoop/hdpdata
fs.defaultFS
hdfs://hdp-01:9000
hadoop.tmp.dir
/home/hadoop/hdpdata
第三个：修改hdfs-site.xml
dfs.replication 2
dfs.replication
2
第四个：修改mapred-site.xml (将文件改名：mv mapred-site.xml.template mapred-site.xml)
mv mapred-site.xml.template mapred-site.xml
mv mapred-site.xml.template mapred-site.xml
mapreduce.framework.name yarn
mapreduce.framework.name
yarn
第五个：修改yarn-site.xml
yarn.resourcemanager.hostname hdp-01 yarn.nodemanager.aux-services mapreduce_shuffle
yarn.resourcemanager.hostname
hdp-01
yarn.nodemanager.aux-services
mapreduce_shuffle
6. 将hdp-01的apps/下的hadoop分发到其它机器上去（现在是hadoop登录，切换到root用户拷贝）
[hadoop@hdp-01 ~] sudo scp -r apps/ hdp-02:/home/hadoop [hadoop@hdp-01 ~] sudo scp -r apps/ hdp-03:/home/hadoop [hadoop@hdp-01 ~] sudo scp -r apps/ hdp-04:/home/hadoo
[hadoop@hdp-01 ~] sudo scp -r apps/ hdp-02:/home/hadoo
[hadoop@hdp-01 ~] sudo scp -r apps/ hdp-03:/home/hadoo
[hadoop@hdp-01 ~] sudo scp -r apps/ hdp-04:/home/hadoo
7. 将HADOOP_HOME配置到/etc/profile下
vi /etc/profile
vi /etc/profile
export JAVA_HOME=/usr/local/jdk1.7.0_45 export HADOOP_HOME=/home/hadoop/apps/hadoop-2.6.4 export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbi
export JAVA_HOME=/usr/local/jdk1.7.0_45
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.6.4
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbi
8. 将/etc/profile拷贝到其它的机器上（现在是hadoop登录，切换到root用户拷贝）
[hadoop@hdp-01 ~] sudo scp /etc/profile hdp-02:/etc/ [hadoop@hdp-01 ~] sudo scp /etc/profile hdp-03:/etc/ [hadoop@hdp-01 ~] sudo scp /etc/profile hdp-04:/etc/
[hadoop@hdp-01 ~] sudo scp /etc/profile hdp-02:/etc/
[hadoop@hdp-01 ~] sudo scp /etc/profile hdp-03:/etc/
[hadoop@hdp-01 ~] sudo scp /etc/profile hdp-04:/etc/
9. 格式化namenode（是对namenode进行初始化）
[hadoop@hdp-01 ~] hdfs namenode -format 或者[hadoop@hdp-01 ~] hadoop namenode -format 18/01/05 21:42:33 INFO common.Storage:Storage directory /home/hadoop/hdpdata/dfs/name has been successfully formatted. 18/01/05 21:42:33 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
[hadoop@hdp-01 ~] hdfs namenode -format 或者[hadoop@hdp-01 ~] hadoop namenode -format
18/01/05 21:42:33 INFO common.Storage:Storage directory /home/hadoop/hdpdata/dfs/name has been successfully formatted.
18/01/05 21:42:33 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
10. 配置免密登录
[hadoop@hdp-01 ~]$ ssh-keygen [hadoop@hdp-01 ~]$ ssh-copy-id hdp-01 [hadoop@hdp-01 ~]$ ssh-copy-id hdp-02 [hadoop@hdp-01 ~]$ ssh-copy-id hdp-03 [hadoop@hdp-01 ~]$ ssh-copy-id hdp-04
[hadoop@hdp-01 ~]$ ssh-keyge
[hadoop@hdp-01 ~]$ ssh-copy-id hdp-01
[hadoop@hdp-01 ~]$ ssh-copy-id hdp-02
[hadoop@hdp-01 ~]$ ssh-copy-id hdp-03
[hadoop@hdp-01 ~]$ ssh-copy-id hdp-04
11. 找到hadoop安装目录下的/etc/hadoop/slaves，将要启动的datenode、nodeManager加入到这个文件中
[hadoop@hdp-01 hadoop]$ vi slaves hdp-02 hdp-03 hdp-04
[hadoop@hdp-01 hadoop]$ vi slave
hdp-02
hdp-03
hdp-04
现在就可以启动hadoop了。以上的文档，都在本人的机器上调试通过，所整理的文档也是将输入命令记录下来。下一篇将介绍hadoop的集群之上的hive的安装
