本文章仅作为个人笔记

CDH官网地址：https://www.cloudera.com/

CDH镜像地址：

    cm5.11.1地址：http://archive.cloudera.com/cm5/repo-as-tarball/5.11.1/cm5.11.1-centos7.tar.gz

        http://archive.cloudera.com/cm5/repo-as-tarball/5.11.1/cm5.11.1-centos7.tar.gz.sha1

    cdh5.11.1地址：http://archive.cloudera.com/cdh5/parcels/5.11.1/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel

        http://archive.cloudera.com/cdh5/parcels/5.11.1/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha1

        http://archive.cloudera.com/cdh5/parcels/5.11.1/manifest.json

    jdk1.8.121官网下载地址：http://download.oracle.com/otn/java/jdk/8u121-b13/e9e7ea248e2c4826b92b3f075a80e441/jdk-8u121-linux-i586.rpm

    mysql5.6.36官网下载地址：https://dev.mysql.com/get/Downloads/MySQL-5.6/mysql-5.6.36-linux-glibc2.5-x86_64.tar.gz

    mysql连接驱动下载地址：https://dev.mysql.com/downloads/connector/j/

笔者使用VMware进行模拟安装，为了满足基础配置，这里分配3台虚拟机（hadoopb01(4g/4核/40g/192.168.75.136/master),hadoopb02(2g/2核/40g/192.168.75.144/slave),hadoopb03(2g/2核/40g/192.168.75.145/slave)），另外wget下载文件非常缓慢并且不方便反复测试及安装，笔者这里下载好所有资源文件放在本地（.sha1文件结尾的文件统一重命名为.sha）。（如果虚拟机开机后无网络（使用net桥接方式），可与命令行运行dhclient分配ip，若后续断开了可运行dhclient -r后再运行dhclient）

本地虚拟机，本地文件

在3台服务器共同操作：

    安装基本工具： yum install vim wget net-tools unzip -y

    关闭SELINUX：

        运行 setenforce 0 临时关闭

        永久关闭SELINUX：

            echo “SELINUX=disabled” > /etc/selinux/config

            echo “SELINUXTYPE=targeted” >> /etc/selinux/config

    关闭iptables防火墙：

        运行 systemctl stop firewalld.service 停止防火墙

        运行 systemctl status firewalld.service 查看防火墙状态

        运行 systemctl disable firewalld.service 关闭防火墙开机启动

    更改主机名：

        运行 hostname hadoopb01 更改主机名，根据不同主机选择不同主机名（hadoopb01/hadoopb02/hadoopb03）

        运行 echo “HOSTNAME=hadoopb01” > /etc/sysconfig/network ，根据不同主机选择不同主机名（hadoopb01/hadoopb02/hadoopb03）

        运行 echo “hadoopb01” > /etc/hostname ，根据不同主机选择不同主机名（hadoopb01/hadoopb02/hadoopb03）

    安装jdk（参考http://www.jianshu.com/p/c5fc4dd47754）

    密钥生成：

        运行 ssh-keygen 一路回车回车生成

    ip与hostname绑定：

        echo “192.168.75.136 hadoopb01” >> /etc/hosts

        echo “192.168.75.144 hadoopb02” >> /etc/hosts

        echo “192.168.75.145 hadoopb03” >> /etc/hosts

    运行python –version 检查python版本，如果不是python2.6.6建议改为python2.6.6

    时间同步：

        运行 yum install ntp -y 安装ntp服务

        运行 systemctl enable ntpd 使ntp服务开机启动

在从服务器操作（hadoopb02(2g/2核/40g/192.168.75.144/slave),hadoopb03(2g/2核/40g/192.168.75.145/slave）：

    配置时钟同步：

        echo “driftfile /var/lib/ntp/ntp.drift” > /etc/ntp.conf

        echo “statistics loopstats peerstats clockstats” >> /etc/ntp.conf

        echo “filegen loopstats file loopstats type day enable” >> /etc/ntp.conf

        echo “filegen peerstats file peerstats type day enable” >> /etc/ntp.conf

        echo “filegen clockstats file clockstats type day enable” >> /etc/ntp.conf

        echo “server 192.168.75.136 prefer” >> /etc/ntp.conf

        echo “fudge 192.168.75.136 stratum 5” >> /etc/ntp.conf

        echo “restrict -4 default kod notrap nomodify nopeer noquery” >> /etc/ntp.conf

        echo “restrict -6 default kod notrap nomodify nopeer noquery” >> /etc/ntp.conf

        echo “restrict 127.0.0.1” >> /etc/ntp.conf

        echo “restrict ::1” >> /etc/ntp.conf

    运行 service ntpd restart 重启ntpd并使配置生效

    运行 ntpdate –u 192.168.75.136 同步时间

在主服务器操作（hadoopb01(4g/4核/40g/192.168.75.136/master）：

    运行 ssh-copy-id hadoopb01

        ssh-copy-id hadoopb02

        ssh-copy-id hadoopb03 完成各服务器免密登录

    配置时钟同步：

        echo “driftfile /var/lib/ntp/ntp.drift” > /etc/ntp.conf

        echo “restrict 192.168.75.0 mask 255.255.255.0 nomodify notrap” >> /etc/ntp.conf

        echo “server 210.72.145.44” >> /etc/ntp.conf

        echo “server 202.112.10.36 perfer” >> /etc/ntp.conf

        echo “server 59.124.196.83” >> /etc/ntp.conf

        echo “restrict 210.72.145.44 nomodify notrap noquery” >> /etc/ntp.conf

        echo “restrict 202.112.10.36 nomodify notrap noquery” >> /etc/ntp.conf

        echo “restrict 59.124.196.83 nomodify notrap noquery” >> /etc/ntp.conf

        echo “server  127.127.1.0” >> /etc/ntp.conf

        echo “fudge  127.127.1.0 stratum 10” >> /etc/ntp.conf

    运行 service ntpd restart 重启ntpd并使配置生效

    运行 ntpdate –u 202.112.10.36 同步时间

    使用filezilla将上文所述文件传输至主服务器（放置于root用户~目录）

    安装配置mysql5.6（可参考http://www.jianshu.com/p/c5fc4dd47754）

        创建相关用户及数据库：

      create database cmf default character set utf8

        grant all on cmf.* to ‘cmf’@’localhost’ identified by ‘cmf_password’

        grant all on cmf.* to ‘cmf’@’%’ identified by ‘cmf_password’

        grant all on cmf.* to ‘cmf’@’${m_hosts[${c_index}]}’ identified by ‘cmf_password’

        drop database cmf

        create database cmf

        create database hive default character set utf8

        grant all on hive.* to ‘hive’@’localhost’ identified by ‘hive_password’

        grant all on hive.* to ‘hive’@’%’ identified by ‘hive_password’

        grant all on hive.* to ‘hive’@’${m_hosts[${c_index}]}’ identified by ‘hive_password’

        drop database hive

        create database hive

        create database hue default character set utf8

        grant all on hue.* to ‘hue’@’localhost’ identified by ‘hue_password’

        grant all on hue.* to ‘hue’@’%’ identified by ‘hue_password’

        grant all on hue.* to ‘hue’@’${m_hosts[${c_index}]}’ identified by ‘hue_password’

        drop database hue

        create database hue

        create database am default character set utf8

        grant all on am.* to ‘am’@’localhost’ identified by ‘am_password’

        grant all on am.* to ‘am’@’%’ identified by ‘am_password’

        grant all on am.* to ‘am’@’${m_hosts[${c_index}]}’ identified by ‘am_password’

        drop database am

        create database am

        create database os default character set utf8

        grant all on os.* to ‘os’@’localhost’ identified by ‘os_password’

        grant all on os.* to ‘os’@’%’ identified by ‘os_password’

        grant all on os.* to ‘os’@’${m_hosts[${c_index}]}’ identified by ‘os_password’

        drop database os

        create database os

        flush privileges

    安装并启动httpd服务：

        运行 yum install -y httpd 安装httpd服务

        运行 systemctl enable httpd 设置httpd服务开机启动

        运行 service httpd start 启动httpd服务

    创建parcels服务（依次运行如下命令）：

        cd /var/www/html

        mkdir parcels

        cd parcels

        mv ~/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel ./

        mv ~/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha ./

        mv ~/manifest.json ./

    创建cm5服务（依次运行如下命令）：

         mkdir /opt/rpminstall

         cd /opt/rpminstall

         mv ~/cm5.11.1-centos7.tar.gz ./

         mv ~/cm5.11.1-centos7.tar.gz.sha ./

         tar -zxf cm5.11.1-centos7.tar.gz -C /var/www/html/

         cd /var/www/html/

         mkdir -p cm5/redhat/7/x86_64/

         mv cm cm5/redhat/7/x86_64/

         chown apache:apache -R /var/www/html

         cd /var/www/html/cm5/redhat/7/x86_64/cm/5/RPMS/x86_64

         yum install -y cloudera-manager-daemons-5.11.1-1.cm5111.p0.9.el7.x86_64.rpm

         yum install -y cloudera-manager-server-5.11.1-1.cm5111.p0.9.el7.x86_64.rpm

         mkdir /usr/share/java

         cd /usr/share/java

         mv ~/mysql-connector-java-5.1.42.zip ./

         unzip mysql-connector-java-5.1.42.zip

         cd mysql-connector-java-5.1.42

         cp mysql-connector-java-5.1.42-bin.jar ../mysql-connector-java.jar

         echo “com.cloudera.cmf.db.type=mysql” > /etc/cloudera-scm-server/db.properties

         echo “com.cloudera.cmf.db.host=localhost” >> /etc/cloudera-scm-server/db.properties

         echo “com.cloudera.cmf.db.name=cmf” >> /etc/cloudera-scm-server/db.properties

         echo “com.cloudera.cmf.db.user=cmf” >> /etc/cloudera-scm-server/db.properties

         echo “com.cloudera.cmf.db.password=cmf_password” >> /etc/cloudera-scm-server/db.properties

    运行 service cloudera-scm-server start 启动 cloudera-scm 服务

    运行 tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log 查看运行日志，待日志停止且显示Started Jetty server.表示服务运行完成。

    与浏览器输入 http://192.168.75.136:7180开始正式的集群部署

默认用户名密码都为admin
同意并继续
选择适合自己的版本并继续
点击继续
填入3台服务器的ip并点击搜索
确认可行后点击继续
与浏览器访问 http://192.168.75.136/parcels/确认parcels服务，若出现如图则表示ok
与浏览器访问 http://192.168.75.136/cm5/redhat/7/x86_64/cm/5/ 确认cm5服务，若出现如图则表示ok
点击更多选项配置parcel服务
填入之前部署的本地parcel服务并删除多余的url点击保存更改
点击自定义存储库配置cm5服务
配置cm5服务并点击继续
直接点击继续
直接点击继续
选择合适的登陆配置（密码或私钥），点击继续
等待安装完成点击继续
等待安装完parcel点击继续
检测完成后点击完成（这里忽略这两个警告，可自行修复）
这里选择所有服务以测试是否可以完成所有服务的安装

这里直接点继续，如果需要自定义可自行定义。
填写创建好的数据库信息测试链接ok后点击继续
这里直接点击继续（用户可自行配置）
等都通过后点击继续，如果hive的安装出现问题可参考文件末尾教程再操作。
点击完成，完成安装

至此完成安装

文章的最后附上个人的脚本（吐槽下简书的书写为了格式会导致粘贴部分文案行不对号，如果要使用这个脚本需要更改（ 主机ip，主机名））：

最后为了使脚本能够运行，需要运行 chmod +x filepath 为文件添加权限，运行时执行 bash filepath n 例如（bash cdh.sh 0），运行此脚本可能mysql初始化不完全，如果可以，最好还是弄懂所有的步骤。

====================================================================================================master

#!/bin/bash

c_index=$1

m_names=(“master” “slave01” “slave02”)

m_hosts=(“192.168.1.131” “192.168.1.129” “192.168.1.226”)

function auto_ssh_keygen(){

expect -c “set timeout -1;

spawn ssh-keygen;

expect {

*id_rsa* {

send \n\r;

exp_continue;

};

*Overwrite* {

send y\n\r;

exp_continue;

};

*passphrase* {

send \n\r;

exp_continue;

};

*again* {

send \n\r;

exp_continue;

};

};

“

}

function auto_ssh_copy(){

expect -c “set timeout -1;

spawn ssh-copy-id $1;

expect {

*yes/no* {

send yes\n\r;

exp_continue;

};

*password* {

send $2\n\r;

exp_continue;

};

};

“

}

#初始化必要工具

yum install expect -y

yum install unzip -y

#关闭selinux

setenforce 0

echo “SELINUX=disabled” > /etc/selinux/config

echo “SELINUXTYPE=targeted” >> /etc/selinux/config

#关闭防火墙

systemctl stop firewalld.service

systemctl status firewalld.service

systemctl disable firewalld.service

#定义主机名

hostname ${m_names[${c_index}]}

echo “HOSTNAME=${m_names[${c_index}]}” > /etc/sysconfig/network

#ip与hostname绑定

echo “${m_names[${c_index}]}” > /etc/hostname

t_index=0

for m_name in ${m_names[@]};

do

echo “${m_hosts[t_index]} $m_name” >> /etc/hosts

let t_index++

done

#时间同步

yum install ntp -y

systemctl enable ntpd

if [ ${c_index} == 0 ]

then

echo “driftfile /var/lib/ntp/ntp.drift” > /etc/ntp.conf

echo “restrict 192.168.75.0 mask 255.255.255.0 nomodify notrap” >> /etc/ntp.conf

echo “server 210.72.145.44” >> /etc/ntp.conf

echo “server 202.112.10.36 perfer” >> /etc/ntp.conf

echo “server 59.124.196.83” >> /etc/ntp.conf

echo “restrict 210.72.145.44 nomodify notrap noquery” >> /etc/ntp.conf

echo “restrict 202.112.10.36 nomodify notrap noquery” >> /etc/ntp.conf

echo “restrict 59.124.196.83 nomodify notrap noquery” >> /etc/ntp.conf

echo “server  127.127.1.0” >> /etc/ntp.conf

echo “fudge  127.127.1.0 stratum 10” >> /etc/ntp.conf

service ntpd restart

ntpdate –u 202.112.10.36

#免密登陆

for m_name in ${m_names[@]};

do

auto_ssh_copy $m_name root

done

else

echo “driftfile /var/lib/ntp/ntp.drift” > /etc/ntp.conf

echo “statistics loopstats peerstats clockstats” >> /etc/ntp.conf

echo “filegen loopstats file loopstats type day enable” >> /etc/ntp.conf

echo “filegen peerstats file peerstats type day enable” >> /etc/ntp.conf

echo “filegen clockstats file clockstats type day enable” >> /etc/ntp.conf

echo “server ${m_hosts[0]} prefer” >> /etc/ntp.conf

echo “fudge ${m_hosts[0]} stratum 5” >> /etc/ntp.conf

echo “restrict -4 default kod notrap nomodify nopeer noquery” >> /etc/ntp.conf

echo “restrict -6 default kod notrap nomodify nopeer noquery” >> /etc/ntp.conf

echo “restrict 127.0.0.1” >> /etc/ntp.conf

echo “restrict ::1” >> /etc/ntp.conf

service ntpd restart

ntpdate –u ${m_hosts[0]}

fi

#生成私钥

rm -rf /root/.ssh/id_rsa

rm -rf /root/.ssh/id_rsa.pub

auto_ssh_keygen

#安装jdk

rpm -Uvh jdk-8u121-linux-x64.rpm

echo “JAVA_HOME=/usr/java/jdk1.8.0_121” > /etc/profile.d/java.sh

echo “PATH=\$PATH:\$JAVA_HOME/bin” >> /etc/profile.d/java.sh

echo “CLASSPATH=.:\$JAVA_HOME/lib/jt.jar:\$JAVA_HOME/lib/tools.jar” >> /etc/profile.d/java.sh

echo “export JAVA_HOME PATH CLASSPATH” >> /etc/profile.d/java.sh

source /etc/profile

if [ ${c_index} == 0 ]

then

#安装并初始化配置mysql

tar -zxvf mysql-5.6.36-linux-glibc2.5-x86_64.tar.gz

mv mysql-5.6.36-linux-glibc2.5-x86_64 /usr/local/mysql

touch /etc/my.cnf

echo “[mysql]” > /etc/my.cnf

echo “default-character-set=utf8” >> /etc/my.cnf

echo “socket=/var/lib/mysql/mysql.sock” >> /etc/my.cnf

echo “[mysqld]” >> /etc/my.cnf

echo “skip-name-resolve” >> /etc/my.cnf

echo “port=3306” >> /etc/my.cnf

echo “socket=/var/lib/mysql/mysql.sock” >> /etc/my.cnf

echo “basedir=/usr/local/mysql” >> /etc/my.cnf

echo “datadir=/usr/local/mysql/data” >> /etc/my.cnf

echo “max_connections=200” >> /etc/my.cnf

echo “character-set-server=utf8” >> /etc/my.cnf

echo “default-storage-engine=INNODB” >> /etc/my.cnf

echo “lower_case_table_names=1” >> /etc/my.cnf

echo “max_allowed_packet=16M” >> /etc/my.cnf

mkdir -p /var/lib/mysql/mysql

groupadd mysql

useradd -g mysql mysql

chown -R mysql:mysql /usr/local/mysql

chown -R mysql:mysql /var/lib/mysql

yum install -y perl-Module-Install.noarch

/usr/local/mysql/scripts/mysql_install_db –user=mysql –basedir=/usr/local/mysql

chown -R mysql:mysql /usr/local/mysql

chown 777 /etc/my.cnf

cp /usr/local/mysql/support-files/mysql.server /etc/rc.d/init.d/mysqld

chmod +x /etc/rc.d/init.d/mysqld

chkconfig –add mysqld

echo “export PATH=\$PATH:/usr/local/mysql/bin” > /etc/profile.d/mysql.sh

source /etc/profile

service mysqld restart

/usr/local/mysql/bin/mysql -e “delete from mysql.user where user = ””

/usr/local/mysql/bin/mysql -e “update mysql.user set password = (select password(‘root’)) where user = ‘root'”

/usr/local/mysql/bin/mysql -e “create database cmf default character set utf8”

/usr/local/mysql/bin/mysql -e “grant all on cmf.* to ‘cmf’@’localhost’ identified by ‘cmf_password'”

/usr/local/mysql/bin/mysql -e “grant all on cmf.* to ‘cmf’@’%’ identified by ‘cmf_password'”

/usr/local/mysql/bin/mysql -e “grant all on cmf.* to ‘cmf’@’${m_hosts[${c_index}]}’ identified by ‘cmf_password'”

/usr/local/mysql/bin/mysql -e “drop database cmf”

/usr/local/mysql/bin/mysql -e “create database cmf”

/usr/local/mysql/bin/mysql -e “create database hive default character set utf8”

/usr/local/mysql/bin/mysql -e “grant all on hive.* to ‘hive’@’localhost’ identified by ‘hive_password'”

/usr/local/mysql/bin/mysql -e “grant all on hive.* to ‘hive’@’%’ identified by ‘hive_password'”

/usr/local/mysql/bin/mysql -e “grant all on hive.* to ‘hive’@’${m_hosts[${c_index}]}’ identified by ‘hive_password'”

/usr/local/mysql/bin/mysql -e “drop database hive”

/usr/local/mysql/bin/mysql -e “create database hive”

/usr/local/mysql/bin/mysql -e “create database hue default character set utf8”

/usr/local/mysql/bin/mysql -e “grant all on hue.* to ‘hue’@’localhost’ identified by ‘hue_password'”

/usr/local/mysql/bin/mysql -e “grant all on hue.* to ‘hue’@’%’ identified by ‘hue_password'”

/usr/local/mysql/bin/mysql -e “grant all on hue.* to ‘hue’@’${m_hosts[${c_index}]}’ identified by ‘hue_password'”

/usr/local/mysql/bin/mysql -e “drop database hue”

/usr/local/mysql/bin/mysql -e “create database hue”

/usr/local/mysql/bin/mysql -e “create database am default character set utf8”

/usr/local/mysql/bin/mysql -e “grant all on am.* to ‘am’@’localhost’ identified by ‘am_password'”

/usr/local/mysql/bin/mysql -e “grant all on am.* to ‘am’@’%’ identified by ‘am_password'”

/usr/local/mysql/bin/mysql -e “grant all on am.* to ‘am’@’${m_hosts[${c_index}]}’ identified by ‘am_password'”

/usr/local/mysql/bin/mysql -e “drop database am”

/usr/local/mysql/bin/mysql -e “create database am”

/usr/local/mysql/bin/mysql -e “create database os default character set utf8”

/usr/local/mysql/bin/mysql -e “grant all on os.* to ‘os’@’localhost’ identified by ‘os_password'”

/usr/local/mysql/bin/mysql -e “grant all on os.* to ‘os’@’%’ identified by ‘os_password'”

/usr/local/mysql/bin/mysql -e “grant all on os.* to ‘os’@’${m_hosts[${c_index}]}’ identified by ‘os_password'”

/usr/local/mysql/bin/mysql -e “drop database os”

/usr/local/mysql/bin/mysql -e “create database os”

/usr/local/mysql/bin/mysql -e “flush privileges”

#安装httpd服务

yum install -y httpd

systemctl enable httpd

service httpd restart

cd /var/www/html

mkdir parcels

cd parcels

mv ~/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel ./

mv ~/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha ./

mv ~/manifest.json ./

mkdir /opt/rpminstall

cd /opt/rpminstall

mv ~/cm5.11.1-centos7.tar.gz ./

mv ~/cm5.11.1-centos7.tar.gz.sha ./

tar -zxf cm5.11.1-centos7.tar.gz -C /var/www/html/

cd /var/www/html/

mkdir -p cm5/redhat/7/x86_64/

mv cm cm5/redhat/7/x86_64/

chown apache:apache -R /var/www/html

cd /var/www/html/cm5/redhat/7/x86_64/cm/5/RPMS/x86_64

yum install -y cloudera-manager-daemons-5.11.1-1.cm5111.p0.9.el7.x86_64.rpm

yum install -y cloudera-manager-server-5.11.1-1.cm5111.p0.9.el7.x86_64.rpm

mkdir /usr/share/java

cd /usr/share/java

mv ~/mysql-connector-java-5.1.42.zip ./

unzip mysql-connector-java-5.1.42.zip

cd mysql-connector-java-5.1.42

cp mysql-connector-java-5.1.42-bin.jar ../mysql-connector-java.jar

echo “com.cloudera.cmf.db.type=mysql” > /etc/cloudera-scm-server/db.properties

echo “com.cloudera.cmf.db.host=localhost” >> /etc/cloudera-scm-server/db.properties

echo “com.cloudera.cmf.db.name=cmf” >> /etc/cloudera-scm-server/db.properties

echo “com.cloudera.cmf.db.user=cmf” >> /etc/cloudera-scm-server/db.properties

echo “com.cloudera.cmf.db.password=cmf_password” >> /etc/cloudera-scm-server/db.properties

service cloudera-scm-server restart

source /etc/profile

fi

======================================================================================================end

坑锦集：

1.脚本运行完成后可能需要再次于master运行source /etc/profile 使环境变量生效。

2.如果遇到hive安装失败的问题可能需要再开个浏览器窗口访问cdh搭建主网站更改hive配置，且需要使用独立的mysql数据库及用户。

更改hive配置

3.ntp时间同步服务如果报警，可于crontab -e添加定时同步时间。

4.当3台机器服务时，可将master也添加namenode节点。