本文共 7501 字，大约阅读时间需要 25 分钟。

脚本和相关的文件如下,提取码: sweh

一.创建一台虚拟机

• 1.准备一台纯净版的centos7系统,修改静态ip为192.168.56.120,主机名为hadoop01,关闭防火墙,重启网络,连接moba
• 2.创建安装包目录:mkdir /opt/software
• 3.创建安装目录: mkdir /opt/install
• 4.将脚本和相关配置文件安装包拖到安装包目录下
• 5.修改脚本的权限:chmod 777 install*
• 6.执行脚本安装jdk:/opt/software/installJdk.sh
• 7.执行脚本安装hadoop和zookeeper:/opt/software/installHadoop.sh
• 8.关闭该虚拟机

二.复制两台虚拟机

• 1.一台修改静态ip为192.168.56.121 ,主机名为hadoop02
• 2.另一台修改静态ip为192.168.56.122 ,主机名为hadoop03
• 3.将hadoop02机器zookeeper的myid修改为2,hadoop03修改为3
• 4.三台机器配置免登录
• 1)ssh-keygen -t rsa -P ""生成私钥
• 2)cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys添加到信任
• 3)远程免登录:ssh-copy-id -i ~/.ssh/id_rsa.pub -p22 root@192.168.56.120 ,注意一台向其他两台发送
• 4)三台机器分别执行:

ssh  -o StrictHostKeyChecking=no `hostname`

三.启动集群

• 1.3台 zkServer.sh start,三台hadoop-daemon.sh start journalnode
• 2.hadoop01 hadoop namenode -format,hadoop-daemon.sh start namenode
• 3.hadoop02 hdfs namenode -bootstrapStandby,只在第一次启动需要同步以后不需要
• 3.hadoop01 hdfs zkfc -formatZK
• 4.三台停掉所有Journalnode进程: hadoop-daemon.sh stop journalnode
• 5.hadoop01hadoop-daemon.sh stop namenode
• 6.hadoop01 start-dfs.sh
• 7.hadoop01 start-yarn.sh
• 8.hadoop02 yarn-daemon.sh start resourcemanager
• 9启动后效果如下:
  
  
  

四.脚本内容如下

1.jdk

#!/bin/bash#安装jdk1.8,需要先创建software目录和install目录,然后将jdk包拖进software目录中tar -zxvf  /opt/software/jdk-8u221-linux-x64.tar.gz -C /opt/install/`echo "export JAVA_HOME=/opt/install/jdk1.8.0_221" >> /etc/profile``echo 'export CLASSPATH=.:$JAVA_HOME/rt.jar:$JAVA_HOME/tools.jar:$JAVA_HOME/dt.jar' >> /etc/profile``echo 'export JRE_HOME=$JAVA_HOME/jre' >> /etc/profile``echo 'export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin' >> /etc/profile`source /etc/profilejava -version

2.hadoop和zookeeper

#!/bin/bash#将zookeeper和hadoop安装包挪到software下,将配置文件也挪到software目录下tar -zxvf  /opt/software/zookeeper-3.4.6.tar.gz -C /opt/install/tar -zxvf  /opt/software/hadoop-2.6.0-cdh5.14.2.tar.gz -C /opt/install/mv /opt/install/hadoop-2.6.0-cdh5.14.2 /opt/install/hadoop`echo "export HADOOP_HOME=/opt/install/hadoop" >> /etc/profile``echo 'export HADOOP_MAPRED_HOME=$HADOOP_HOME' >> /etc/profile``echo 'export HADOOP_COMMON_HOME=$HADOOP_HOME' >> /etc/profile``echo 'export HADOOP_HDFS_HOME=$HADOOP_HOME' >> /etc/profile``echo 'export YARN_HOME=$HADOOP_HOME' >> /etc/profile``echo 'export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native' >> /etc/profile``echo 'export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"' >> /etc/profile``echo 'export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin' >> /etc/profile``echo 'export ZK_HOME=/opt/install/zookeeper-3.4.6' >> /etc/profile``echo 'export PATH=$PATH:$ZK_HOME/bin' >> /etc/profile`source /etc/profilesed -i  '24,26s/\${JAVA_HOME}/\/opt\/install\/jdk1.8.0_221/gi' /opt/install/hadoop/etc/hadoop/hadoop-env.shcat $PWD/core-site.xml > /opt/install/hadoop/etc/hadoop/core-site.xmlcat $PWD/hdfs-site.xml > /opt/install/hadoop/etc/hadoop/hdfs-site.xmlcat $PWD/mapred-site.xml > /opt/install/hadoop/etc/hadoop/mapred-site.xmlcat $PWD/yarn-site.xml > /opt/install/hadoop/etc/hadoop/yarn-site.xmlcat $PWD/zoo.cfg > /opt/install/zookeeper-3.4.6/conf/zoo.cfgcat $PWD/slaves > /opt/install/hadoop/etc/hadoop/slavesmkdir /opt/install/zookeeper-3.4.6/zkDatamkdir   /opt/install/zookeeper-3.4.6/zkLogtouch /opt/install/zookeeper-3.4.6/zkData/myidecho '1' >  /opt/install/zookeeper-3.4.6/zkData/myidecho '192.168.56.120 hadoop01' >> /etc/hostsecho '192.168.56.121 hadoop02' >> /etc/hostsecho '192.168.56.122 hadoop03' >> /etc/hostssource /etc/profiletar -xvf $PWD/hadoop-native-64-2.6.0.tar -C $HADOOP_HOME/lib/nativetar -xvf $PWD/hadoop-native-64-2.6.0.tar -C $HADOOP_HOME/libecho 'hadoop 版本信息:' hadoop version

3.一键启动集群

#!/usr/bin/env bashssh hadoop01 "echo '1' > /opt/install/zookeeper-3.4.6/zkData/myid"ssh hadoop02 "echo '2' > /opt/install/zookeeper-3.4.6/zkData/myid"ssh hadoop03 "echo '3' > /opt/install/zookeeper-3.4.6/zkData/myid"for s in hadoop01 hadoop02 hadoop03do	ssh $s "source /etc/profile; cd /opt/install/hadoop/; rm -rf hdfsTmp/"donesleep 2for s in hadoop01 hadoop02 hadoop03do	ssh $s "source /etc/profile; zkServer.sh start; hadoop-daemon.sh start journalnode"donesleep 2ssh hadoop01 "source /etc/profile; hadoop namenode -format; hadoop-daemon.sh start namenode"sleep 2ssh hadoop02 "source /etc/profile; hdfs namenode -bootstrapStandby"sleep 2ssh hadoop01 "source /etc/profile; hdfs zkfc -formatZK; hadoop-daemon.sh stop journalnode"sleep 2ssh hadoop02 "source /etc/profile; hadoop-daemon.sh stop journalnode"sleep 2ssh hadoop03 "source /etc/profile; hadoop-daemon.sh stop journalnode"sleep 2ssh hadoop01 "source /etc/profile; hadoop-daemon.sh stop namenode; start-dfs.sh; start-yarn.sh"sleep 2ssh hadoop02 "source /etc/profile; yarn-daemon.sh start resourcemanager"sleep 2for s in hadoop01 hadoop02 hadoop03do	echo "=================$s-jps================="	ssh $s "source /etc/profile; jps"donesleep 2for s in hadoop01 do	echo "=================$s -resourcemanager status================="	ssh $s "source /etc/profile; yarn rmadmin -getServiceState rm1"	echo "=================$s -namenode status================="	ssh $s "source /etc/profile; hdfs haadmin -getServiceState nn1"donesleep 2for s in hadoop02do	echo "=================$s -resourcemanager status================="	ssh $s "source /etc/profile; yarn rmadmin -getServiceState rm2"	echo "=================$s -namenode status================="	ssh $s "source /etc/profile; hdfs haadmin -getServiceState nn2"donesleep 2for s in hadoop01 hadoop02 hadoop03do	echo "=================$s- zkServer status================="	ssh $s "source /etc/profile; zkServer.sh status"done

使用脚本运行效果如下:

=================hadoop01-jps=================22896 NameNode22002 QuorumPeerMain23430 DFSZKFailoverController23016 DataNode23674 NodeManager24010 Jps23549 ResourceManager23231 JournalNode=================hadoop02-jps=================15138 NameNode15218 DataNode14900 QuorumPeerMain15317 JournalNode15445 DFSZKFailoverController15787 Jps15534 NodeManager15710 ResourceManager=================hadoop03-jps=================12996 NodeManager12794 DataNode13147 Jps12621 QuorumPeerMain12893 JournalNode=================hadoop01 -resourcemanager status=================20/07/09 08:48:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-jav                                    a classes where applicableactive=================hadoop01 -namenode status=================active=================hadoop02 -resourcemanager status=================20/07/09 08:49:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-jav                                    a classes where applicablestandby=================hadoop02 -namenode status=================standby=================hadoop01- zkServer status=================JMX enabled by defaultUsing config: /opt/install/zookeeper-3.4.6/bin/../conf/zoo.cfgMode: follower=================hadoop02- zkServer status=================JMX enabled by defaultUsing config: /opt/install/zookeeper-3.4.6/bin/../conf/zoo.cfgMode: leader=================hadoop03- zkServer status=================JMX enabled by defaultUsing config: /opt/install/zookeeper-3.4.6/bin/../conf/zoo.cfgMode: follower

注:需要下载psmisc依赖包,否则无法完成自动切换节点

yum install -y psmisc

转载地址：http://jcjxi.baihongyu.com/