真实的国产乱ⅩXXX66竹夫人,五月香六月婷婷激情综合,亚洲日本VA一区二区三区,亚洲精品一区二区三区麻豆

成都創(chuàng)新互聯(lián)網(wǎng)站制作重慶分公司

centos7hadoop-2.7.3完全分布式安裝配置-創(chuàng)新互聯(lián)

操作系統(tǒng)版本:

新民ssl適用于網(wǎng)站、小程序/APP、API接口等需要進行數(shù)據(jù)傳輸應(yīng)用場景,ssl證書未來市場廣闊!成為創(chuàng)新互聯(lián)公司的ssl證書銷售渠道,可以享受市場價格4-6折優(yōu)惠!如果有意向歡迎電話聯(lián)系或者加微信:13518219792(備注:SSL證書合作)期待與您的合作!

centos7 64位

hadoop版本:

hadoop-2.7.3

hbase版本:

hbase-1.2.4

機器:

192.168.11.131 master1       Namenode    ResourceManager  QuorumPeerMain Jobhistory  HMaster  DFSZKFailoverController

192.168.11.132 master2       Namenode        HMaster       DFSZKFailoverController

192.168.11.133 slave1        Datanode    HRegionServer   NodeManager           JournalNode

192.168.11.134 slave2        Datanode    HRegionServer   NodeManager           JournalNode

192.168.11.135 slave3        Datanode    HRegionServer   NodeManager           JournalNode

所有節(jié)點關(guān)閉防火墻及selinux

# firewall-cmd --state

running

# systemctl stop firewalld.service

# systemctl disable firewalld.service

# setenforce 0

# vi /etc/sysconfig/selinux

SELINUX=enforcing --> disabled

所有節(jié)點配置yum源

# cd

# mkdir apps

http://mirrors.163.com/centos/7/os/x86_64/Packages/wget-1.14-15.el7.x86_64.rpm

# rpm -i wget-1.14-15.el7.x86_64.rpm

# cd /etc/yum.repos.d

# wget http://mirrors.aliyun.com/repo/Centos-7.repo

# mv Centos-7.repo CentOS-Base.repo

# scp CentOS-Base.repo root@192.168.11.131:/etc/yum.repos.d/

# scp CentOS-Base.repo root@192.168.11.132:/etc/yum.repos.d/

# scp CentOS-Base.repo root@192.168.11.133:/etc/yum.repos.d/

# scp CentOS-Base.repo root@192.168.11.134:/etc/yum.repos.d/

# yum clean all

# yum makecache

# yum update

配置ntp時間同步

所有節(jié)點安裝ntp

# yum install -y ntp

ntp server端:

# date -s "2018-05-27 23:03:30"

# vi /etc/ntp.conf

在注釋下添加兩行

#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

server 127.127.1.0

fudge 127.127.1.0 stratum 11

注釋下面

#server 0.centos.pool.ntp.org iburst

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst

# systemctl start ntpd.service

# systemctl enable ntpd.service

ntp客戶端(其余四臺都為ntp客戶端):

# vi /etc/ntp.conf

同樣注釋下添加兩行

#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

server 192.168.11.131

fudge 127.127.1.0 stratum 11

四行添加注釋

#server 0.centos.pool.ntp.org iburst

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst

# systemctl start ntpd.service

# systemctl enable ntpd.service

# ntpdate 192.168.11.131

28 May 07:04:50 ntpdate[1714]: the NTP socket is in use, exiting

# lsof -i:123

-bash: lsof: command not found

# yum install -y lsof

# lsof -i:123

COMMAND PID USER  FD  TYPE DEVICE SIZE/OFF NODE NAME

ntpd  1693 ntp  16u IPv4 25565   0t0 UDP *:ntp

ntpd  1693 ntp  17u IPv6 25566   0t0 UDP *:ntp

ntpd  1693 ntp  18u IPv4 25572   0t0 UDP localhost:ntp

ntpd  1693 ntp  19u IPv4 25573   0t0 UDP localhost.localdomain:ntp

ntpd  1693 ntp  20u IPv6 25574   0t0 UDP localhost:ntp

ntpd  1693 ntp  21u IPv6 25575   0t0 UDP localhost.localdomain:ntp

# kill -9 1693

# ntpdate 192.168.11.131

27 May 23:06:14 ntpdate[1728]: step time server 192.168.11.131 offset -28808.035509 sec

# date

Sun May 27 23:06:17 CST 2018

所有節(jié)點修改主機名(永久)

# hostnamectl set-hostname master1~slave3

臨時修改主機名

# hostname master1~slave3

主節(jié)點修改hosts文件

# vi /etc/hosts

192.168.11.131 master1

192.168.11.132 master2

192.168.11.133 slave1

192.168.11.134 slave2

192.168.11.135 slave3

把hosts文件覆蓋到其他機器

# scp /etc/hosts root@192.168.11.132~135:/etc/

所有節(jié)點創(chuàng)建管理用戶和組

創(chuàng)建組和用戶

# groupadd hduser

# useradd -g hduser hduser

# passwd hduser

創(chuàng)建目錄并賦權(quán)

每臺機器上創(chuàng)建如下文件夾

# mkdir /data1

# mkdir /data2

修改權(quán)限

# chown hudser:hduser /data1

# chown hudser:hduser /data2

# su hduser

$ mkdir -p /data1/hadoop_data/hdfs/namenode

$ mkdir -p /data2/hadoop_data/hdfs/namenode

$ mkdir -p /data1/hadoop_data/hdfs/datanode(NameNode不要)

$ mkdir -p /data2/hadoop_data/hdfs/datanode(NameNode不要)

$ mkdir -p /data1/hadoop_data/pids

$ mkdir -p /data2/hadoop_data/pids

$ mkdir -p /data1/hadoop_data/hadoop_tmp

$ mkdir -p /data2/hadoop_data/hadoop_tmp

無密驗證

master1和master2節(jié)點操作

# su - hduser

$ ssh-keygen -t rsa

$ cd ~/.ssh

$ cat id_rsa.pub >> authorized_keys

master1節(jié)點操作

$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@master2

master2節(jié)點操作

$ scp ~/.ssh/authorized_keys hduser@master1:~/.ssh/

slave1、slave2和slave3節(jié)點創(chuàng)建.ssh目錄

# mkdir /home/hduser/.ssh

# chown hduser:hduser /home/hduser/.ssh

master1節(jié)點操作

$ scp ~/.ssh/authorized_keys hduser@slave1:~/.ssh

$ scp ~/.ssh/authorized_keys hduser@slave2:~/.ssh

$ scp ~/.ssh/authorized_keys hduser@slave3:~/.ssh

master1和master2節(jié)點驗證

驗證方法,分別在兩個節(jié)點,ssh登陸本機(hdusser用戶)及其他四個節(jié)點,看看是不是無密登陸。

如果未通過驗證,所有機器執(zhí)行下面命令

$ chmod 600 ~/.ssh/authorized_keys

$ chmod 700 ~/.ssh

所有節(jié)點配置java環(huán)境

$ mkdir -p /data1/usr/src

上傳包到/data1/usr/src目錄下

$ cd /data1/usr/src

$ tar xf jdk1.7.0_79.tar -C /data1/usr/

$ vi ~/.bashrc

export JAVA_HOME=/data1/usr/jdk1.7.0_79

export JRE_HOME=$JAVA_HOME/jre

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib/rt.jar

export PATH=$PATH:$JAVA_HOME/bin

$ source ~/.bashrc

mastar1節(jié)點配置hadoop(hdsuer用戶)

下載hadoop-2.7.3.tar.gz,上傳到/data1/usr/src

http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz

$ cd /data1/usr/src

$ tar -zxf hadoop-2.7.3.tar.gz -C /data1/usr/

$ vi /data1/usr/hadoop-2.7.3/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/data1/usr/jdk1.7.0_79

export HADOOP_PID_DIR=/data1/hadoop_data/pids

export HADOOP_PID_DIR=/data2/hadoop_data/pids

export HADOOP_MAPRED_PID_DIR=/data1/hadoop_data/pids

$ vi /data1/usr/hadoop-2.7.3/etc/hadoop/mapred-env.sh

export HADOOP_MAPRED_PID_DIR=/data2/hadoop_data/pids

$ vi /data1/usr/hadoop-2.7.3/etc/hadoop/yarn-env.sh

export YARN_PID_DIR=/data2/hadoop_data/pids

$ vi /data1/usr/hadoop-2.7.3/etc/hadoop/core-site.xml

 

  fs.defaultFS

  hdfs://masters

 

 

 

  hadoop.tmp.dir

  /data2/hadoop_data/hadoop_tmp

 

 

 

  ha.zookeeper.quorum

  master1:2181,master2:2181,slave1:2181,slave2:2181,slave3:2181

 

$ vi /data1/usr/hadoop-2.7.3/etc/hadoop/hdfs-site.xml

    

    

        dfs.nameservices

        masters

    

    

    

        dfs.ha.namenodes.masters

        master1,master2

    

    

    

        dfs.namenode.rpc-address.masters.master1

        master1:9000

    

    

    

        dfs.namenode.http-address.masters.master1

        master1:50070

    

    

    

        dfs.namenode.rpc-address.masters.master2

        master2:9000

    

    

    

        dfs.namenode.http-address.masters.master2

        master2:50070

    

    

    

        dfs.namenode.name.dir

        file:///data2/hadoop_data/hdfs/namenode

    

    

    

        dfs.datanode.data.dir

        file:///data1/hadoop_data/hdfs/datanode,data2/hadoop_data/hdfs/datanode

    

    

    

        dfs.namenode.shared.edits.dir

        qjournal://slave1:8485;slave2:8485;slave3:8485/masters

    

    

    

        dfs.journalnode.edits.dir

        /data2/hadoop_data/journal

    

    

    

        dfs.ha.automatic-failover.enabled

        true

    

    

    

        dfs.client.failover.proxy.provider.masters

        org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

    

    

    

        dfs.ha.fencing.methods

        

         sshfence

         shell(/bin/true)

        

    

    

    

        dfs.ha.fencing.ssh.private-key-files

        /home/hduser/.ssh/id_rsa

    

    

    

        dfs.ha.fencing.ssh.connect-timeout

        30000

    

    

    

        dfs.datanode.max.xcievers

        8192

    

    

        dfs.qjournal.write-txns.timeout.ms

        60000

    

$ vi /data1/usr/hadoop-2.7.3/etc/hadoop/yarn-site.xml

    

        yarn.resourcemanager.ha.enabled

        true

    

    

    

        yarn.resourcemanager.cluster-id

        RM_HA_ID

    

    

    

        yarn.resourcemanager.ha.rm-ids

        rm1,rm2

    

    

    

        yarn.resourcemanager.hostname.rm1

        master1

    

    

        yarn.resourcemanager.hostname.rm2

        master2

    

    

        yarn.resourcemanager.recovery.enabled

        true

    

    

        yarn.resourcemanager.store.class

        org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore

    

    

    

        yarn.resourcemanager.zk-address

        master1:2181,master2:2181,slave1:2181,slave2:2181,slave3:2181

    

    

        yarn.nodemanager.aux-services

        mapreduce_shuffle

    

    

        yarn.nodemanager.aux-services.mapreduce.shuffle.class

        org.apache.hadoop.mapred.ShuffleHandler

    

$ cp /data1/usr/hadoop-2.7.3/etc/hadoop/mapred-site.xml.template /data1/usr/hadoop-2.7.3/etc/hadoop/mapred-site.xml

$ vi /data1/usr/hadoop-2.7.3/etc/hadoop/mapred-site.xml

 

 

  mapreduce.framework.name

  yarn

 

$ vi /data1/usr/hadoop-2.7.3/etc/hadoop/slaves

slave3

slave4

slave5

$ for ip in `seq 2 5`;do scp -rpq /data1/usr/hadoop-2.7.3 192.168.11.13$ip:/data1/usr;done

各節(jié)點zookeeper配置

http://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

上傳包到/data1/usr/src目錄下

創(chuàng)建目錄

$ mkdir -p /home/hduser/storage/zookeeper

$ cd /data1/usr/src

$ tar -zxf zookeeper-3.4.6.tar.gz -C /data1/usr

$ cp /data1/usr/zookeeper-3.4.6/conf/zoo_sample.cfg /data1/usr/zookeeper-3.4.6/conf/zoo.cfg

$ vi /data1/usr/zookeeper-3.4.6/conf/zoo.cfg

dataDir=/home/hduser/storage/zookeeper

server.1=master1:2888:3888

server.2=master2:2888:3888

server.3=slave1:2888:3888

server.4=slave2:2888:3888

server.5=slave3:2888:3888

master1-slave3各節(jié)點依次做操作

$ echo "1" > /home/hduser/storage/zookeeper/myid

$ echo "2" > /home/hduser/storage/zookeeper/myid

$ echo "3" > /home/hduser/storage/zookeeper/myid

$ echo "4" > /home/hduser/storage/zookeeper/myid

$ echo "5" > /home/hduser/storage/zookeeper/myid

$ cd /data1/usr/zookeeper-3.4.6/bin

$ ./zkServer.sh start

slave1、slave2和slave3啟動journalnode

$ cd /data1/usr/hadoop-2.7.3/sbin

$ ./sbin/hadoop-daemon.sh start journalnode

用jps確認(rèn)啟動結(jié)果

在master1上格式化zookeeper節(jié)點格式化(第一次)

$ cd /data1/usr/hadoop-2.7.3

$ ./bin/hdfs zkfc -formatZK

在master1上執(zhí)行命令:

./bin/hadoop namenode -format

在master1上啟動namenode

./sbin/hadoop-daemon.sh start namenode

需要在master2(備節(jié)點)上執(zhí)行數(shù)據(jù)同步

./bin/hdfs namenode –bootstrapStandby

scp -r /data2/hadoop_data/hdfs/namenode hduser@mster2:/data2/hadoop_data/hdfs/

在master2上啟動namenode

./sbin/hadoop-daemon.sh start namenode

設(shè)置master1為active

./bin/hdfs haadmin -transitionToActive master1

./bin/hdfs haadmin -getServiceState master1

在master1上啟動datanode

./sbin/hadoop-daemons.sh start datanode

啟動HDFS(第二次以后)

在master1上執(zhí)行命令:

./sbin/start-dfs.sh

啟動YARN

在master1上執(zhí)行命令:

./sbin/start-yarn.sh

驗證

驗證namenode

http://master1:50070

Overview 'master1:9000' (active)

http://master2:50070

Overview 'master2:9000' (standby)

上傳文件

./bin/hadoop fs -put /data1/usr/hadoop-2.7.3/etc/hadoop /test

./bin/hadoop fs -ls /test

namenode的備份驗證

殺死m(xù)aster1,master2變?yōu)閍ctive

驗證yarn

./bin/hadoop jar /data1/usr/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /test/hadoop /test/out

安裝HBASE

master1節(jié)點操作:

下載hbase-1.2.4-bin.tar.gz,解壓

$ cd /data1/usr/src

$ tar -zxvf hbase-1.2.4-bin.tar.gz -C /data1/usr/

$ mkdir -p /data1/hadoop_data/hbase_tmp

$ mkdir -p /data2/hadoop_data/hbase_tmp

配置master1的hbase環(huán)境

配置hbase-env.sh

$ vi /data1/usr/hbase-1.2.4/conf/hbase-env.sh

export JAVA_HOME=/data1/usr/jdk1.7.0_79

export HBASE_PID_DIR=/data2/hadoop_data/pids

export HBASE_MANAGES_ZK=false

export HADOOP_HOME=/data1/usr/hadoop-2.7.3

配置hbase-site.xml

$ vi /data1/usr/hbase-1.2.4/conf/hbase-site.xml

  

    hbase.rootdir

    hdfs://masters/hbase

  

  

  

    hbase.cluster.distributed

    true

  

  

    hbase.master.port

    60000

  

  

    hbase.tmp.dir

    /data2/hadoop_data/hbase_tmp

  

  

    hbase.zookeeper.quorum

    master1,master2,slave1,slave2,slave3

  

配置regionservers

$ vi /data1/usr/hbase-1.2.4/conf/regionservers

slave1

slave2

slave3

配置backup-masters

$ vi /data1/usr/hbase-1.2.4/conf/backup-masters

移除 HBase 里面的不必要 log4j 的 jar 包

cd ${HBASE_HOME}/lib

mv slf4j-log4j12-1.7.5.jar slf4j-log4j12-1.7.5.jar.bak

將master1的hbase環(huán)境傳輸?shù)狡渌?jié)點

$ for ip in `seq 2 5`;do scp -rpq /data1/usr/hbase-1.2.4 192.168.11.13$ip:/data1/usr;done

啟動順序

按hadoop集群的啟動步驟,啟動hadoop集群

master1上啟動Hbase

$ cd /data1/usr/hbase-1.2.4/bin

$ ./start-hbase.sh

驗證

$ /data1/usr/hadoop-2.7.3/bin/hadoop fs -ls / 查看hbase是否在HDFS文件系統(tǒng)創(chuàng)建成功

執(zhí)行: bin/hbase shell 可以進入Hbase管理界面、

輸入 status 查看狀態(tài)

創(chuàng)建表

create 'test', 'cf'

顯示表信息

list 'test'

表中插入數(shù)據(jù)

put 'test', 'row1', 'cf:a', 'value1'

put 'test', 'row2', 'cf:b', 'value2'

put 'test', 'row3', 'cf:c', 'value3'

查詢表

scan 'test'

取一行數(shù)據(jù)

get 'test', 'row1'

失效表

disable 'test'

刪除表

drop 'test'

瀏覽器輸入http://master1:16010可以打開Hbase管理界面

http://192.168.11.131/master-status

啟動thrift2

hbase-daemons.sh start thrift2

去datanode節(jié)點用jps確認(rèn)

另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)scvps.cn,海內(nèi)外云服務(wù)器15元起步,三天無理由+7*72小時售后在線,公司持有idc許可證,提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國服務(wù)器、虛擬主機、免備案服務(wù)器”等云主機租用服務(wù)以及企業(yè)上云的綜合解決方案,具有“安全穩(wěn)定、簡單易用、服務(wù)可用性高、性價比高”等特點與優(yōu)勢,專為企業(yè)上云打造定制,能夠滿足用戶豐富、多元化的應(yīng)用場景需求。


當(dāng)前題目:centos7hadoop-2.7.3完全分布式安裝配置-創(chuàng)新互聯(lián)
鏈接地址:http://weahome.cn/article/dohged.html

其他資訊

在線咨詢

微信咨詢

電話咨詢

028-86922220(工作日)

18980820575(7×24)

提交需求

返回頂部