Hadoop 均衡器
Hadoop在運(yùn)行過程中，其datanode的塊會越來越不平衡，不平衡的集群會導(dǎo)致部分datanode相對更繁忙。
Hadoop的均衡器是一個守護(hù)進(jìn)程。它會重新分配塊，將塊從忙碌的datanode移到相對空閑的datanode。同時堅(jiān)持復(fù)本策略，將復(fù)本分散到不同機(jī)架，以降低數(shù)據(jù)損壞率。
集群均衡標(biāo)準(zhǔn)：每個datanode的使用率和集群的使用率非常接近，差距不超過給定的閥值。
datanode使用率：該節(jié)點(diǎn)上已使用的空間與空間總量之間的比率；
集群的使用率：集群中已使用的空間與集群的總空間總量之間的比率。
為了降低集群負(fù)荷、避免干擾其他用戶，均衡器后臺運(yùn)行，并限制不同節(jié)點(diǎn)之間復(fù)制數(shù)據(jù)的帶寬，默認(rèn)值為1MB/s。
Hadoop的均衡器默認(rèn)是關(guān)閉的，啟動需要執(zhí)行%start-balancer.sh，集群均衡后會自動停止。
HBase 負(fù)載均衡
HBase的均衡器默認(rèn)五分鐘執(zhí)行一次，由 hbase.balancer.period屬性設(shè)置。均衡器運(yùn)行后會嘗試均勻分配region到所有region服務(wù)器。
先確定一個region分配計(jì)劃，然后開始移動region。
通過均衡器的開關(guān)類控制開啟和關(guān)閉狀態(tài)。
HBase的均衡器默認(rèn)是開啟的，并周期性運(yùn)行。
下面是官網(wǎng)的解釋說明：
Over time, the distribution of blocks across datanodes can become unbalanced. An unbalanced cluster can affect locality for MapReduce, and it puts a greater strain on the highly utilized datanodes, so it’s best avoided.
The balancer program is a Hadoop daemon that redistributes blocks by moving them from overutilized datanodes to underutilized datanodes, while adhering to the block replica placement policy that makes data loss unlikely by placing block replicas on different racks (see Replica Placement). It moves blocks until the cluster is deemed to be balanced, which means that the utilization of every datanode (ratio of used space on the node to total capacity of the node) differs from the utilization of the cluster (ratio of used space on the cluster to total capacity of the cluster) by no more than a given threshold percentage. You can start the balancer with:
% start-balancer.sh
The -threshold argument specifies the threshold percentage that defines what it means for the cluster to be balanced. The flag is optional; if omitted, the threshold is 10%. At any one time, only one balancer may be running on the cluster.
The balancer runs until the cluster is balanced, it cannot move any more blocks, or it loses contact with the namenode. It produces a logfile in the standard log directory, where it writes a line for every iteration of redistribution that it carries out. Here is the output from a short run on a small cluster (slightly reformatted to fit the page):
Time Stamp Iteration# Bytes Already Moved ...Left To Move ...Being Moved
Mar 18, 2009 5:23:42 PM 0 0 KB 219.21 MB 150.29 MB
Mar 18, 2009 5:27:14 PM 1 195.24 MB 22.45 MB 150.29 MB
The cluster is balanced. Exiting...
Balancing took 6.072933333333333 minutes
The balancer is designed to run in the background without unduly taxing the cluster or interfering with other clients using the cluster. It limits the bandwidth that it uses to copy a block from one node to another. The default is a modest 1 MB/s, but this can be changed by setting the dfs.datanode.balance.bandwidthPerSec property in hdfs-site.xml, specified in bytes.
The master has a built-in feature, called the balancer. By default, the balancer runs every five minutes, and it is configured by the hbase.balancer.period property. Once the balancer is started, it will attempt to equal out the number of assigned regions per region server so that they are within one region of the average number per server. The call first determines a new assignment plan, which describes which regions should be moved where. Then it starts the process of moving the regions by calling the unassign() method of the administrative API iteratively.
The balancer has an upper limit on how long it is allowed to run, which is configured using the hbase.balancer.max.balancing property anddefaults to half of the balancer period value, or two and a half minutes.
You can control the balancer by means of the balancer switch: either use the shell’s balance_switch command to toggle the balancer status between enabled and disabled, or use the balanceSwitch() API method to do the same. When you disable the balancer, it no longer runs as expected.
The balancer can be explicitly started using the shell’s balancer command, or using the balancer() API method. The time-controlled invocation mentioned previously calls this method implicitly. It will determine if there is any work to be done and return true if that is the case. The return value of false means that it was not able to run the balancer, because either it was switched off, there was no work to be done (all is balanced), or something else was prohibiting the process. One example for this is the region in transition list (see Main page): if there is a region currently in transition, the balancer will be skipped.
Instead of relying on the balancer to do its work properly, you can use the move command and API method to assign regions to other servers. This is useful when you want to control where the regions of a particular table are assigned. See Region Hotspotting for an example.
另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)scvps.cn，海內(nèi)外云服務(wù)器15元起步，三天無理由+7*72小時售后在線，公司持有idc許可證，提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國服務(wù)器、虛擬主機(jī)、免備案服務(wù)器”等云主機(jī)租用服務(wù)以及企業(yè)上云的綜合解決方案，具有“安全穩(wěn)定、簡單易用、服務(wù)可用性高、性價比高”等特點(diǎn)與優(yōu)勢，專為企業(yè)上云打造定制，能夠滿足用戶豐富、多元化的應(yīng)用場景需求。
重慶分公司
重慶分公司
