這篇文章給大家分享的是有關(guān)Tachyon 0.7.1如何實現(xiàn)偽分布式集群安裝與測試 的內(nèi)容。小編覺得挺實用的，因此分享給大家做個參考，一起跟隨小編過來看看吧。
創(chuàng)新互聯(lián)公司是少有的做網(wǎng)站、網(wǎng)站設(shè)計、營銷型企業(yè)網(wǎng)站、成都微信小程序、手機APP,開發(fā)、制作、設(shè)計、賣鏈接、推廣優(yōu)化一站式服務(wù)網(wǎng)絡(luò)公司,于2013年成立,堅持透明化,價格低,無套路經(jīng)營理念。讓網(wǎng)頁驚喜每一位訪客多年來深受用戶好評
Tachyon是一個高容錯的分布式文件系統(tǒng)，允許文件以內(nèi)存的速度在集群框架中進行可靠的共享，就像Spark和 MapReduce那樣。通過利用信息繼承，內(nèi)存侵入，Tachyon獲得了高性能。Tachyon工作集文件緩存在內(nèi)存中，并且讓不同的 Jobs/Queries以及框架都能內(nèi)存的速度來訪問緩存文件。因此，Tachyon可以減少那些需要經(jīng)常使用的數(shù)據(jù)集通過訪問磁盤來獲得的次數(shù)。
源碼地址:https://github.com/amplab/tachyon
git clone https://github.com/amplab/tachyon.git
進入 TACHYON_HOME
git tagv0.1.0v0.2.0v0.2.1v0.3.0v0.4.0v0.4.1v0.4.1-thriftv0.5.0v0.6.0v0.6.1v0.6.2v0.6.3v0.6.4v0.7.0v0.7.0-rc1v0.7.112345678910111213141516171819
切換到v0.7.1版本
git checkout v0.7.1 Note: checking out 'v0.7.1'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you maydo so (now or later) by using -b with the checkout command again. Example: git checkout -b new_branch_name HEAD 目前位于 a2b69c9... [maven-release-plugin] prepare release v0.7.11234567891011121314
執(zhí)行如下命令:
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"mvn clean package -Dhadoop.version=2.3.0-cdh6.1.0 -Dspark.version=1.4.1 -DskipTests12
創(chuàng)建journal目錄
mkdir -p /home/cluster/apps/tachyon/journal
創(chuàng)建ramdisk
mkdir -p /home/cluster/apps/tachyon/ramdisk
將tachyon_home/conf/tachyon-env.sh.template 修改為tachyon_home/conf/tachyon-env.sh
mv conf/tachyon-env.sh.template conf/tachyon-env.sh
編輯tachyon_home/conf/tachyon-env.sh 增加如下內(nèi)容
export JAVA_HOME=/home/cluster/share/java1.7export HADOOP_HOME=/home/cluster/apps/hadoopexport TACHYON_HOME=/home/cluster/apps/tachyonexport TACHYON_RAM_FOLDER=$TACHYON_HOME/ramdisk export TACHYON_MASTER_ADDRESS=master12345
修改tachyon日志存放路徑:
tachyon_home/下創(chuàng)建logs目錄
mkdir logs
vim conf/log4j.properties 增加如下內(nèi)容:
tachyon.logs.dir=/home/cluster/apps/tachyon/logs
將tachyon_home/conf/core-site.xml.template 修改為tachyon_home/conf/core-site.xml
mv conf/core-site.xml.template conf/core-site.xml
編輯 conf/core-site.xml,增加如下內(nèi)容:
格式化Tachyon
bin/tachyon formatConnecting to localhost as wangyue... Formatting Tachyon Worker @ wangyue-um Connection to localhost closed. Formatting Tachyon Master @ master12345
bin/tachyon-start.sh localKilled 0 processes on wangyue-um Killed 0 processes on wangyue-um Connecting to localhost as wangyue... Killed 0 processes on wangyue-um Connection to localhost closed. [sudo] password for wangyue: Formatting RamFS: /home/cluster/apps/tachyon/ramdisk (1gb) Starting master @ master Starting worker @ wangyue-um jps下:21025 ZeppelinServer22214 Worker21987 Master21360 TachyonWorker32212 RemoteInterpreterServer23361 NameNode29603 RemoteInterpreterServer21324 TachyonMaster23493 DataNode21412 Jps27005 RemoteInterpreterServer 已經(jīng)啟動TachyonMaster & TachyonWorker12345678910111213141516171819202122232425
tachyon-start.sh local命令將同時在本地啟動Master和Worker進程。需要注意的是，運行tachyon-start.sh local命令一定要擁有切換root的密碼，否者會無法啟動。這是因為RamFS的格式化需要root權(quán)限。
webui:http://master:19999 
/home/cluster/apps/tachyon$ bin/tachyon runTest Basic CACHE_THROUGH /default_tests_files/BasicFile_CACHE_THROUGH has been removed2015-09-09 20:00:17,911 INFO (MasterClient.java:connect) - Tachyon client (version 0.7.1) is trying to connect with master @ master/127.0.0.1:199982015-09-09 20:00:17,938 INFO (MasterClient.java:connect) - User registered with the master @ master/127.0.0.1:19998; got UserId 52015-09-09 20:00:17,964 INFO (CommonUtils.java:printTimeTakenMs) - createFile with fileId 3 took 57 ms.2015-09-09 20:00:17,999 INFO (WorkerClient.java:connect) - Trying to get local worker host : wangyue-um.local2015-09-09 20:00:18,013 INFO (WorkerClient.java:connect) - Connecting local worker @ wangyue-um.local/10.32.24.78:299982015-09-09 20:00:18,098 INFO (BlockOutStream.java:get) - Writing with local stream. tachyonFile: /default_tests_files/BasicFile_CACHE_THROUGH, blockIndex: 0, opType: CACHE_THROUGH2015-09-09 20:00:18,136 INFO (CommonUtils.java:createBlockPath) - Folder /home/cluster/apps/tachyon/ramdisk/tachyonworker/5 was created!2015-09-09 20:00:18,140 INFO (LocalBlockOutStream.java:) - /home/cluster/apps/tachyon/ramdisk/tachyonworker/5/3221225472 was created! tachyonFile: /default_tests_files/BasicFile_CACHE_THROUGH, blockIndex: 0, blockId: 3221225472, blockCapacityByte: 5368709122015-09-09 20:00:18,203 INFO (CommonUtils.java:printTimeTakenMs) - writeFile to file /default_tests_files/BasicFile_CACHE_THROUGH took 238 ms.2015-09-09 20:00:18,246 INFO (CommonUtils.java:printTimeTakenMs) - readFile file /default_tests_files/BasicFile_CACHE_THROUGH took 43 ms. Passed the test! 執(zhí)行 bin/tachyon runTests 循環(huán)測試123456789101112131415
wangyue@wangyue-um:/home/cluster/apps/tachyon$ bin/tachyon-stop.sh Killed 1 processes on wangyue-umKilled 1 processes on wangyue-umConnecting to localhost as wangyue...Killed 0 processes on wangyue-umConnection to localhost closed.執(zhí)行jps:21025 ZeppelinServer22214 Worker21987 Master32212 RemoteInterpreterServer23361 NameNode29603 RemoteInterpreterServer27469 Jps23493 DataNode27005 RemoteInterpreterServer 沒有tachyon相關(guān)進程123456789101112131415161718
感謝各位的閱讀！關(guān)于“Tachyon 0.7.1如何實現(xiàn)偽分布式集群安裝與測試 ”這篇文章就分享到這里了，希望以上內(nèi)容可以對大家有一定的幫助，讓大家可以學(xué)到更多知識，如果覺得文章不錯，可以把它分享出去讓更多的人看到吧！
重慶分公司
重慶分公司
