這篇文章主要為大家展示了“nagios小白故障的示例分析”，內(nèi)容簡而易懂，條理清晰，希望能夠幫助大家解決疑惑，下面讓小編帶領大家一起研究并學習一下“nagios小白故障的示例分析”這篇文章吧。
運行了一年多的nagios突然出問題，網(wǎng)站可以打開，但是所有host、service都看不到狀態(tài)。提示nagios maybe not running
第一反應是nagios掛了
重啟nagios服務
[root@nagios-server ~]#/etc/init.d/nagiosstop Stopping nagios: done. [root@nagios-server ~]#/etc/init.d/nagiosstart Starting nagios: done.
刷新網(wǎng)頁還是一樣，清理瀏覽器緩存，重開瀏覽器還是看不到。
難道配置文件有錯誤？近期沒改什么啊。
檢查配置文件
[root@nagios-server ~]#/etc/init.d/nagioscheckconfig
沒有error
服務器重啟
[root@nagios-server ~]#reboot
還是不行
看看nagios錯誤日志吧
[root@nagios-server ~]#cat/usr/local/nagios/nagios.log
日志空空如也！
奇怪了，那再看看系統(tǒng)日志
[root@nagios-server ~]#cat/var/log/messages
滿屏都是這種記錄
The check of service 'MEM' on host 'xxx'looks like it was orphaned (results never came back). I'm scheduling an immediate check of theservice...
看的頭疼，于是做了一個錯誤的決定，這個決定導致我喪失了第一時間發(fā)現(xiàn)問題根源的機會!
對，我把系統(tǒng)日志清空了
[root@nagios-server ~]#>/var/log/messages
整個世界清凈了，再重啟nagios服務，看日志會記錄什么錯誤
[root@nagios-server ~]#/etc/init.d/nagiosstop [root@nagios-server ~]#/etc/init.d/nagiosstart [root@nagios-server etc]# tail -30/var/log/messages Dec 30 17:37:38 nagios-server nagios: Nagios3.5.1 starting... (PID=50752) Dec 30 17:37:38 nagios-server nagios:Local time is Fri Dec 30 17:37:38 CST 2016 Dec 30 17:37:38 nagios-server nagios:LOG VERSION: 2.0 Dec 30 17:37:38 nagios-server nagios:Warning: Host 'xxx' has no services associated with it! Dec 30 17:37:38 nagios-server nagios:Finished daemonizing... (New PID=50753) Dec 30 17:39:21 nagios-server nagios:Nagios 3.5.1 starting... (PID=51146) Dec 30 17:39:21 nagios-server nagios:Local time is Fri Dec 30 17:39:21 CST 2016 Dec 30 17:39:21 nagios-server nagios:LOG VERSION: 2.0 Dec 30 17:39:21 nagios-server nagios:Warning: Host 'xxx' has no services associated with it! Dec 30 17:39:21 nagios-server nagios:Lockfile '/usr/local/nagios/var/nagios.lock' looks like its already held byanother instance of Nagios (PID 50753). Bailing out...
嗯，發(fā)現(xiàn)異常了，nagios.lock被其他進程占用了。
4.根據(jù)這個異常提示打開度娘搜索，找了半天發(fā)現(xiàn)有個帖子說的情況和我很像。說是刪除nagios.log、objects.cache、retention.dat，再重啟nagios服務
[root@nagios-server ~]# cd/usr/local/nagios/var [root@nagios-server var]# ll total 36 drwxrwxr-x. 2 nagios nagios 24576 Dec 3000:00 archives -rw-r--r-- 1 nagios nagios 0 Dec 30 17:27 nagios.lock -rw-rw-r-- 1 nagios nagios 0 Dec 30 17:27 nagios.log -rw-r--r--. 1 nagios nagios 0 Dec 30 17:27 objects.cache -rw------- 1 nagios nagios 0 Dec 30 17:27 retention.dat drwxrwsr-x. 2 nagios nagcmd 4096 Dec 30 17:27 rw drwxr-xr-x. 4 root root 4096 Oct 22 2015 spool -rw-rw-r-- 1 nagios nagios 0 Dec 30 17:27 status.dat
怎么nagios.log、objects.cache、retention.dat文件大小全部為0呢？不管它，按照帖子操作試試。
到了這里，老鳥應該發(fā)現(xiàn)問題在哪了。但我忽略了這個重要細節(jié)，再一次喪失了發(fā)現(xiàn)問題根源的機會！
[root@nagios-server var]# rm -fobjects.cache retention.dat status.dat [root@nagios-server var]/etc/init.d/nagiosstop [root@nagios-server var]/etc/init.d/nagiosstart
然并卵，故障依舊。這個時候有點方了，不知道如何下手去排查。查看nagios.cfg、nrpe.cfg都正常。
5.反復重啟nagios，發(fā)現(xiàn)另外一個異常點，nagios服務“貌似沒啟動”，注意這里的引號。
[root@nagios-server etc]#/etc/init.d/nagios start Starting nagios: done. [root@nagios-server etc]#/etc/init.d/nagios status nagios is not running
就是說雖然啟動命令執(zhí)行成功，但是查詢nagios狀態(tài)去提示沒運行，這就奇怪了，難道nagios程序有問題了？
繼續(xù)看日志。從日志中能看出重復啟動了nagios，可能有多實例(instance)運行
[root@nagios-server etc]# tail -30/var/log/messages Dec 30 17:37:38 nagios-server nagios:Nagios 3.5.1 starting... (PID=50752) Dec 30 17:37:38 nagios-server nagios:Local time is Fri Dec 30 17:37:38 CST 2016 Dec 30 17:37:38 nagios-server nagios:LOG VERSION: 2.0 Dec 30 17:37:38 nagios-server nagios:Warning: Host 'xxx' has no services associated with it! Dec 30 17:37:38 nagios-server nagios:Finished daemonizing... (New PID=50753) Dec 30 17:39:21 nagios-server nagios:Nagios 3.5.1 starting... (PID=51146) Dec 30 17:39:21 nagios-server nagios:Local time is Fri Dec 30 17:39:21 CST 2016 Dec 30 17:39:21 nagios-server nagios:LOG VERSION: 2.0 Dec 30 17:39:21 nagios-server nagios:Warning: Host 'xxx' has no services associated with it! Dec 30 17:39:21 nagios-server nagios:Lockfile '/usr/local/nagios/var/nagios.lock' looks like its already held byanother instance of Nagios (PID 50753). Bailing out...
看一下nagios.lock的內(nèi)容吧
[root@nagios-server etc]#echo /usr/local/nagios/var/nagios.lock
文件是空的。
把nagios進程干掉
[root@nagios-server ~]# kill -9 50753 [root@nagios-server ~]# kill -9 50753 -bash: kill: (50753) - No such process
找不到之前的日志了，反正N次重啟，發(fā)現(xiàn)一個問題：其實nagios進程一直是啟動的，只是它啟動的時候，沒有把PID寫入/usr/local/nagios/var/nagios.lock里面！
手動把PID寫進去
[root@nagios-server ~]# echo 44336 >/usr/local/nagios/var/nagios.lock [root@nagios-server ~]# /etc/init.d/nagiosstatus nagios (pid 44336) is running...
原來nagios啟動腳本會去檢查nagios.lock里面的內(nèi)容，如果為空，就返回nagios is not running，但實際nagios進程是正常啟動了的。
所以反復重啟之后，系統(tǒng)日志里面會有這句話“nagios-server nagios: Lockfile '/usr/local/nagios/var/nagios.lock'looks like its already held by another instance of Nagios (PID 50753). Bailing out...”
發(fā)現(xiàn)這個問題之后，把所有PID殺掉，重啟nagios。果然里面沒有這個nagios.lock錯誤。取代的是滿屏的“The check of service 'MEM' on host 'xxx' looks like it was orphaned(results never came back). I'mscheduling an immediate check of the service...”
改用谷歌，以這個關鍵字進行搜索nagios The check of service looks like it was orphaned (resultsnever came back)，在搜索的結果里面看了N多帖子，終于看到一句，去看看磁盤是不是沒空間了。嗯，看看磁盤。
[root@nagios-server spool]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 8.7G 8.3G 1.1M 100% / tmpfs 935M 0 935M 0% /dev/shm /dev/sda1 194M 34M 151M 19% /boot [root@nagios-server spool]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda3 577088 104974 472114 19% / tmpfs 239320 1 239319 1% /dev/shm /dev/sda1 51200 39 51161 1% /boot
原因找到了！??！磁盤可用空間只有1.1M，之前的故障點都有合理的解釋。為什么nagios.log日志為0;為什么nagios.lock是空的。全部都是磁盤沒剩余空間的原因！??！
[root@nagios-server nagios]# du -a /var| sort -n -r | head -n 10
6044444 /var 5946296 /var/log 5882592 /var/log/httpd 3127308/var/log/httpd/error_log-20161225 2659808 /var/log/httpd/error_log 66488 /var/lib 54616 /var/lib/rpm 45952 /var/log/httpd/access_log-20161225 43136 /var/lib/rpm/Packages 31512 /var/log/httpd/access_log [root@nagios-server nagios]# du -sh /var 5.8G /var [root@nagios-server nagios]# cd/var/log/httpd/ [root@nagios-server httpd]# du -hsx * |sort -rh | head -10 3.0G error_log-20161225 2.6G error_log 45M access_log-20161225 31M access_log 7.2M access_log-20161211 7.1M access_log-20161218 2.0M access_log-20161204 532K ssl_request_log-20160630 460K ssl_access_log-20160630 140K ssl_request_log-20161024
教訓：
回看之前的/var/log/messages，其實里面隱藏著非常重要的消息！
Dec 28 12:30:01 nagios-server auditd[1032]:Audit daemon is low on disk space for logging
Dec 28 13:30:01 nagios-server auditd[1032]:Audit daemon is suspending logging due to low disk space.
Dec 28 14:54:30 nagios-server nagios:Warning: The check of host 'xxx' looks like it was orphaned (results never cameback). I'm scheduling an immediate checkof the host...
Dec 28 14:54:30 nagios-server nagios:Warning: The check of host 'xxx' looks like it was orphaned (results never cameback). I'm scheduling an immediate checkof the host...
以上是“nagios小白故障的示例分析”這篇文章的所有內(nèi)容，感謝各位的閱讀！相信大家都有了一定的了解，希望分享的內(nèi)容對大家有所幫助，如果還想學習更多知識，歡迎關注創(chuàng)新互聯(lián)行業(yè)資訊頻道！
另外有需要云服務器可以了解下創(chuàng)新互聯(lián)scvps.cn，海內(nèi)外云服務器15元起步，三天無理由+7*72小時售后在線，公司持有idc許可證，提供“云服務器、裸金屬服務器、高防服務器、香港服務器、美國服務器、虛擬主機、免備案服務器”等云主機租用服務以及企業(yè)上云的綜合解決方案，具有“安全穩(wěn)定、簡單易用、服務可用性高、性價比高”等特點與優(yōu)勢，專為企業(yè)上云打造定制，能夠滿足用戶豐富、多元化的應用場景需求。
重慶分公司
重慶分公司
