本次性能調(diào)優(yōu)項(xiàng)目中由于涉及的環(huán)節(jié)較多,最好能夠?qū)⑸森h(huán)境中的所有內(nèi)容進(jìn)行監(jiān)控,同時(shí)考慮最低開銷,這樣就從應(yīng)用服務(wù)器和數(shù)據(jù)庫服務(wù)器兩個(gè)服務(wù)器進(jìn)行,以nmon作為監(jiān)控基礎(chǔ)數(shù)據(jù),同時(shí)監(jiān)控JVM和數(shù)據(jù)庫告警和快照。
成都創(chuàng)新互聯(lián)公司提供高防主機(jī)、云服務(wù)器、香港服務(wù)器、綿陽主機(jī)托管等
所有監(jiān)控的內(nèi)容都是手段,只有從海量的監(jiān)控日志中得到規(guī)律性、有意義的數(shù)據(jù)才是性能優(yōu)化的基礎(chǔ)。有了數(shù)據(jù)就是對數(shù)據(jù)的分析,本文將首先介紹需要獲取的數(shù)據(jù),內(nèi)容也將是我從項(xiàng)目獲取的經(jīng)驗(yàn)。
基礎(chǔ)環(huán)境:兩臺數(shù)據(jù)庫服務(wù)器,做的數(shù)據(jù)庫集群。
項(xiàng)目中主要使用tongweb(老系統(tǒng)版本很低),監(jiān)控內(nèi)容類似如下:
...
"2018-01-11T02:25:55.663+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnCreated","10",
"2018-01-11T02:25:55.663+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnAcquired","111292",
"2018-01-11T02:25:55.663+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnNotSuccessfullyMatched","0",
"2018-01-11T02:26:25.670+0800","com.tongtech.tongweb:type=jvm,category=monitor,server=server","UpTime","222520621",
"2018-01-11T02:26:25.670+0800","com.tongtech.tongweb:type=jvm,category=monitor,server=server","HeapSize","2143485952",
"2018-01-11T02:26:25.671+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnUsed","0",
"2018-01-11T02:26:25.671+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnSuccessfullyMatched","0",
"2018-01-11T02:26:25.671+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","WaitQueueLength","0",
"2018-01-11T02:26:25.671+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnDestroyed","0",
"2018-01-11T02:26:25.671+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","ConnRequestWaitTime","4",
"2018-01-11T02:26:25.672+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnFailedValidation","0",
"2018-01-11T02:26:25.672+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnReleased","111292",
"2018-01-11T02:26:25.672+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnFree","10",
...
tongweb的監(jiān)控?cái)?shù)據(jù)獲取連接池狀態(tài)等信息,我們的方法是通過Excel宏的方式將日志內(nèi)轉(zhuǎn)換成可讀數(shù)據(jù),并進(jìn)行圖形分析。具體內(nèi)容將單獨(dú)說明。
JVM線程監(jiān)控說明
通過對tongweb的JVM監(jiān)控,可初步判定性能高峰時(shí)間點(diǎn)、連接池是否滿,同時(shí)進(jìn)一步判定連接高峰期的性能瓶頸是否出現(xiàn)在應(yīng)用上,這對今后的性能分析尤為重要,可將主要性能問題歸類,減少不必要的工作。
在Internet RFC標(biāo)準(zhǔn)中,Netstat的定義是: Netstat是在內(nèi)核中訪問網(wǎng)絡(luò)連接狀態(tài)及其相關(guān)信息的程序,它能提供TCP連接,TCP和UDP監(jiān)聽,進(jìn)程內(nèi)存管理的相關(guān)報(bào)告。
以下是在項(xiàng)目中獲取的日志摘取
...
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:427 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:427 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:58862 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2544 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:631 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:669 0.0.0.0:* LISTEN
...
作為本次性能優(yōu)化主要的分析手段,nmon起著尤為重要的作用,以下是wiki的解釋,有時(shí)間可以了解
nmon collects the following operating system statistics:
CPU and CPU threads Utilisation
CPU frequency for servers or virtual machines that can alter their clock rate
GPU stats including utilisation, MHz and temperatures
Physical and Virtual Memory use
Disk read & write and transfers
Disk Groups decided by the user
Swap and Paging
Network read & write and transfers
Local File-systems
Network File-system (NFS)
Top Processes by CPU use, Memory size and I/O rates
Kernel stats including Run Queue, context-switch, fork, Load Average & Uptime
Large and Huge memory pages
Virtual Machine stats (depending on the hardware) - useful for Linux running KVM to host virtual machines
Resources in the Server and virtual machine
總結(jié)其實(shí)nmon更像是系統(tǒng)性能開銷的快照,結(jié)合對nmon的分析工具可以很清楚的掌握系統(tǒng)的各項(xiàng)指標(biāo)。
下載分析工具
了解數(shù)據(jù)庫的告警日志也是掌握當(dāng)前性能的關(guān)鍵環(huán)節(jié)。
日志如下,如出現(xiàn)error可以針對具體情況進(jìn)行分析解決。
2018-01-11-00.36.36.090562+480 I13363168A459 LEVEL: Error
PID : 2228842 TID : 142490 PROC : db2sysc
INSTANCE: db2 NODE : 000 DB : TRADE
EDUID : 142490 EDUNAME: db2agent (**) 0
FUNCTION: DB2 UDB, Query Gateway, sqlqg_fedstp_hook, probe:40
MESSAGE : Unexpected error returned from outer RC=
DATA #1 : Hexdump, 4 bytes
0x07000007053F28D0 : 8126 0012 .&..
數(shù)據(jù)庫日志快照將作為主要分析依據(jù),在快照中可以分析數(shù)據(jù)庫時(shí)間的開銷情況,如下:
...
Number of automatic storage paths = 1
Automatic storage path = /db2data
Node number = 0
State = In Use
File system ID = 9223372079804448776
Storage path free space (bytes) = 69730709504
File system used space (bytes) = 139648946176
File system total space (bytes) = 209379655680
...
本文只是列出了分析的方法,具體操作有時(shí)間我會慢慢總結(jié)。
工具的利用固然重要,但是性能調(diào)優(yōu)并不是僅僅如此,必須步步為營
做好長期作戰(zhàn)的準(zhǔn)備。
另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)cdcxhl.cn,海內(nèi)外云服務(wù)器15元起步,三天無理由+7*72小時(shí)售后在線,公司持有idc許可證,提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國服務(wù)器、虛擬主機(jī)、免備案服務(wù)器”等云主機(jī)租用服務(wù)以及企業(yè)上云的綜合解決方案,具有“安全穩(wěn)定、簡單易用、服務(wù)可用性高、性價(jià)比高”等特點(diǎn)與優(yōu)勢,專為企業(yè)上云打造定制,能夠滿足用戶豐富、多元化的應(yīng)用場景需求。