這篇文章給大家分享的是有關(guān)redis內(nèi)存詭異增長如何排查問題的內(nèi)容。小編覺得挺實(shí)用的，因此分享給大家做個(gè)參考，一起跟隨小編過來看看吧。
創(chuàng)新互聯(lián)是一家以重慶網(wǎng)站建設(shè)公司、網(wǎng)頁設(shè)計(jì)、品牌設(shè)計(jì)、軟件運(yùn)維、成都網(wǎng)站營銷、小程序App開發(fā)等移動(dòng)開發(fā)為一體互聯(lián)網(wǎng)公司。已累計(jì)為酒店設(shè)計(jì)等眾行業(yè)中小客戶提供優(yōu)質(zhì)的互聯(lián)網(wǎng)建站和軟件開發(fā)服務(wù)。
一、現(xiàn)象
實(shí)例名：r-bp1cxxxxxxxxxd04(主從)
問題：一分鐘內(nèi)存上漲了2G，如下圖所示：
鍵值規(guī)模：6000萬左右
內(nèi)存一分鐘增長2G.png
二、Redis內(nèi)存分析
1. 內(nèi)存組成
上圖中的內(nèi)存統(tǒng)計(jì)的是Redis的info memory命令中的used_memory屬性，例如：
redis>infomemory#Memoryused_memory:9195978072used_memory_human:8.56Gused_memory_rss:9358786560used_memory_peak:10190212744used_memory_peak_human:9.49Gused_memory_lua:38912mem_fragmentation_ratio:1.02mem_allocator:jemalloc-3.6.0
每個(gè)屬性的詳細(xì)說明
| 屬性名 | 屬性說明 |
|---|---|
| used_memory | Redis 分配器分配的內(nèi)存量，也就是實(shí)際存儲(chǔ)數(shù)據(jù)的內(nèi)存總量 |
| used_memory_human | 以可讀格式返回 Redis 使用的內(nèi)存總量 |
| used_memory_rss | 從操作系統(tǒng)的角度，Redis進(jìn)程占用的總物理內(nèi)存 |
| used_memory_peak | 內(nèi)存分配器分配的最大內(nèi)存，代表used_memory的歷史峰值 |
| used_memory_peak_human | 以可讀的格式顯示內(nèi)存消耗峰值 |
| used_memory_lua | Lua引擎所消耗的內(nèi)存 |
| mem_fragmentation_ratio | used_memory_rss /used_memory比值，表示內(nèi)存碎片率 |
| mem_allocator | Redis 所使用的內(nèi)存分配器。默認(rèn): jemalloc |
計(jì)算公式如下：
used_memory = 自身內(nèi)存+對(duì)象內(nèi)存+緩沖內(nèi)存+lua內(nèi)存used_rss = used_memory + 內(nèi)存碎片
如下圖所示：
2. 內(nèi)存分析
(1) 自身內(nèi)存：一個(gè)空的Redis占用很小，可以忽略不計(jì)
(2) kv內(nèi)存：key對(duì)象 + value對(duì)象
(3) 緩沖區(qū)：客戶端緩沖區(qū)(普通 + slave偽裝 + pubsub)以及aof緩沖區(qū)(比較固定，一般沒問題)
(4) Lua：Lua引擎所消耗的內(nèi)存
3. 內(nèi)存突增常見問題
(1) kv內(nèi)存：bigkey、大量寫入
(2) 客戶端緩沖區(qū)：一般常見的有普通客戶端緩沖區(qū)(例如monitor命令)或者pubsub客戶端緩沖區(qū)
三、問題排查
(1) bigkey ? 經(jīng)掃描未發(fā)現(xiàn)bigkey
Sampled 67234427 keys in the keyspace! Total key length in bytes is 1574032382 (avg len 23.41) Biggest string found 'CCARD_DEVICE_CARD_REF_MAP_KEY_016817000004209' has 20862 bytes Biggest list found 'CCARD_VALID_DEVICE_TRAIN_QUEUE_KEY' has 51 items Biggest hash found 'CCARD_VALID_DEVICE_TRAIN_MAP_KEY' has 51 fields 67234359 strings with 71767890 bytes (100.00% of keys, avg size 1.07) 67 lists with 151 items (00.00% of keys, avg size 2.25) 0 sets with 0 members (00.00% of keys, avg size 0.00) 1 hashs with 51 fields (00.00% of keys, avg size 51.00) 0 zsets with 0 members (00.00% of keys, avg size 0.00)
(2) 鍵值個(gè)數(shù)增加？未發(fā)現(xiàn)鍵值有明顯變化
.png)
(3) 客戶端緩沖區(qū)
由于內(nèi)存增上去后，長時(shí)間沒下落，如果是因?yàn)榫彌_區(qū)問題，會(huì)從info clients找到明顯問題，執(zhí)行后發(fā)現(xiàn)：
redis> info clients # Clients connected_clients:43 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 admin_clients:6 rejected_vpc_conn_count:0 close_idle_unknown_conn_count:0
執(zhí)行client中也沒有明顯的omem大于0的情況
id=80207addr=10.xx.0.4:63920fd=46name=age=624idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80215addr=10.xx.0.23:43489fd=36name=age=591idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80366addr=10.xx.0.8:59785fd=18name=age=84idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=delread=0write=0type=user
id=80356addr=10.xx.0.33:32117fd=13name=age=114idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80064addr=10.xx.59.4:53446fd=38name=age=1070idle=1070flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=80276addr=10.xx.0.23:48511fd=8name=age=387idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80188addr=10.xx.0.33:16265fd=42name=age=681idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80326addr=10.xx.0.32:59779fd=16name=age=209idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80065addr=10.xx.59.4:53447fd=45name=age=1070idle=1070flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=79936addr=10.xx.0.22:10607fd=30name=age=1480idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80174addr=10.xx.0.5:60914fd=6name=age=722idle=2flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80300addr=10.xx.0.22:22757fd=48name=age=298idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80037addr=10.xx.0.5:55189fd=15name=age=1143idle=2flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80330addr=10.xx.0.8:48533fd=17name=age=199idle=10flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=79896addr=10.xx.0.30:26814fd=11name=age=1616idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80299addr=10.xx.0.24:11227fd=44name=age=303idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80086addr=10.xx.0.32:52526fd=40name=age=1002idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80202addr=10.xx.0.33:16658fd=26name=age=636idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80256addr=10.xx.0.24:60496fd=19name=age=448idle=2flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=79908addr=10.xx.0.29:18975fd=12name=age=1583idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80365addr=10.xx.0.29:46429fd=14name=age=85idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=79869addr=10.xx.27.4:48455fd=35name=age=1700idle=1700flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=80334addr=10.xx.0.23:50012fd=39name=age=189idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80041addr=10.xx.0.32:51107fd=33name=age=1132idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=79992addr=10.xx.0.22:12068fd=28name=age=1289idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80251addr=10.xx.0.30:44213fd=23name=age=468idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80006addr=10.xx.0.2:45895fd=31name=age=1242idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80321addr=10.xx.0.30:48048fd=5name=age=224idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80381addr=10.xx.0.8:13360fd=22name=age=24idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=delread=0write=0type=user
id=80200addr=10.xx.0.24:59183fd=24name=age=640idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80113addr=10.xx.0.2:52492fd=21name=age=915idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=174addr=11.216.117.242:53027fd=9name=age=281390idle=0flags=S db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=replconf read=0write=0type=admin
id=79991addr=10.xx.0.4:48412fd=25name=age=1296idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80301addr=127.0.0.1:47869fd=49name=age=291idle=261flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=strlen read=0write=0type=admin
id=80047addr=10.xx.59.4:53184fd=41name=age=1114idle=1114flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=80236addr=10.xx.0.5:62546fd=47name=age=516idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80364addr=10.xx.0.4:18794fd=7name=age=85idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80175addr=10.xx.0.4:62245fd=29name=age=718idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80336addr=10.xx.0.29:45701fd=50name=age=180idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80050addr=10.xx.59.4:53188fd=43name=age=1114idle=1114flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=79765addr=10.xx.0.2:33832fd=37name=age=2027idle=177flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=info read=0write=0type=user
id=80170addr=10.xx.0.2:57853fd=20name=age=728idle=24flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80390addr=127.0.0.1:49449fd=27name=age=0idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=client read=0write=0type=admin
四、揪出元兇
常用的幾招都用了，還是不行，同事@徑遠(yuǎn)幫忙一起分析，懷疑是不是因?yàn)镽edis的kv哈希表做了 rehash。
1. Redis的kv存儲(chǔ)結(jié)構(gòu)
如下圖所示，Redis的所有kv保存在dict中，其中ht對(duì)應(yīng)兩個(gè)哈希表ht[0]和ht[1]，平時(shí)一個(gè)空閑，一個(gè)用于存儲(chǔ)數(shù)據(jù)，只有當(dāng)需要rehash時(shí)，ht[1]才會(huì)用到。
2. Redis的字典rehash
為了保證哈希表的負(fù)載，當(dāng)哈希表的元素個(gè)數(shù)等于哈希表槽數(shù)時(shí)候，會(huì)進(jìn)行rehash擴(kuò)容。擴(kuò)容后h[1]的容量等于第一個(gè)大于等于ht[0].size*2的2n，例如hash表的初始化容量是4，那么下一次擴(kuò)容就是8，以此類推。
3. 測(cè)試
(1) 測(cè)試方法
先批量寫入到rehash閾值附近，然后在逐條去寫，觀察內(nèi)存變化
重慶分公司
重慶分公司
// 為每個(gè)鍵設(shè)置1天過期時(shí)間
int expireTime = 60 * 60 * 24;
// rehash閾值 - 50為了方便觀察rehash內(nèi)存變化
int rehashThreshold = (int) Math.pow(2, 25) - 50;
// 1.批量寫入：pipeline批量寫入，由于是本機(jī)測(cè)試，這里用10000，實(shí)際生產(chǎn)不要這么用
Pipeline pipeline = jedis.pipelined();
pipeline = jedis.pipelined();
for (int i = 0; i < rehashThreshold; i++) {
pipeline.setex(String.valueOf(i), expireTime, String.valueOf(i));
if (i % 10000 == 0) {
pipeline.sync();
}
}
pipeline.sync();
// 2.等待寫增量
TimeUnit.SECONDS.sleep(5);
for (int i = rehashThreshold; i < rehashThreshold + 200; i++) {
jedis.setex(String.valueOf(i), expireTime, String.valueOf(i));
TimeUnit.SECONDS.sleep(1);
}(2) 開始測(cè)試
(a) 當(dāng)閾值=215=32768，從下面可以看出到key的個(gè)數(shù)為32769時(shí)，內(nèi)存漲了一些，但是還不明顯。
keys mem clients blocked requests connections32766 4.69M 3 0 32797 (+2) 4
32767 4.69M 3 0 32799 (+2) 4
32768 4.69M 3 0 32801 (+2) 4
32769 5.44M 3 0 32803 (+2) 4
(b) 當(dāng)閾值=220=1048576，從下面可以看出到key的個(gè)數(shù)為1048577時(shí)，內(nèi)存漲了32M。因?yàn)閞ehash會(huì)擴(kuò)容，所以新的哈希表中的槽位變?yōu)榱?21 * 2（因?yàn)槊總€(gè)key都設(shè)置了過期時(shí)間，expires表），指針為8個(gè)字節(jié)，221 ? 2 ? 8 = 225 = 32MB。
keys mem clients blocked requests connections1048574 128.69M 3 0 3364129 (+2) 16
1048575 128.69M 3 0 3364131 (+2) 16
1048576 128.69M 3 0 3364133 (+2) 16
1048577 160.69M 3 0 3364135 (+2) 16
1048578 160.69M 3 0 3364137 (+2) 16
(c) 當(dāng)閾值=226=67108864，從下面可以看出到key的個(gè)數(shù)為67108865時(shí)，內(nèi)存漲了2GB。因?yàn)閞ehash會(huì)擴(kuò)容，所以新的哈希表中的槽位變?yōu)榱?27 * 2（因?yàn)槊總€(gè)key都設(shè)置了過期時(shí)間，expires表），指針為8個(gè)字節(jié)，227 ? 2 ? 8 = 231 = 2GB。
keys mem clients blocked requests connections67108862 9.70G 3 0 70473683 (+2) 18
67108863 9.70G 3 0 70473685 (+2) 18
67108864 9.70G 3 0 70473687 (+2) 18
67108865 11.70G 3 0 70473689 (+2) 18
67108866 11.70G 3 0 70473691 (+2) 18
67108867 11.70G 3 0 70473693 (+2) 18
回過來看r-bp1c15fd9b142d04的key和內(nèi)存變化圖，可以發(fā)現(xiàn)上面的規(guī)則是正確的：
4. 后續(xù)觀察
17點(diǎn)時(shí)，rehash結(jié)束，內(nèi)存降了增加的2G的一半。
感謝各位的閱讀！關(guān)于“Redis內(nèi)存詭異增長如何排查問題”這篇文章就分享到這里了，希望以上內(nèi)容可以對(duì)大家有一定的幫助，讓大家可以學(xué)到更多知識(shí)，如果覺得文章不錯(cuò)，可以把它分享出去讓更多的人看到吧！
