排錯背景：在一次生產(chǎn)環(huán)境的部署過程中，配置文件中配置的訪問地址為集群的Service，配置好后發(fā)現(xiàn)服務(wù)不能正常訪問，遂啟動了一個busybox進(jìn)行測試，測試發(fā)現(xiàn)在busybox中，能通過coreDNS正常的解析到IP，然后去ping了一下service，發(fā)現(xiàn)不能ping通，ping clusterIP也不能ping通。

為萊西等地區(qū)用戶提供了全套網(wǎng)頁設(shè)計制作服務(wù)，及萊西網(wǎng)站建設(shè)行業(yè)解決方案。主營業(yè)務(wù)為網(wǎng)站建設(shè)、做網(wǎng)站、萊西網(wǎng)站設(shè)計，以傳統(tǒng)方式定制建設(shè)網(wǎng)站，并提供域名空間備案等一條龍服務(wù)，秉承以專業(yè)、用心的態(tài)度為用戶提供真誠的服務(wù)。我們深信只要達(dá)到每一位用戶的要求，就會得到認(rèn)可，從而選擇與我們長期合作。這樣，我們也可以走得更遠(yuǎn)！

排錯經(jīng)歷：首先排查了kube-proxy是否正常，發(fā)現(xiàn)啟動都是正常的，然后也重啟了，還是一樣ping不通，然后又排查了網(wǎng)絡(luò)插件，也重啟過flannel，依然沒有任何效果。后來想到自己的另一套k8s環(huán)境，是能正常ping通service的，就對比這兩套環(huán)境檢查配置，發(fā)現(xiàn)所有配置中只有kube-proxy的配置有一點差別，能ping通的環(huán)境kube-proxy使用了--proxy-mode=ipvs ,不能ping通的環(huán)境使用了默認(rèn)模式（iptables）。

iptables沒有具體設(shè)備響應(yīng)。

然后就是開始經(jīng)過多次測試，添加--proxy-mode=ipvs 后，清空node上防火墻規(guī)則，重啟kube-proxy后就能正常的ping通了。

在學(xué)習(xí)K8S的時候，自己一直比較忽略底層流量轉(zhuǎn)發(fā)，也即IPVS和iptables的相關(guān)知識，認(rèn)為不管哪種模式，只要能轉(zhuǎn)發(fā)訪問到pod就可以，不用太在意這些細(xì)節(jié)，以后還是得更加仔細(xì)才行。

補(bǔ)充：kubeadm 部署方式修改kube-proxy為 ipvs模式。

默認(rèn)情況下，我們部署的kube-proxy通過查看日志，能看到如下信息：Flag proxy-mode="" unknown，assuming iptables proxy

[root@k8s-master?~]#?kubectl?logs?-n?kube-system?kube-proxy-ppdb6?
W1013?06:55:35.773739???????1?proxier.go:513]?Failed?to?load?kernel?module?ip_vs?with?modprobe.?You?can?ignore?this?message?when?kube-proxy?is?running?inside?container?without?mounting?/lib/modules
W1013?06:55:35.868822???????1?proxier.go:513]?Failed?to?load?kernel?module?ip_vs_rr?with?modprobe.?You?can?ignore?this?message?when?kube-proxy?is?running?inside?container?without?mounting?/lib/modules
W1013?06:55:35.869786???????1?proxier.go:513]?Failed?to?load?kernel?module?ip_vs_wrr?with?modprobe.?You?can?ignore?this?message?when?kube-proxy?is?running?inside?container?without?mounting?/lib/modules
W1013?06:55:35.870800???????1?proxier.go:513]?Failed?to?load?kernel?module?ip_vs_sh?with?modprobe.?You?can?ignore?this?message?when?kube-proxy?is?running?inside?container?without?mounting?/lib/modules
W1013?06:55:35.876832???????1?server_others.go:249]?Flag?proxy-mode=""?unknown,?assuming?iptables?proxy
I1013?06:55:35.890892???????1?server_others.go:143]?Using?iptables?Proxier.
I1013?06:55:35.892136???????1?server.go:534]?Version:?v1.15.0
I1013?06:55:35.909025???????1?conntrack.go:100]?Set?sysctl?'net/netfilter/nf_conntrack_max'?to?131072
I1013?06:55:35.909053???????1?conntrack.go:52]?Setting?nf_conntrack_max?to?131072
I1013?06:55:35.919298???????1?conntrack.go:83]?Setting?conntrack?hashsize?to?32768
I1013?06:55:35.945969???????1?conntrack.go:100]?Set?sysctl?'net/netfilter/nf_conntrack_tcp_timeout_established'?to?86400
I1013?06:55:35.946044???????1?conntrack.go:100]?Set?sysctl?'net/netfilter/nf_conntrack_tcp_timeout_close_wait'?to?3600
I1013?06:55:35.946623???????1?config.go:96]?Starting?endpoints?config?controller
I1013?06:55:35.946660???????1?controller_utils.go:1029]?Waiting?for?caches?to?sync?for?endpoints?config?controller
I1013?06:55:35.946695???????1?config.go:187]?Starting?service?config?controller
I1013?06:55:35.946713???????1?controller_utils.go:1029]?Waiting?for?caches?to?sync?for?service?config?controller
I1013?06:55:36.047121???????1?controller_utils.go:1036]?Caches?are?synced?for?endpoints?config?controller
I1013?06:55:36.047195???????1?controller_utils.go:1036]?Caches?are?synced?for?service?config?controller

這里我們需要修改kube-proxy的配置文件,添加mode 為ipvs。

[root@k8s-master?~]#?kubectl?edit?cm?kube-proxy?-n?kube-system
...
ipvs:
??????excludeCIDRs:?null
??????minSyncPeriod:?0s
??????scheduler:?""
??????strictARP:?false
??????syncPeriod:?30s
????kind:?KubeProxyConfiguration
????metricsBindAddress:?127.0.0.1:10249
????mode:?"ipvs"
???...

ipvs模式需要注意的是要添加ip_vs相關(guān)模塊：

cat?>?/etc/sysconfig/modules/ipvs.modules?<
chmod?755?/etc/sysconfig/modules/ipvs.modules?&&?bash?/etc/sysconfig/modules/ipvs.modules?&&?lsmod?|?grep?-e?ip_vs?-e?nf_conntrack_ipv4
重啟kube-proxy 的pod
[root@k8s-master?~]#?kubectl?get?pod?-n?kube-system?|?grep?kube-proxy?|awk?'{system("kubectl?delete?pod?"$1"?-n?kube-system")}'
pod?"kube-proxy-62gvr"?deleted
pod?"kube-proxy-n2rml"?deleted
pod?"kube-proxy-ppdb6"?deleted
pod?"kube-proxy-rr9cg"?deleted
在pod重啟后再查看日志，發(fā)現(xiàn)模式已經(jīng)變?yōu)閕pvs了。
[root@k8s-master?~]#?kubectl?get?pod?-n?kube-system?|grep?kube-proxy
kube-proxy-cbm8p?????????????????????1/1?????Running???0??????????85s
kube-proxy-d97pn?????????????????????1/1?????Running???0??????????83s
kube-proxy-gmq6s?????????????????????1/1?????Running???0??????????76s
kube-proxy-x6tcg?????????????????????1/1?????Running???0??????????81s
[root@k8s-master?~]#?kubectl?logs?-n?kube-system?kube-proxy-cbm8p?
I1013?07:34:38.685794???????1?server_others.go:170]?Using?ipvs?Proxier.
W1013?07:34:38.686066???????1?proxier.go:401]?IPVS?scheduler?not?specified,?use?rr?by?default
I1013?07:34:38.687224???????1?server.go:534]?Version:?v1.15.0
I1013?07:34:38.692777???????1?conntrack.go:52]?Setting?nf_conntrack_max?to?131072
I1013?07:34:38.693378???????1?config.go:187]?Starting?service?config?controller
I1013?07:34:38.693391???????1?controller_utils.go:1029]?Waiting?for?caches?to?sync?for?service?config?controller
I1013?07:34:38.693406???????1?config.go:96]?Starting?endpoints?config?controller
I1013?07:34:38.693411???????1?controller_utils.go:1029]?Waiting?for?caches?to?sync?for?endpoints?config?controller
I1013?07:34:38.793684???????1?controller_utils.go:1036]?Caches?are?synced?for?endpoints?config?controller
I1013?07:34:38.793688???????1?controller_utils.go:1036]?Caches?are?synced?for?service?config?controller
再次測試ping service
[root@k8s-master?~]#?kubectl?exec?-it?dns-test?sh
/?#?ping?nginx-service
PING?nginx-service?(10.1.58.65):?56?data?bytes
64?bytes?from?10.1.58.65:?seq=0?ttl=64?time=0.033?ms
64?bytes?from?10.1.58.65:?seq=1?ttl=64?time=0.069?ms
64?bytes?from?10.1.58.65:?seq=2?ttl=64?time=0.094?ms
64?bytes?from?10.1.58.65:?seq=3?ttl=64?time=0.057?ms
^C
---?nginx-service?ping?statistics?---
4?packets?transmitted,?4?packets?received,?0%?packet?loss
round-trip?min/avg/max?=?0.033/0.063/0.094?ms
            
            
                        

            分享名稱：【K8S排錯】在集群的POD內(nèi)不能訪問clusterIP和service            

            鏈接URL：http://weahome.cn/article/jopcpd.html