云計算 K8S使用dashboard管理集群

創(chuàng)新互聯(lián)公司長期為1000+客戶提供的網(wǎng)站建設(shè)服務(wù)，團隊從業(yè)經(jīng)驗10年，關(guān)注不同地域、不同群體，并針對不同對象提供差異化的產(chǎn)品和服務(wù)；打造開放共贏平臺，與合作伙伴共同營造健康的互聯(lián)網(wǎng)生態(tài)環(huán)境。為康馬企業(yè)提供專業(yè)的網(wǎng)站設(shè)計、成都網(wǎng)站制作，康馬網(wǎng)站改版等技術(shù)服務(wù)。擁有10年豐富建站經(jīng)驗和眾多成功案例,為您定制開發(fā)。

今年3月份在公司的內(nèi)部k8s培訓(xùn)會上，開發(fā)同事表示使用dashboard的可以滿足日常開發(fā)需求，例如查看pod的日志，執(zhí)行exec指令，查看pod的運行狀態(tài)等，但對basic認(rèn)證的權(quán)限控制表示擔(dān)憂。
之前介紹過在1.5.2版本上部署dashboard服務(wù)，在1.9.1版本離線部署中，也介紹過dashboard服務(wù)的RBAC配置和使用技巧。因此本文將在前文基礎(chǔ)上完善Heapster的整合與利用token對用戶權(quán)限進行控制。
dashboard的特點主要如下：
1、能夠直觀的看到rc、deployment、pod、services等k8s組件的運行情況和日志信息。
2、結(jié)合heapster和influxdb后，dashboard的監(jiān)控圖表上可以看到pod的cpu和內(nèi)存消耗情況。

Heapster介紹

1、Heapster是容器集群監(jiān)控和性能分析工具，支持Kubernetes和CoreOS。
2、K8S集群的HPA功能的實現(xiàn)就依賴于這些metric數(shù)據(jù)，HPA將Heapster作為Resource Metrics API，向其獲取metric。
3、Kubernetes有個cAdvisor監(jiān)控（在1.9版本里面，cAdvisor已經(jīng)和kubelet整合在一起）。
在每個kubernetes Node上都會運行cAdvisor，它會收集本機以及容器的監(jiān)控數(shù)據(jù)(cpu,memory,filesystem,network,uptime)。Heapster是一個收集者，Heapster可以收集Node節(jié)點上的cAdvisor數(shù)據(jù)，將每個Node上的cAdvisor的數(shù)據(jù)進行匯總，還可以按照kubernetes的資源類型來集合資源，比如Pod、Namespace，可以分別獲取它們的CPU、內(nèi)存、網(wǎng)絡(luò)和磁盤的metric。默認(rèn)的metric數(shù)據(jù)聚合時間間隔是1分鐘。還可以把數(shù)據(jù)導(dǎo)入到第三方工具(如InfluxDB)。

Influxdb數(shù)據(jù)庫介紹

2、Influxdb數(shù)據(jù)庫的相關(guān)知識介紹，可參考文檔：https://www.jianshu.com/p/d2935e99006e
2、如果對Heapster收集到的metric數(shù)據(jù)沒有持久化的需求，可以不配置Influxdb數(shù)據(jù)庫
3、本文Influxdb數(shù)據(jù)庫的存儲采用emptydir的方式實現(xiàn)，實際使用過程中，可以選擇吧Influxdb數(shù)據(jù)庫部署在k8s集群外部，或者使用其他存儲方案。
4、如果有需要的話，還可以集成一個grafana做web展示。Grafana配置可參考文檔：https://blog.51cto.com/ylw6006/2084403

一、獲取相關(guān)鏡像

需要科學(xué)上網(wǎng)方式獲取到dashboard相關(guān)的鏡像文件，倉庫可納入本地倉庫統(tǒng)一管理

#cat/etc/systemd/system/docker.service.d/http-proxy.conf[Service]
Environment="HTTP_PROXY=http://192.168.115.2:1080"#systemctldaemon-reload#systemctlrestartdocker#dockerpullk8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3#dockerpullk8s.gcr.io/heapster-influxdb-amd64:v1.3.3#dockerpullk8s.gcr.io/heapster-amd64:v1.4.2

二、準(zhǔn)備配置文件

1、k8s-dashborad-sa.yaml文件，secrct和serviceaccount配置

#catk8s-dashborad-sa.yaml#-------------------DashboardSecret-------------------#apiVersion:v1kind:Secretmetadata:
labels:
k8s-app:kubernetes-dashboard
name:kubernetes-dashboard-certs
namespace:kube-systemtype:Opaque---#-------------------DashboardServiceAccount-------------------#apiVersion:v1kind:ServiceAccountmetadata:
labels:
k8s-app:kubernetes-dashboard
name:kubernetes-dashboard
namespace:kube-system

2、k8s-dashborad-rbac.yaml文件，配置 Role和Role Binding

#catk8s-dashborad-rbac.yaml#-------------------DashboardRole&RoleBinding-------------------#kind:RoleapiVersion:rbac.authorization.k8s.io/v1metadata:
name:kubernetes-dashboard-minimal
namespace:kube-systemrules:
#AllowDashboardtocreate'kubernetes-dashboard-key-holder'secret.-apiGroups:[""]
resources:["secrets"]
verbs:["create"]#AllowDashboardtocreate'kubernetes-dashboard-settings'configmap.-apiGroups:[""]
resources:["configmaps"]
verbs:["create"]#AllowDashboardtoget,updateanddeleteDashboardexclusivesecrets.-apiGroups:[""]
resources:["secrets"]
resourceNames:["kubernetes-dashboard-key-holder","kubernetes-dashboard-certs"]
verbs:["get","update","delete"]#AllowDashboardtogetandupdate'kubernetes-dashboard-settings'configmap.-apiGroups:[""]
resources:["configmaps"]
resourceNames:["kubernetes-dashboard-settings"]
verbs:["get","update"]#AllowDashboardtogetmetricsfromheapster.-apiGroups:[""]
resources:["services"]
resourceNames:["heapster"]
verbs:["proxy"]
-apiGroups:[""]
resources:["services/proxy"]
resourceNames:["heapster","http:heapster:","https:heapster:"]
verbs:["get"]

---apiVersion:rbac.authorization.k8s.io/v1kind:RoleBindingmetadata:
name:kubernetes-dashboard-minimal
namespace:kube-systemroleRef:
apiGroup:rbac.authorization.k8s.io
kind:Role
name:kubernetes-dashboard-minimalsubjects:-kind:ServiceAccount
name:kubernetes-dashboard
namespace:kube-system

3、k8s-dashborad-deployment.yaml配置文件，定義創(chuàng)建pod的模板和副本數(shù)

#catk8s-dashborad-deployment.yaml#-------------------DashboardDeployment-------------------#kind:DeploymentapiVersion:apps/v1beta2metadata:
labels:
k8s-app:kubernetes-dashboard
name:kubernetes-dashboard
namespace:kube-systemspec:
replicas:1
revisionHistoryLimit:10
selector:
matchLabels:
k8s-app:kubernetes-dashboard
template:
metadata:
labels:
k8s-app:kubernetes-dashboard
spec:
containers:
-name:kubernetes-dashboard
image:k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
ports:
-containerPort:8443
protocol:TCP
args:
---auto-generate-certificates#UncommentthefollowinglinetomanuallyspecifyKubernetesAPIserverHost
#Ifnotspecified,DashboardwillattempttoautodiscovertheAPIserverandconnect
#toit.Uncommentonlyifthedefaultdoesnotwork.
#---apiserver-host=http://my-address:port
volumeMounts:
-name:kubernetes-dashboard-certs
mountPath:/certs#Createon-diskvolumetostoreexeclogs
-mountPath:/tmp
name:tmp-volume
livenessProbe:
httpGet:
scheme:HTTPS
path:/
port:8443
initialDelaySeconds:30
timeoutSeconds:30
volumes:
-name:kubernetes-dashboard-certs
secret:
secretName:kubernetes-dashboard-certs
-name:tmp-volume
emptyDir:{}
serviceAccountName:kubernetes-dashboard#CommentthefollowingtolerationsifDashboardmustnotbedeployedonmaster
tolerations:
-key:node-role.kubernetes.io/master
effect:NoSchedule

4、 k8s-dashborad-service.yaml配置文件，定義service

#catk8s-dashborad-service.yaml#-------------------DashboardService-------------------#kind:ServiceapiVersion:v1metadata:
labels:
k8s-app:kubernetes-dashboard
name:kubernetes-dashboard
namespace:kube-systemspec:
ports:
-port:443
targetPort:8443
nodePort:8490
type:NodePort
selector:
k8s-app:kubernetes-dashboard

三、通過配置文件創(chuàng)建dashboard

#kubectlcreate-f.#kubectlgetpod,deployment,svc-nkube-system

四、配置使用basic認(rèn)證方式

默認(rèn)情況下只支持kubeconfig和令牌認(rèn)證

#echo'admin,admin,1'>/etc/kubernetes/basic_auth_file#grep'auth'/usr/lib/systemd/system/kube-apiserver.service
--authorization-mode=Node,RBAC\\
--runtime-config=rbac.authorization.k8s.io/v1alpha1\\
--enable-bootstrap-token-auth=true\\
--token-auth-file=/etc/kubernetes/token.csv\\
--basic-auth-file=/etc/kubernetes/basic_auth_file\\#grep‘basic’k8s-dashborad-deployment.yaml（配置在args下面）
---authentication-mode=basic#systemctldaemon-reload#systemctlrestartkube-apiserver#kubectlapply-fk8s-dashborad-deployment.yaml

將admin用戶和cluter-admin role進行角色綁定

#curl--insecurehttps://vm1:6443-basic-uadmin:admin#kubectlcreateclusterrolebinding\\login-on-dashboard-with-cluster-admin\\
--clusterrole=cluster-admin--user=admin#curl--insecurehttps://vm1:6443-basic-uadmin:admin

五、訪問測試

六、整合heapster和influxdb

在沒有配置heapster和influxdb的情況下，pod的metric信息是無法獲取到的，而早前版本K8S的HPA特性依賴的metric數(shù)據(jù)來源恰巧就是heapster和influxdb。

1、準(zhǔn)備yaml配置文件

#catheapster-sa.yamlapiVersion:v1kind:ServiceAccountmetadata:
name:heapster
namespace:kube-system

#catheapster-rbac.yamlkind:ClusterRoleBindingapiVersion:rbac.authorization.k8s.io/v1beta1metadata:
name:heapsterroleRef:
apiGroup:rbac.authorization.k8s.io
kind:ClusterRole
name:system:heapstersubjects:-kind:ServiceAccount
name:heapster
namespace:kube-system

#catheapster-deployment.yamlapiVersion:extensions/v1beta1
kind:Deployment
metadata:
name:heapsternamespace:kube-system
spec:
replicas:1
template:
metadata:
labels:
task:monitoring
k8s-app:heapster
spec:
serviceAccountName:heapster
containers:
-name:heapster
image:k8s.gcr.io/heapster-amd64:v1.4.2
imagePullPolicy:IfNotPresent
command:
-/heapster
---source=kubernetes:https://kubernetes.default
---sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086

#catheapster-service.yamlapiVersion:v1kind:Servicemetadata:
labels:
task:monitoring
kubernetes.io/cluster-service:'true'
kubernetes.io/name:Heapster
name:heapster
namespace:kube-systemspec:
ports:
-port:80
targetPort:8082
selector:k8s-app:heapster

#catinfluxdb-deployment.yamlapiVersion:extensions/v1beta1kind:Deploymentmetadata:
name:monitoring-influxdb
namespace:kube-systemspec:
replicas:1
template:
metadata:
labels:
task:monitoring
k8s-app:influxdb
spec:
containers:
-name:influxdb
image:k8s.gcr.io/heapster-influxdb-amd64:v1.3.3
volumeMounts:
-mountPath:/data
name:influxdb-storage
volumes:
-name:influxdb-storage
emptyDir:{}

#catinfluxdb-service.yamlapiVersion:v1kind:Servicemetadata:
labels:
task:monitoring
kubernetes.io/cluster-service:'true'
kubernetes.io/name:monitoring-influxdb
name:monitoring-influxdb
namespace:kube-systemspec:
ports:
-port:8086
targetPort:8086
selector:
k8s-app:influxdb

獲取heapster中的獲取支持的metrics

#kubectlrun-i--ttycurl--namespace=kube-system\\--image=registry.59iedu.com/webwurst/curl-utils/bin/sh
#curlhttp://heapster/api/v1/model/metrics#curlhttp://heapster/api/v1/model/debug/allkeys

#kubectlgetnode#kubectltopnode

當(dāng)heapster和influxdb pod都正常運行的時候，在dashboard里面就可以看到CPU和內(nèi)存的監(jiān)控數(shù)據(jù)了。

七、配置用戶權(quán)限

1、刪除apiserver里面basic認(rèn)證相關(guān)的配置后重啟apiserver
--basic-auth-file=/etc/kubernetes/basic_auth_file

#systemctldaemon-reload#systemctlrestartkube-apiserver

2、刪除clusterrolebinding

#kubectldeleteclusterrolebindinglogin-on-dashboard-with-cluster-admin

3、修改k8s-dashborad-deployment.yaml文件
去掉- --authentication-mode=basic參數(shù)

4、創(chuàng)建普通用戶,賦予所有namespace下資源的get、watch和list權(quán)限。
這里通過clusterrole和culsterrolebinding賦予所有namespace相關(guān)資源的get、watch、list權(quán)限，實際應(yīng)用環(huán)境建議使用創(chuàng)建role和rolebinding指定特定的namespace相關(guān)資源權(quán)限，各資源權(quán)限的賦予規(guī)則遵循最小權(quán)限原則。

#catrbac-yang.yamlkind:ClusterRoleapiVersion:rbac.authorization.k8s.io/v1metadata:
name:role-yangrules:-apiGroups:[""]
resources:["*"]
verbs:["get","watch","list"]
-apiGroups:["storage.k8s.io"]
resources:["*"]
verbs:["get","watch","list"]
-apiGroups:["rbac.authorization.k8s.io"]
resources:["*"]
verbs:["get","watch","list"]
-apiGroups:["batch"]
resources:["*"]
verbs:["get","watch","list"]
-apiGroups:["apps"]
resources:["*"]
verbs:["get","watch","list"]
-apiGroups:["extensions"]
resources:["*"]
verbs:["get","watch","list"]
---kind:ClusterRoleBindingapiVersion:rbac.authorization.k8s.io/v1metadata:
name:role-bind-yangsubjects:-kind:ServiceAccount
name:yang
namespace:kube-systemroleRef:
kind:ClusterRole
name:role-yang
apiGroup:rbac.authorization.k8s.io

#kubectlcreatesayang-nkube-system#kubectlcreate-frbac-yang.yaml#kubectl-nkube-systemdescribesecret$(kubectl-nkube-systemgetsecret|grepyang|awk'{print$1}')

5、測試普通用戶的權(quán)限




6、創(chuàng)建super用戶admin

#kubectlcreatesaadmin-nkube-system#catrbac-admin.yamlapiVersion:rbac.authorization.k8s.io/v1beta1
kind:ClusterRoleBinding
metadata:
name:admin
roleRef:
apiGroup:rbac.authorization.k8s.io
kind:ClusterRole
name:cluster-admin
subjects:
-kind:ServiceAccount
name:admin
namespace:kube-system#kubectlcreate-frbac-admin.yaml#kubectl-nkube-systemdescribesecret$(kubectl-nkube-systemgetsecret|grepadmin|awk'{print$1}')

使用admin用戶的token登陸后繼承cluster-admin的權(quán)限

參考：
https://github.com/kubernetes/dashboard/wiki/Creating-sample-user
https://github.com/kubernetes/dashboard/wiki/Access-control
https://github.com/kubernetes/heapster/blob/master/docs/model.md

當(dāng)前名稱：K8S使用dashboard管理集群
標(biāo)題鏈接：http://weahome.cn/article/cgpehi.html