Kubernetes调试环境搭建
K8s MacBook本地debug环境搭建
1. K8s集群部署
可以在本地通过https://github.com/flannel-io/flannel
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
查看系统pod运行情况
kubectl get pods -n kube-system
部署测试服务，验证集群状态
[root@k8s-master kubernetes]# kubectl create deployment nginx --image=nginx deployment.apps/nginx created
[root@k8s-master kubernetes]# kubectl expose deployment nginx --port=80 --type=NodePort service/nginx exposed
[root@k8s-master kubernetes]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.1.0.1443/TCP 25h nginx NodePort 10.1.167.250 80:30700/TCP 25h [root@k8s-master kubernetes]# curl 10.1.167.250 <head> Welcome to nginx!
选装部署dashboard，dashboard其实对本地开发调试用处不大，可选择安装
安装dashboard：（https://github.com/kubernetes/dashboard）
2.调试环境（开发环境默认已安装）
我们使用Goland+delve进行远程调试
需要调试的组件运行的节点需要安装Go环境，注意Go版本和Kubernetes的对应关系，我们本次演示以当前最新版本1.23版本的kube-apiserver进行调试
1.23需要Go版本为1.17+
我们在kube-apiserver所在的节点192.168.1.121安装Go1.17.5
[root@k8s-master app]# wget https://go.dev/dl/go1.17.5.linux-amd64.tar.gz [root@k8s-master app]# tar zxf go1.17.5.linux-amd64.tar.gz # add env [root@k8s-master app]# vim /etc/profile #追加 #add go export GOROOT=/app/go export PATH=$PATH:$GOROOT/bin export GOPATH=/root/go export PATH=$PATH:$GOPATH/bin [root@k8s-master app]# source /etc/profile #验证版本 [root@k8s-master app]# go version go version go1.17.5 linux/amd64
安装delve
参考：https://github.com/go-delve/delve/tree/master/Documentation/installation
# Install the latest release: $ go install github.com/go-delve/delve/cmd/dlv@latest # Install at tree head: $ go install github.com/go-delve/delve/cmd/dlv@master # Install at a specific version or pseudo-version: $ go install github.com/go-delve/delve/cmd/dlv@v1.7.3 $ go install github.com/go-delve/delve/cmd/dlv@v1.7.4-0.20211208103735-2f13672765fe
kubeadm创建的集群中的控制组件是static pod，所以我们只要把配置文件移动到别的目录，对应的static pod就会自己销毁，于是我们可以用移除配置文件的方式停止集群中的kube-apiserver
#进入kubeadm 默认的配置路径 [root@k8s-master ~]# cd /etc/kubernetes #新建static pod配置备份目录 [root@k8s-master ~]# mkdir manifests-down [root@k8s-master kubernetes]# mv ./manifests/kube-apiserver.yaml ./manifests #看到apiserver组件对应的pod已经停止 [root@k8s-master kubernetes]# kubectl get po --all-namespaces
本地编译K8s源码，本地环境MacBook Pro为darwin，因此需要交叉编译出Linux amd64对应的apiserver二级制文件
参考编译命令：
make all GOGCFLAGS="-N -l" GOLDFLAGS=""
需要添加：GOLDFLAGS="" 不然无法支持远程调试，报错为：
could not launch process: could not open debug info
解决方案参考：https://github.com/kubernetes/kubernetes/issues/77527编译好的目录结构如图所示：
把编译好的二进制文件上传至master机器192.168.1.121
scp ./_output/local/bin/linux/amd64/kube-apiserver root@192.168.1.121:/etc/kubernetes/mycode
从manifests-down/kube-apiserver.yaml 获取启动配置信息
cd manifests-down/kube-apiserver.yaml
配置示例：
apiVersion: v1 kind: Pod metadata: annotations: kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.1.121:6443 creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver - --advertise-address=192.168.1.121 - --allow-privileged=true - --authorization-mode=Node,RBAC - --client-ca-file=/etc/kubernetes/pki/ca.crt - --enable-admission-plugins=NodeRestriction - --enable-bootstrap-token-auth=true - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key - --etcd-servers=https://127.0.0.1:2379 - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key - --requestheader-allowed-names=front-proxy-client - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt - --requestheader-extra-headers-prefix=X-Remote-Extra- - --requestheader-group-headers=X-Remote-Group - --requestheader-username-headers=X-Remote-User - --secure-port=6443 - --service-account-issuer=https://kubernetes.default.svc.cluster.local - --service-account-key-file=/etc/kubernetes/pki/sa.pub - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key - --service-cluster-ip-range=10.1.0.0/16 - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key image: k8s.gcr.io/kube-apiserver:v1.23.1 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 192.168.1.121 path: /livez port: 6443 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 15 name: kube-apiserver readinessProbe: failureThreshold: 3 httpGet: host: 192.168.1.121 path: /readyz port: 6443 scheme: HTTPS periodSeconds: 1 timeoutSeconds: 15 resources: requests: cpu: 250m startupProbe: failureThreshold: 24 httpGet: host: 192.168.1.121 path: /livez port: 6443 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 15 volumeMounts: - mountPath: /etc/ssl/certs name: ca-certs readOnly: true - mountPath: /etc/pki name: etc-pki readOnly: true - mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true hostNetwork: true priorityClassName: system-node-critical securityContext: seccompProfile: type: RuntimeDefault volumes: - hostPath: path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs - hostPath: path: /etc/pki type: DirectoryOrCreate name: etc-pki - hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certs status: {}
拼接后的启动命令,用默认的2345端口
dlv exec --listen=:2345 --headless=true --api-version=2 ./kube-apiserver -- --advertise-address=192.168.1.121 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.1.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
参数过长，可以写入一个启动脚本文件kube-apiserver-start.sh 保存起来方便使用
启动脚本：
chmod +x kube-apiserver-start.sh ./kube-apiserver-start.sh
切换到本地的Goland 添加远程调试参数
在apiserver的main方法 添加断点，启动debug，已经阻塞在断点处，开始进行远程debug