Kubernetes后台数据库etcd：安装部署etcd集群，数据备份与恢复
目录
- http://192.168.110.133:2379,http://localhost:2379"，并重启etcd服务
[root@etcd1 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.110.133:2380,http://localhost:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379" ETCD_NAME="default" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379" #重启 [root@etcd1 ~]# systemctl restart etcd [root@etcd1 ~]# systemctl status etcd此时客户端连接192.168.110.133成功，ls /表示查询根目录下内容
[root@etcd2 ~]# etcdctl --endpoints http://192.168.110.133:2379 ls /4.4.1 使用2版本API管理etcd
etcd写入数据的时候有两个版本：2版本和3版本，默认是2版本，Set environment variable ETCDCTL_API=3 to use v3 API or ETCDCTL_API=2 to use v2 API
现在还没有数据
[root@etcd1 ~]# etcdctl ls /etcd1创建数据
[root@etcd1 ~]# etcdctl mkdir /comeon [root@etcd1 ~]# etcdctl ls / /comeon客户端连接也可以看到数据
[root@etcd2 ~]# etcdctl --endpoints http://192.168.110.133:2379 ls / /comeon删除数据
[root@etcd1 ~]# etcdctl rmdir /comeon [root@etcd1 ~]# etcdctl ls /4.4.2 使用3版本API管理etcd
现在使用3版本的API
[root@etcd1 ~]# export ETCDCTL_API=3设置为3版本的API之后，help显示的内容也会不同
[root@etcd1 ~]# etcdctl --help NAME: etcdctl - A simple command line client for etcd3. USAGE: etcdctl VERSION: 3.3.11 API VERSION: 3.3 COMMANDS: get Gets the key or a range of keys put Puts the given key into the store del Removes the specified key or range of keys [key, range_end) txn Txn processes all the requests in one transaction compaction Compacts the event history in etcd alarm disarm Disarms all alarms alarm list Lists all alarms defrag Defragments the storage of the etcd members with given endpoints endpoint health Checks the healthiness of endpoints specified in `--endpoints` flag endpoint status Prints out the status of endpoints specified in `--endpoints` flag endpoint hashkv Prints the KV history hash for each endpoint in --endpoints move-leader Transfers leadership to another etcd cluster member. watch Watches events stream on keys or prefixes version Prints the version of etcdctl lease grant Creates leases lease revoke Revokes leases lease timetolive Get lease information lease list List all active leases lease keep-alive Keeps leases alive (renew) member add Adds a member into the cluster member remove Removes a member from the cluster member update Updates a member in the cluster member list Lists all members in the cluster snapshot save Stores an etcd node backend snapshot to a given file snapshot restore Restores an etcd member snapshot to an etcd directory snapshot status Gets backend snapshot status of a given file make-mirror Makes a mirror at the destination etcd cluster migrate Migrates keys in a v2 store to a mvcc store lock Acquires a named lock elect Observes and participates in leader election auth enable Enables authentication auth disable Disables authentication user add Adds a new user user delete Deletes a user user get Gets detailed information of a user user list Lists all users user passwd Changes password of user user grant-role Grants a role to a user user revoke-role Revokes a role from a user role add Adds a new role role delete Deletes a role role get Gets detailed information of a role role list Lists all roles role grant-permission Grants a key to a role role revoke-permission Revokes a key from a role check perf Check the performance of the etcd cluster help Help about any command ...... -w, --write-out="simple" set the output format (fields, json, protobuf, simple, table)写数据
[root@etcd1 ~]# etcdctl put student1 99 OK查数据
[root@etcd1 ~]# etcdctl get student1 student1 99注意：2版本API和3版本API不可互用，从k8s1.5版本，etcd就开始使用3版本往etcd里写数据
[root@etcd2 ~]# export ETCDCTL_API=3 [root@etcd2 ~]# etcdctl --endpoints http://192.168.110.133:2379 get student1 student1 99etcd单节点搭建完毕，接下来添加两个节点变为etcd集群。
五.安装部署etcd集群
5.1 环境介绍
etcd集群架构：etcd1为leader，etcd2为follower，etcd3为follower
服务器 操作系统版本 CPU架构 进程 功能描述 etcd1/192.168.110.133 CentOS Linux release 7.4.1708 (Core) x86_64 etcd leader etcd2/192.168.110.131 CentOS Linux release 7.4.1708 (Core) x86_64 etcd follower etcd3/192.168.110.132 CentOS Linux release 7.4.1708 (Core) x86_64 etcd follower 5.2 把etcd2机器加入集群
首先还原环境变量
[root@etcd1 ~]# unset ETCDCTL_API停止etcd1机器的etcd服务
[root@etcd1 ~]# systemctl stop etcd [root@etcd1 ~]# systemctl status etcd ● etcd.service - Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled) Active: inactive (dead) since 二 2022-01-11 15:21:30 CST; 49s ago Process: 1582 ExecStart=/bin/bash -c GOMAXPROCS=$(nproc) /usr/bin/etcd --name="${ETCD_NAME}" --data-dir="${ETCD_DATA_DIR}" --listen-client-urls="${ETCD_LISTEN_CLIENT_URLS}" (code=killed, signal=TERM) Main PID: 1582 (code=killed, signal=TERM)查看etcd的数据目录
[root@etcd1 ~]# cat /etc/etcd/etcd.conf | grep -i data ETCD_DATA_DIR="/var/lib/etcd/default.etcd"删除原始数据
[root@etcd1 ~]# rm -rf /var/lib/etcd/*修改配置文件，把etcd2机器加入配置
[root@etcd1 ~]# vim /etc/etcd/etcd.conf [root@etcd1 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$" #配置数据目录 ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.110.133:2380,http://localhost:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379" ETCD_NAME="etcd133" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.110.133:2380" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379" #目前是两个节点，所以这里是两个节点的etcd ETCD_INITIAL_CLUSTER="etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380" #集群token ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #新创建etcd集群的时候ETCD_INITIAL_CLUSTER_STATE="new"，往已经存在的etcd集群添加etcd节点时：ETCD_INITIAL_CLUSTER_STATE="existing" ETCD_INITIAL_CLUSTER_STATE="new"复制配置文件到etcd2
[root@etcd1 ~]# scp /etc/etcd/etcd.conf etcd2:/etc/etcd/etcd.conf root@etcd2's password: etcd.conf 100% 1813 1.7MB/s 00:00etcd2机器修改配置文件
[root@etcd2 ~]# unset ETCDCTL_API [root@etcd2 ~]# vim /etc/etcd/etcd.conf [root@etcd2 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$" ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.110.131:2380,http://localhost:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.110.131:2379,http://localhost:2379" ETCD_NAME="etcd131" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.110.131:2380" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.131:2379,http://localhost:2379" ETCD_INITIAL_CLUSTER="etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"两个节点都启动etcd
[root@etcd1 ~]# systemctl start etcd [root@etcd1 ~]# systemctl status etcd [root@etcd2 ~]# systemctl enable etcd --now Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service. [root@etcd2 ~]# systemctl status etcd查看etcd集群成员，可以看到192.168.110.133节点是Leader
[root@etcd1 ~]# etcdctl member list 341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true集群健康状态
[root@etcd1 ~]# etcdctl cluster-health member 341a3c460c1c993a is healthy: got healthy result from http://192.168.110.131:2379 member ab23bcc86cf3190b is healthy: got healthy result from http://192.168.110.133:2379 cluster is healthy现在两个节点的etcd集群搭建完毕，数据也同步了
[root@etcd1 ~]# etcdctl ls / [root@etcd1 ~]# etcdctl mkdir /public [root@etcd1 ~]# export ETCDCTL_API=3 [root@etcd1 ~]# etcdctl put student1 59 OK [root@etcd2 ~]# etcdctl ls / /public [root@etcd2 ~]# export ETCDCTL_API=3 [root@etcd2 ~]# etcdctl get student1 student1 595.3 把etcd3机器加入集群
现在添加一个节点etcd3到集群
etcd3安装etcd
[root@etcd3 ~]# yum -y install etcd加入新节点的时候，使用API2版本
[root@etcd1 ~]# export ETCDCTL_API=2执行添加节点命令，注意：ETCD_INITIAL_CLUSTER_STATE="existing"
[root@etcd1 ~]# etcdctl member add etcd132 http://192.168.110.132:2380 Added member named etcd132 with ID 7d816f4fa2bea295 to cluster ETCD_NAME="etcd132" ETCD_INITIAL_CLUSTER="etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380,etcd133=http://192.168.110.133:2380" ETCD_INITIAL_CLUSTER_STATE="existing"查看集群成员，发现192.168.110.132显示不正常
[root@etcd1 ~]# etcdctl member list 341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false 7d816f4fa2bea295[unstarted]: peerURLs=http://192.168.110.132:2380 ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true复制配置文件到etcd3
[root@etcd1 ~]# scp /etc/etcd/etcd.conf etcd3:/etc/etcd/etcd.conf root@etcd3's password: etcd.conf 100% 1813 1.1MB/s 00:00etcd3修改配置文件
#注意：添加到一个已经存在的集群，etcd133和etcd131配置文件不变，只修改etcd132配置文件 [root@etcd3 ~]# vim /etc/etcd/etcd.conf [root@etcd3 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$" ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.110.132:2380,http://localhost:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.110.132:2379,http://localhost:2379" ETCD_NAME="etcd132" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.110.132:2380" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.132:2379,http://localhost:2379" ETCD_INITIAL_CLUSTER="etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="existing"etcd3启动etcd
[root@etcd3 ~]# systemctl enable etcd --now Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.查看etcd集群成员，etcd集群有三个节点了
[root@etcd1 ~]# etcdctl member list 341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false 7d816f4fa2bea295: name=etcd132 peerURLs=http://192.168.110.132:2380 clientURLs=http://192.168.110.132:2379,http://localhost:2379 isLeader=false ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true自此etcd集群搭建完毕，接下来进行etcd服务的管理。
六.etcd做快照备份数据
生产环境中，有些重要数据是要备份的，以免造成数据丢失。etcd对数据做快照可以进行数据备份。
设置API版本为3
[root@etcd1 ~]# export ETCDCTL_API=3为了避免数据丢失，可以进行数据备份，即数据做快照，数据快照的help如下
[root@etcd1 ~]# etcdctl snap --help NAME: snapshot - Manages etcd node snapshots USAGE: etcdctl snapshotAPI VERSION: 3.3 COMMANDS: save Stores an etcd node backend snapshot to a given file restore Restores an etcd member snapshot to an etcd directory status Gets backend snapshot status of a given file GLOBAL OPTIONS: --cacert="" verify certificates of TLS-enabled secure servers using this CA bundle --cert="" identify secure client using this TLS certificate file --command-timeout=5s timeout for short running command (excluding dial timeout) --debug[=false] enable client-side debug logging --dial-timeout=2s dial timeout for client connections -d, --discovery-srv="" domain name to query for SRV records describing cluster endpoints --endpoints=[127.0.0.1:2379] gRPC endpoints --hex[=false] print byte strings as hex encoded strings --insecure-discovery[=true] accept insecure SRV records describing cluster endpoints --insecure-skip-tls-verify[=false] skip server certificate verification --insecure-transport[=true] disable transport security for client connections --keepalive-time=2s keepalive time for client connections --keepalive-timeout=6s keepalive timeout for client connections --key="" identify secure client using this TLS key file --user="" username[:password] for authentication (prompt if password is not supplied) -w, --write-out="simple" set the output format (fields, json, protobuf, simple, table) 进行快照
[root@etcd1 ~]# etcdctl snap save student.data Snapshot saved at student.data七.etcd恢复数据
删除数据使用快照进行恢复
[root@etcd1 ~]# etcdctl del student1 1 [root@etcd1 ~]# etcdctl del student2 1 [root@etcd1 ~]# etcdctl get student1 [root@etcd1 ~]# etcdctl get student2使用快照恢复数据的时候需要把快照复制到其他节点
[root@etcd1 ~]# scp student.data etcd2:~/ root@etcd2's password: student.data 100% 20KB 14.5MB/s 00:00 [root@etcd1 ~]# scp student.data etcd3:~/ root@etcd3's password: student.data 100% 20KB 8.1MB/s 00:00在所有节点停止etcd并清空数据
[root@etcd1 ~]# systemctl stop etcd [root@etcd1 ~]# rm -rf /var/lib/etcd/* [root@etcd1 ~]# chown etcd:etcd student.data [root@etcd2 ~]# systemctl stop etcd [root@etcd2 ~]# rm -rf /var/lib/etcd/* [root@etcd2 ~]# chown etcd:etcd student.data [root@etcd3 ~]# systemctl stop etcd [root@etcd3 ~]# rm -rf /var/lib/etcd/* [root@etcd3 ~]# chown etcd:etcd student.data在每个节点上使用快照恢复数据
[root@etcd1 ~]# etcdctl snapshot restore student.data --name etcd133 --initial-cluster etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380 --initial-advertise-peer-urls http://192.168.110.133:2380 --data-dir /var/lib/etcd/cluster.etcd 2022-01-11 16:44:08.375319 I | etcdserver/membership: added member 341a3c460c1c993a [http://192.168.110.131:2380] to cluster dd7594df5e81191b 2022-01-11 16:44:08.375393 I | etcdserver/membership: added member 4679fe0fcb37326d [http://192.168.110.132:2380] to cluster dd7594df5e81191b 2022-01-11 16:44:08.375404 I | etcdserver/membership: added member ab23bcc86cf3190b [http://192.168.110.133:2380] to cluster dd7594df5e81191b [root@etcd2 ~]# etcdctl snapshot restore student.data --name etcd131 --initial-cluster etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380 --initial-advertise-peer-urls http://192.168.110.131:2380 --data-dir /var/lib/etcd/cluster.etcd 2022-01-11 16:45:18.378931 I | etcdserver/membership: added member 341a3c460c1c993a [http://192.168.110.131:2380] to cluster dd7594df5e81191b 2022-01-11 16:45:18.378991 I | etcdserver/membership: added member 4679fe0fcb37326d [http://192.168.110.132:2380] to cluster dd7594df5e81191b 2022-01-11 16:45:18.379000 I | etcdserver/membership: added member ab23bcc86cf3190b [http://192.168.110.133:2380] to cluster dd7594df5e81191b [root@etcd3 ~]# etcdctl snapshot restore student.data --name etcd132 --initial-cluster etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380 --initial-advertise-peer-urls http://192.168.110.132:2380 --data-dir /var/lib/etcd/cluster.etcd 2022-01-11 16:46:26.826533 I | etcdserver/membership: added member 341a3c460c1c993a [http://192.168.110.131:2380] to cluster dd7594df5e81191b 2022-01-11 16:46:26.826584 I | etcdserver/membership: added member 4679fe0fcb37326d [http://192.168.110.132:2380] to cluster dd7594df5e81191b 2022-01-11 16:46:26.826595 I | etcdserver/membership: added member ab23bcc86cf3190b [http://192.168.110.133:2380] to cluster dd7594df5e81191b修改所有节点数据目录的属主
[root@etcd1 ~]# chown -R etcd:etcd /var/lib/etcd/ [root@etcd2 ~]# chown -R etcd:etcd /var/lib/etcd/ [root@etcd3 ~]# chown -R etcd:etcd /var/lib/etcd/启动etcd
[root@etcd1 ~]# systemctl start etcd [root@etcd2 ~]# systemctl start etcd [root@etcd3 ~]# systemctl start etcd可以发现数据已经恢复
[root@etcd1 ~]# etcdctl get student1 student1 59 [root@etcd3 ~]# etcdctl get student2 student2 62八.Kubernetes(k8s)中以pod方式运行的etcd
etcd在Kubernetes集群中可以以pod的方式运行，也可以以物理机部署的方式运行，本章讲解以pod方式运行的etcd。
在此之前，需要有一套可以正常运行的Kubernetes集群，关于Kubernetes(k8s)集群的安装部署，可以查看博客《Centos7 安装部署Kubernetes(k8s)集群》
在k8s中etcd以pod的方式运行，那配置文件在哪里，数据目录在哪里？
查看etcd pod，k8s中etcd为etcd-k8scloude1
[root@k8scloude1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE calico-kube-controllers-6b9fbfff44-4jzkj 1/1 Running 4 2d4h calico-node-bdlgm 1/1 Running 2 2d4h calico-node-hx8bk 1/1 Running 2 2d4h calico-node-nsbfs 1/1 Running 2 2d4h coredns-545d6fc579-7wm95 1/1 Running 2 2d4h coredns-545d6fc579-87q8j 1/1 Running 2 2d4h etcd-k8scloude1 1/1 Running 2 2d4h kube-apiserver-k8scloude1 1/1 Running 2 2d4h kube-controller-manager-k8scloude1 1/1 Running 2 2d4h kube-proxy-599xh 1/1 Running 2 2d4h kube-proxy-lpj8z 1/1 Running 2 2d4h kube-proxy-zxlk9 1/1 Running 2 2d4h kube-scheduler-k8scloude1 1/1 Running 2 2d4h metrics-server-bcfb98c76-k5dmj 1/1 Running 1 33h在k8s中etcd以pod的形式运行， 此etcd的配置文件在哪？在/etc/kubernetes/manifests/etcd.yaml
[root@k8scloude1 ~]# ls /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/manifests/etcd.yaml [root@k8scloude1 ~]# cat !$ cat /etc/kubernetes/manifests/etcd.yaml apiVersion: v1 kind: Pod metadata: annotations: kubeadm.kubernetes.io/etcd.advertise-client-urls: https://192.168.110.130:2379 creationTimestamp: null labels: component: etcd tier: control-plane name: etcd namespace: kube-system spec: containers: - command: - etcd - --advertise-client-urls=https://192.168.110.130:2379 - --cert-file=/etc/kubernetes/pki/etcd/server.crt - --client-cert-auth=true - --data-dir=/var/lib/etcd - --initial-advertise-peer-urls=https://192.168.110.130:2380 - --initial-cluster=k8scloude1=https://192.168.110.130:2380 - --key-file=/etc/kubernetes/pki/etcd/server.key - --listen-client-urls=https://127.0.0.1:2379,https://192.168.110.130:2379 - --listen-metrics-urls=http://127.0.0.1:2381 - --listen-peer-urls=https://192.168.110.130:2380 - --name=k8scloude1 - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt - --peer-client-cert-auth=true - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt - --snapshot-count=10000 - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt image: registry.aliyuncs.com/google_containers/etcd:3.4.13-0 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 127.0.0.1 path: /health port: 2381 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 15 name: etcd resources: requests: cpu: 100m ephemeral-storage: 100Mi memory: 100Mi startupProbe: failureThreshold: 24 httpGet: host: 127.0.0.1 path: /health port: 2381 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 15 volumeMounts: - mountPath: /var/lib/etcd name: etcd-data - mountPath: /etc/kubernetes/pki/etcd name: etcd-certs hostNetwork: true priorityClassName: system-node-critical volumes: - hostPath: path: /etc/kubernetes/pki/etcd type: DirectoryOrCreate name: etcd-certs - hostPath: path: /var/lib/etcd type: DirectoryOrCreate name: etcd-data status: {}可以发现挂载了数据卷，数据目录在/var/lib/etcd/
[root@k8scloude1 ~]# ls /var/lib/etcd/ member [root@k8scloude1 ~]# ls /var/lib/etcd/member/ snap wal