etcd（读作 et-see-dee）是一种开源的分布式统一键值存储，用于分布式系统或计算机集群的共享配置、服务发现和的调度协调。etcd 有助于促进更加安全的自动更新，协调向主机调度的工作，并帮助设置容器的覆盖网络。

etcd 是许多其他项目的核心组件。最值得注意的是，它是 Kubernetes 的首要数据存储，也是容器编排的实际标准系统。使用 etcd， 云原生应用可以保持更为一致的运行时间，而且在个别服务器发生故障时也能正常工作。应用从 etcd 读取数据并写入到其中；通过分散配置数据，为节点配置提供冗余和弹性。

Kubernetes 和 etcd

作为 Kubernetes 的首要数据存储，etcd 存储和复制所有的 Kubernetes 集群状态。etcd 是 Kubernetes 集群的关键组件，因此必须要有一种可靠的方法来进行配置和管理。

etcd 属于基于共识的分布式系统，因此 etcd 的配置可能比较复杂。引导、维护仲裁、重新配置集群成员身份、创建备份、处理灾难恢复以及监控重要事件都是复杂、繁冗的任务，需要运用相关的专业技能。

注意选择etcd 不要选择etcd二的版本，需要选择3点多以后的版本，原因是3以前的版本数据是存储在内存中，三以后的版本是在磁盘中。如果节点异常会导致数据丢失！

集群规划（lead只能有一个，如果有多个或者经常变动需要检查配置，这个俗称脑裂）

主机名 角色 ip
HDss7-12.host.com ectc lead 10.4.7.12
HDss7-21.host.com ectc follow 10.4.7.21
HDss7-22.host.com ectc follow 10.4.7.22

注释：这里部署文档以HDss7-12.host.com主机为例，另外两台安装部署方法类似

在HDss7-200上创建基于根证书的config配置文件

[root@hdss7-200 ~]# vim /opt/certs/ca-config.json
{
    "signing": {
        "default": {
            "expiry": "175200h"
        },
        "profiles": {
            "server": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {                
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

IP地址必须在文档内更改好再粘贴复制进去，IP地址为有可能装ETCD的主机，多一个IP为预备，

[root@hdss7-200 ~]# vim /opt/certs/etcd-peer-csr.json
{
    "CN": "k8s-etcd",
    "hosts": [
        "10.4.7.11",
        "10.4.7.12",
        "10.4.7.21",
        "10.4.7.22"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

注意这里引用了ca的根证书，可以了解下ca的证书签发过程，二进制的安装方式其实最复杂的就是证书签发。

[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssl-json -bare etcd-peer

在etcd主机上创建etcd用户

# useradd -s /sbin/nologin -M etcd
# id etcd
mkdir -p /data/soft/
tar xf etcd-v3.1.20-linux-amd64.tar.gz 
mv etcd-v3.1.20-linux-amd64 /usr/local/
cd /usr/local/
ln -sf etcd-v3.1.20-linux-amd64/ etcd

三台机器创建文件夹存放ca证书

mkdir -p /usr/local/etcd/certs

拷贝证书到三台节点上。

[root@hdss7-200 certs]# scp ca.pem 10.4.7.22:/usr/local/etcd/certs 
[root@hdss7-200 certs]# scp etcd-peer-key.pem 10.4.7.22:/usr/local/etcd/certs
[root@hdss7-200 certs]# scp etcd-peer.pem 10.4.7.22:/usr/local/etcd/certs


[root@hdss7-200 certs]# scp ca.pem 10.4.7.21:/usr/local/etcd/certs 
[root@hdss7-200 certs]# scp etcd-peer-key.pem 10.4.7.21:/usr/local/etcd/certs
[root@hdss7-200 certs]# scp etcd-peer.pem 10.4.7.21:/usr/local/etcd/certs

[root@hdss7-200 certs]# scp ca.pem 10.4.7.12:/usr/local/etcd/certs 
[root@hdss7-200 certs]# scp etcd-peer-key.pem 10.4.7.12:/usr/local/etcd/certs
[root@hdss7-200 certs]# scp etcd-peer.pem 10.4.7.12:/usr/local/etcd/certs

建立启动脚本，注意这里的换行不能有空格，不然会导致无法找到参数。（坑注意了）

第一台配置

[root@hdss7-12 ~]# vim /usr/local/etcd/etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-7-12 \
       --data-dir /data/etcd/etcd-server \
       --listen-peer-urls https://10.4.7.12:2380 \
       --listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
       --quota-backend-bytes 8000000000 \
       --initial-advertise-peer-urls https://10.4.7.12:2380 \
       --advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
       --initial-cluster  etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
       --ca-file ./certs/ca.pem \
       --cert-file ./certs/etcd-peer.pem \
       --key-file ./certs/etcd-peer-key.pem \
       --client-cert-auth  \
       --trusted-ca-file ./certs/ca.pem \
       --peer-ca-file ./certs/ca.pem \
       --peer-cert-file ./certs/etcd-peer.pem \
       --peer-key-file ./certs/etcd-peer-key.pem \
       --peer-client-cert-auth \
       --peer-trusted-ca-file ./certs/ca.pem \
       --log-output stdout

允许执行创建数据存储目录和日志目录

chmod +x etcd-server-startup.sh

mkdir -p /data/etcd /data/logs/etcd-server && chown -R etcd.etcd /data/etcd /data/logs/etcd-server /usr/local/etcd-v3.1.20-linux-amd64 /usr/local/etcd

安装supervisor作为etcd后台运行，您也可以使用systemctl代替！

[root@hdss7-12 logs]# yum install supervisor -y
[root@hdss7-12 logs]# systemctl start supervisord
[root@hdss7-12 logs]# systemctl enable supervisord
[root@hdss7-12 ~]# vim /etc/supervisord.d/etcd-server.ini
[program:etcd-server-7-12]
command=/usr/local/etcd/etcd-server-startup.sh                  ; the program (relative uses PATH, can take args)    
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/usr/local/etcd                                       ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=30                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd                                                       ; setuid to this UNIX account to run the program
redirect_stderr=true                                            ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log           ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)

第二台配置内容

vim /usr/local/etcd/etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-7-21 \
       --data-dir /data/etcd/etcd-server \
       --listen-peer-urls https://10.4.7.21:2380 \
       --listen-client-urls https://10.4.7.21:2379,http://127.0.0.1:2379 \
       --quota-backend-bytes 8000000000 \
       --initial-advertise-peer-urls https://10.4.7.21:2380 \
       --advertise-client-urls https://10.4.7.21:2379,http://127.0.0.1:2379 \
       --initial-cluster  etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
       --ca-file ./certs/ca.pem \
       --cert-file ./certs/etcd-peer.pem \
       --key-file ./certs/etcd-peer-key.pem \
       --client-cert-auth  \
       --trusted-ca-file ./certs/ca.pem \
       --peer-ca-file ./certs/ca.pem \
       --peer-cert-file ./certs/etcd-peer.pem \
       --peer-key-file ./certs/etcd-peer-key.pem \
       --peer-client-cert-auth \
       --peer-trusted-ca-file ./certs/ca.pem \
       --log-output stdout


[root@hdss7-21 ~]# vim /etc/supervisord.d/etcd-server.ini
[program:etcd-server-7-21]
command=/usr/local/etcd/etcd-server-startup.sh                  ; the program (relative uses PATH, can take args)    
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/usr/local/etcd                                       ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=30                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd                                                       ; setuid to this UNIX account to run the program
redirect_stderr=true                                            ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log           ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)

第三台配置内容

vim /usr/local/etcd/etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-7-22 \
       --data-dir /data/etcd/etcd-server \
       --listen-peer-urls https://10.4.7.22:2380 \
       --listen-client-urls https://10.4.7.22:2379,http://127.0.0.1:2379 \
       --quota-backend-bytes 8000000000 \
       --initial-advertise-peer-urls https://10.4.7.22:2380 \
       --advertise-client-urls https://10.4.7.22:2379,http://127.0.0.1:2379 \
       --initial-cluster  etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
       --ca-file ./certs/ca.pem \
       --cert-file ./certs/etcd-peer.pem \
       --key-file ./certs/etcd-peer-key.pem \
       --client-cert-auth  \
       --trusted-ca-file ./certs/ca.pem \
       --peer-ca-file ./certs/ca.pem \
       --peer-cert-file ./certs/etcd-peer.pem \
       --peer-key-file ./certs/etcd-peer-key.pem \
       --peer-client-cert-auth \
       --peer-trusted-ca-file ./certs/ca.pem \
       --log-output stdout

[root@hdss7-22 ~]# vim /etc/supervisord.d/etcd-server.ini
[program:etcd-server-7-22]
command=/usr/local/etcd/etcd-server-startup.sh                  ; the program (relative uses PATH, can take args)    
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/usr/local/etcd                                       ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=30                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd                                                       ; setuid to this UNIX account to run the program
redirect_stderr=true                                            ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log           ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)

每台机器授权脚本的可执行权限并授权用户和用户组

chmod +x etcd-server-startup.sh
chown etcd.etcd etcd-server-startup.sh

#启动supervisord并设置开机启动
[root@hdss7-21 logs]# systemctl start supervisord
[root@hdss7-21 logs]# systemctl enable supervisord


[root@hdss7-22 logs]# systemctl start supervisord
[root@hdss7-22 logs]# systemctl enable supervisord

检查集群状态，如果是一下结果说明成功，现在我的leader在10.4.7.12上！

[root@hdss7-12 etcd]# ./etcdctl cluster-health
member 988139385f78284 is healthy: got healthy result from http://127.0.0.1:2379
member 5a0ef2a004fc4349 is healthy: got healthy result from http://127.0.0.1:2379
member f4a0cb0a765574a8 is healthy: got healthy result from http://127.0.0.1:2379
cluster is healthy


[root@hdss7-21 etcd]# ./etcdctl cluster-health
member 988139385f78284 is healthy: got healthy result from http://127.0.0.1:2379
member 5a0ef2a004fc4349 is healthy: got healthy result from http://127.0.0.1:2379
member f4a0cb0a765574a8 is healthy: got healthy result from http://127.0.0.1:2379
cluster is healthy


[root@hdss7-22 etcd]# ./etcdctl cluster-health
member 988139385f78284 is healthy: got healthy result from http://127.0.0.1:2379
member 5a0ef2a004fc4349 is healthy: got healthy result from http://127.0.0.1:2379
member f4a0cb0a765574a8 is healthy: got healthy result from http://127.0.0.1:2379
cluster is healthy


[root@hdss7-22 etcd]# ./etcdctl member list
988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.22:2379 isLeader=false
5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.21:2379 isLeader=false
f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=true

etcd搭建结束。

Last modification：May 5, 2022
© Allow specification reprint
如果觉得我的文章对你有用，请随意赞赏