Etcd+TLS集群部署 作者: sysit 分类: d 发表于 2019-03-14 151人围观 etcd是key-value存储（同zookeeper），在整个kubernetes集群中处于中心数据库地位，以集群的方式部署，可有效避免单点故障。 这里采用静态配置的方式部署（另也可通过etcd提供的rest api在运行时动态添加，修改或删除集群成员）。 # 1. 主机 ``` 192.168.112.51 master1.sysit.cn master1 192.168.112.52 master2.sysit.cn master2 192.168.112.53 master3.sysit.cn master3 ``` # 2. 安装配置etcd ## 2.1 安装etcd ``` # 三台上都安装 yum install etcd ``` ## 2.2. etcd TLS证书和私钥 ### 2.2.1 cfssl工具 本文档采用 `CloudFlare` 的 `PKI` 工具集 `cfssl` 来生成 `Certificate Authority (CA)` 证书和秘钥文件，`CA` 是自签名的证书，用来签名后续创建的其它 `TLS `证书。 ``` mkdir -p /usr/local/cfssl/bin cd /usr/local/cfssl/bin wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O cfssl wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O cfssl-certinfo wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O cfssljson chmod +x cfssl cfssl-certinfo cfssljson echo 'export PATH=$PATH:/usr/local/cfssl/bin' >>/etc/bashrc ``` ### 2.2.2 CA配置文件 ``` # ca-config.json：1个profiles，分别指定不同的过期时间，使用场景等参数，根据需要在不同场景使用不同的profile签名证书；这里以生成的模板为基础修改； cat > ca-config.json <<EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "etcd": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } EOF ``` * 字段说明: * ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile； * signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE； * server auth"：表示client可以用该 CA 对server提供的证书进行验证； * client auth"：表示server可以用该CA对client提供的证书进行验证； ### 2.2.3 CA证书签名请求 ``` cat > ca-csr.json <<EOF { "CN": "etcd", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Chengdu", "L": "Chengdu", "O": "etcd", "OU": "System" } ] } EOF ``` ### 2.2.4 生成CA证书与秘钥 ``` cfssl gencert -initca ca-csr.json | cfssljson -bare ca ls ca* ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem #简单查看 cfssl-certinfo -cert ca.pem ``` ### 2.2.5 创建etcd TLS证书与私钥 > 客户端（etcdctl）与etcd集群，etcd集群之间通信采用TLS加密。 ``` # hosts 字段指定授权使用该证书的 etcd 节点 IP 或域名列表，这里将 etcd 集群的三个节点 IP 都列在其中； cat > etcd-csr.json <<'EOF' { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.112.51", "192.168.112.52", "192.168.112.53" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Chengdu", "L": "Chengdu", "O": "etcd", "OU": "System" } ] } EOF # 生成etcd证书与私钥 cfssl gencert -ca=ca.pem \ -ca-key=ca-key.pem \ -config=ca-config.json \ -profile=etcd etcd-csr.json | cfssljson -bare etcd ``` ### 2.2.6 分发证书和私钥 ``` mkdir -p /etc/etcd/cert cp etcd*pem /etc/etcd/cert/ cp ca.pem /etc/etcd/cert/ chown -R etcd:etcd /etc/etcd/cert scp -r /etc/etcd/cert root@master2:/etc/etcd/cert ssh root@master2 "chown -R etcd:etcd /etc/etcd/cert" scp -r /etc/etcd/cert root@master3:/etc/etcd/cert ssh root@master3 "chown -R etcd:etcd /etc/etcd/cert" # 分发ca.pem # ca.pem必须存在并分发。 ``` ## 2.3. 安装配置etcd ### 2.3.1 配置etcd的配置文件 ``` # master1 [root@master1 ~]# egrep -v "^#|^$" /etc/etcd/etcd.conf ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.112.51:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.112.51:2379,http://127.0.0.1:2379" ETCD_NAME="master1" ETCD_HEARTBEAT_INTERVAL="200" ETCD_ELECTION_TIMEOUT="2000" ETCD_QUOTA_BACKEND_BYTES="6442450944" ETCD_MAX_REQUEST_BYTES="33554432" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.112.51:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.112.51:2379" ETCD_INITIAL_CLUSTER="master1=https://192.168.112.51:2380,master2=https://192.168.112.52:2380,master3=https://192.168.112.53:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-0" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem" ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/cert/ca.pem" ETCD_AUTO_TLS="true" ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem" ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/cert/ca.pem" ETCD_PEER_AUTO_TLS="true" # master2 [root@master2 ~]# egrep -v "^#|^$" /etc/etcd/etcd.conf ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.112.52:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.112.52:2379,http://127.0.0.1:2379" ETCD_NAME="master2" ETCD_HEARTBEAT_INTERVAL="200" ETCD_ELECTION_TIMEOUT="2000" ETCD_QUOTA_BACKEND_BYTES="6442450944" ETCD_MAX_REQUEST_BYTES="33554432" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.112.52:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.112.52:2379" ETCD_INITIAL_CLUSTER="master1=https://192.168.112.51:2380,master2=https://192.168.112.52:2380,master3=https://192.168.112.53:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-0" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem" ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/cert/ca.pem" ETCD_AUTO_TLS="true" ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem" ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/cert/ca.pem" ETCD_PEER_AUTO_TLS="true" # k8s-master3 [root@master3 ~]# egrep -v "^#|^$" /etc/etcd/etcd.conf ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.112.53:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.112.53:2379,http://127.0.0.1:2379" ETCD_NAME="master3" ETCD_HEARTBEAT_INTERVAL="200" ETCD_ELECTION_TIMEOUT="2000" ETCD_QUOTA_BACKEND_BYTES="6442450944" ETCD_MAX_REQUEST_BYTES="33554432" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.112.53:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.112.53:2379" ETCD_INITIAL_CLUSTER="master1=https://192.168.112.51:2380,master2=https://192.168.112.52:2380,master3=https://192.168.112.53:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-0" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem" ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/cert/ca.pem" ETCD_AUTO_TLS="true" ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem" ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/cert/ca.pem" ETCD_PEER_AUTO_TLS="true" ``` * ETCD_CERT_FILE，ETCD_KEY_FILE：etcd server 与 client 通信时使用的证书和私钥； * ETCD_PEER_CERT_FILE，ETCD_PEER_KEY_FILE：分别指定etcd的peers通信的公钥证书和私钥； * ETCD_TRUSTED_CA_FILE：签名 client 证书的 CA 证书，用于验证 client 证书； * ETCD_PEER_TRUSTED_CA_FILE：指定签名 peer 证书的 CA 证书，用于验证 peer 证书； * ETCD_NAME：指定节点名称，当 ETCD_INITIAL_CLUSTER_STATE 值为 new 时，ETCD_NAME 的参数值必须位于 ETCD_INITIAL_CLUSTER 列表中； * ETCD_WAL_DIR：指定 wal 目录，为了提高性能，一般使用 SSD 或者和 --data-dir 不同的磁盘； * ETCD_DATA_DIR: 数据目录 ### 2.3.2 启动etcd ``` # 第一次启动的etcd进程会卡顿，等待其他etcd节点启动进程加入集群； # 如果等待超时，则第一个etcd节点进程启动会失败 systemctl enable etcd.service systemctl restart etcd.service ``` ### 2.3.3 验证 ``` # 查看member list； # ETCDCTL_API=3：api版本； # 通信采用TLS加密，客户端访问时间需要指定对应公钥&私钥 ETCDCTL_API=3 etcdctl \ --endpoints=https://192.168.112.53:2379 \ --cacert=/etc/etcd/cert/ca.pem \ --cert=/etc/etcd/cert/etcd.pem \ --key=/etc/etcd/cert/etcd-key.pem \ member list 942d154c76172, started, master2, https://192.168.112.52:2380, https://192.168.112.52:2379 9276d361160c2692, started, master1, https://192.168.112.51:2380, https://192.168.112.51:2379 ed64cbd6dc4b3af9, started, master3, https://192.168.112.53:2380, https://192.168.112.53:2379 ``` ### 2.3.5 查看节点健康状态 ``` # 查看节点健康状态 for ip in 192.168.112.{51,52,53}; do ETCDCTL_API=3 etcdctl \ --endpoints=https://${ip}:2379 \ --cacert=/etc/etcd/cert/ca.pem \ --cert=/etc/etcd/cert/etcd.pem \ --key=/etc/etcd/cert/etcd-key.pem \ endpoint health; done https://192.168.112.51:2379 is healthy: successfully committed proposal: took = 3.169004ms https://192.168.112.52:2379 is healthy: successfully committed proposal: took = 2.786429ms https://192.168.112.53:2379 is healthy: successfully committed proposal: took = 2.216989ms ``` # 3. 防火墙 ``` vim /etc/sysconfig/iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT ``` 如果觉得我的文章对您有用，请随意赞赏。您的支持将鼓励我继续创作！ 赞赏支持
