Etcd+TLS集群部署 作者: sysit 分类: d 发表于 2019-03-14 249人围观 etcd是key-value存储（同zookeeper），在整个kubernetes集群中处于中心数据库地位，以集群的方式部署，可有效避免单点故障。 这里采用静态配置的方式部署（另也可通过etcd提供的rest api在运行时动态添加，修改或删除集群成员）。 # 1. 主机 ``` 192.168.112.51 master1.sysit.cn master1 192.168.112.52 master2.sysit.cn master2 192.168.112.53 master3.sysit.cn master3 ``` ## 2. 安装配置etcd ### 2.1 yum安装 ``` # 三台上都安装 yum install etcd ``` ### 2.2 二进制安装 * 安装并分发 ``` mkdir /usr/local/etcd wget https://github.com/etcd-io/etcd/releases/download/v3.4.16/etcd-v3.4.16-linux-amd64.tar.gz tar xzvf etcd-v3.4.16-linux-amd64.tar.gz -C /usr/local/etcd --strip-components=1 chmod +x /usr/local/etcd/etcd* echo "PATH=$PATH:/usr/local/etcd" >>/etc/bashrc scp -r /usr/local/etcd root@master2:/usr/local/ scp -r /usr/local/etcd root@master3:/usr/local ``` * systemd ``` cat >/usr/lib/systemd/system/etcd.service <<'EOF' [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify WorkingDirectory=/var/lib/etcd/ EnvironmentFile=-/etc/etcd/etcd.conf User=etcd # set GOMAXPROCS to number of processors ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/etcd/etcd" Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF ``` * 用户和目录 ``` groupadd etcd useradd -c "Etcd user" -g etcd -s /sbin/nologin -r etcd mkdir /var/lib/etcd/default.etcd -p mkdir /etc/etcd chown -R etcd:etcd /var/lib/etcd/default.etcd chown -R etcd:etcd /etc/etcd ``` ## 3. etcd TLS证书和私钥 ### 3.1 cfssl工具 本文档采用 `CloudFlare` 的 `PKI` 工具集 `cfssl` 来生成 `Certificate Authority (CA)` 证书和秘钥文件，`CA` 是自签名的证书，用来签名后续创建的其它 `TLS `证书。 ``` mkdir -p /usr/local/cfssl/bin cd /usr/local/cfssl/bin wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl_1.5.0_linux_amd64 -O cfssl wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl-certinfo_1.5.0_linux_amd64 -O cfssl-certinfo wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssljson_1.5.0_linux_amd64 -O cfssljson chmod +x cfssl cfssl-certinfo cfssljson echo 'export PATH=$PATH:/usr/local/cfssl/bin' >>/etc/bashrc ``` ### 3.2 CA配置文件 ``` # ca-config.json：1个profiles，分别指定不同的过期时间，使用场景等参数，根据需要在不同场景使用不同的profile签名证书；这里以生成的模板为基础修改； cat > ca-config.json <<EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "etcd": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } EOF ``` * 字段说明: * ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile； * signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE； * server auth"：表示client可以用该 CA 对server提供的证书进行验证； * client auth"：表示server可以用该CA对client提供的证书进行验证； ### 3.3 CA证书签名请求 ``` cat > ca-csr.json <<EOF { "CN": "etcd", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Chengdu", "L": "Chengdu", "O": "etcd", "OU": "System" } ] } EOF ``` ### 3.4 生成CA证书与秘钥 ``` cfssl gencert -initca ca-csr.json | cfssljson -bare ca ls ca* ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem #简单查看 cfssl-certinfo -cert ca.pem ``` ### 3.5 创建etcd TLS证书与私钥 > 客户端（etcdctl）与etcd集群，etcd集群之间通信采用TLS加密。 ``` # hosts 字段指定授权使用该证书的 etcd 节点 IP 或域名列表，这里将 etcd 集群的三个节点 IP 都列在其中； cat > etcd-csr.json <<'EOF' { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.112.140", "192.168.112.141", "192.168.112.142", "192.168.112.143" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Chengdu", "L": "Chengdu", "O": "etcd", "OU": "System" } ] } EOF # 生成etcd证书与私钥 cfssl gencert -ca=ca.pem \ -ca-key=ca-key.pem \ -config=ca-config.json \ -profile=etcd etcd-csr.json | cfssljson -bare etcd ``` ### 3.6 分发证书和私钥 ``` mkdir -p /etc/etcd/cert cp etcd*pem /etc/etcd/cert/ cp ca.pem /etc/etcd/cert/ chown -R etcd:etcd /etc/etcd/cert scp -r /etc/etcd/cert root@master2:/etc/etcd/cert ssh root@master2 "chown -R etcd:etcd /etc/etcd/cert" scp -r /etc/etcd/cert root@master3:/etc/etcd/cert ssh root@master3 "chown -R etcd:etcd /etc/etcd/cert" # 分发ca.pem # ca.pem必须存在并分发。 ``` ## 4. 配置etcd的配置文件 ``` # master1 [root@master1 ~]# egrep -v "^#|^$" /etc/etcd/etcd.conf ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.112.141:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.112.141:2379,http://127.0.0.1:2379" ETCD_NAME="master1" ETCD_HEARTBEAT_INTERVAL="200" ETCD_ELECTION_TIMEOUT="2000" ETCD_QUOTA_BACKEND_BYTES="6442450944" ETCD_MAX_REQUEST_BYTES="33554432" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.112.141:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.112.141:2379" ETCD_INITIAL_CLUSTER="master1=https://192.168.112.141:2380,master2=https://192.168.112.142:2380,master3=https://192.168.112.143:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-0" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem" ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/cert/ca.pem" ETCD_AUTO_TLS="true" ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem" ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/cert/ca.pem" ETCD_PEER_AUTO_TLS="true" # master2 [root@master2 ~]# egrep -v "^#|^$" /etc/etcd/etcd.conf ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.112.142:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.112.142:2379,http://127.0.0.1:2379" ETCD_NAME="master2" ETCD_HEARTBEAT_INTERVAL="200" ETCD_ELECTION_TIMEOUT="2000" ETCD_QUOTA_BACKEND_BYTES="6442450944" ETCD_MAX_REQUEST_BYTES="33554432" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.112.142:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.112.142:2379" ETCD_INITIAL_CLUSTER="master1=https://192.168.112.141:2380,master2=https://192.168.112.142:2380,master3=https://192.168.112.143:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-0" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem" ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/cert/ca.pem" ETCD_AUTO_TLS="true" ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem" ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/cert/ca.pem" ETCD_PEER_AUTO_TLS="true" # k8s-master3 [root@master3 ~]# egrep -v "^#|^$" /etc/etcd/etcd.conf ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.112.143:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.112.143:2379,http://127.0.0.1:2379" ETCD_NAME="master3" ETCD_HEARTBEAT_INTERVAL="200" ETCD_ELECTION_TIMEOUT="2000" ETCD_QUOTA_BACKEND_BYTES="6442450944" ETCD_MAX_REQUEST_BYTES="33554432" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.112.143:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.112.143:2379" ETCD_INITIAL_CLUSTER="master1=https://192.168.112.141:2380,master2=https://192.168.112.142:2380,master3=https://192.168.112.143:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-0" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem" ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/cert/ca.pem" ETCD_AUTO_TLS="true" ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem" ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/cert/ca.pem" ETCD_PEER_AUTO_TLS="true" ``` * ETCD_CERT_FILE，ETCD_KEY_FILE：etcd server 与 client 通信时使用的证书和私钥； * ETCD_PEER_CERT_FILE，ETCD_PEER_KEY_FILE：分别指定etcd的peers通信的公钥证书和私钥； * ETCD_TRUSTED_CA_FILE：签名 client 证书的 CA 证书，用于验证 client 证书； * ETCD_PEER_TRUSTED_CA_FILE：指定签名 peer 证书的 CA 证书，用于验证 peer 证书； * ETCD_NAME：指定节点名称，当 ETCD_INITIAL_CLUSTER_STATE 值为 new 时，ETCD_NAME 的参数值必须位于 ETCD_INITIAL_CLUSTER 列表中； * ETCD_WAL_DIR：指定 wal 目录，为了提高性能，一般使用 SSD 或者和 --data-dir 不同的磁盘； * ETCD_DATA_DIR: 数据目录 ### 2.3.2 启动etcd ``` # 第一次启动的etcd进程会卡顿，等待其他etcd节点启动进程加入集群； # 如果等待超时，则第一个etcd节点进程启动会失败 chown -R etcd:etcd /var/lib/etcd chown -R etcd:etcd /etc/etcd systemctl enable etcd.service systemctl daemon-reload systemctl restart etcd.service ``` ### 2.3.3 验证 ``` # 查看member list； # ETCDCTL_API=3：api版本； # 通信采用TLS加密，客户端访问时间需要指定对应公钥&私钥 ETCDCTL_API=3 etcdctl \ --endpoints=https://192.168.112.143:2379 \ --cacert=/etc/etcd/cert/ca.pem \ --cert=/etc/etcd/cert/etcd.pem \ --key=/etc/etcd/cert/etcd-key.pem \ member list 51cdba47802fcd2a, started, master1, https://192.168.112.141:2380, https://192.168.112.141:2379, false 8d410a193af0e9b1, started, master2, https://192.168.112.142:2380, https://192.168.112.142:2379, false cf1e2e7c1ec51759, started, master3, https://192.168.112.143:2380, https://192.168.112.143:2379, false ``` ### 2.3.5 查看节点健康状态 ``` # 查看节点健康状态 for ip in 192.168.112.{141,142,143}; do ETCDCTL_API=3 etcdctl \ --endpoints=https://${ip}:2379 \ --cacert=/etc/etcd/cert/ca.pem \ --cert=/etc/etcd/cert/etcd.pem \ --key=/etc/etcd/cert/etcd-key.pem \ endpoint health; done https://192.168.112.141:2379 is healthy: successfully committed proposal: took = 3.169004ms https://192.168.112.142:2379 is healthy: successfully committed proposal: took = 2.786429ms https://192.168.112.143:2379 is healthy: successfully committed proposal: took = 2.216989ms ``` # 3. 防火墙 ``` vim /etc/sysconfig/iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT ``` 如果觉得我的文章对您有用，请随意赞赏。您的支持将鼓励我继续创作！ 赞赏支持
