1. 高可用Kubernetes集群架构解析在容器编排领域构建生产级高可用(HA)Kubernetes集群是每个DevOps工程师的必修课。今天我将分享基于kubeadm工具部署HA集群的完整架构设计这套方案已经在多个金融级生产环境稳定运行超过两年。高可用集群与单节点集群的核心区别在于控制平面的冗余设计。传统单Master架构存在单点故障风险当API Server不可用时整个集群的调度和管理功能将瘫痪。而HA架构通过多Master节点负载均衡的组合确保即使某个控制平面节点宕机集群仍能持续提供服务。2. 核心组件与拓扑设计2.1 基础架构组成典型的HA集群包含以下核心组件3个Master节点推荐奇数个以实现quorum2个Worker节点根据业务需求扩展负载均衡器通常采用KeepalivedHAProxy方案etcd集群可独立部署或与Master节点共存2.2 网络拓扑方案我们采用分层式网络设计[外部负载均衡器] | [Keepalived VIP] | [HAProxy Layer] → [Master Node 1 API Server] | → [Master Node 2 API Server] | → [Master Node 3 API Server] | [Worker Nodes]这种架构下客户端请求首先到达负载均衡层由HAProxy将流量分发到健康的API Server实例。Keepalived负责维护虚拟IP(VIP)的高可用当主负载均衡器故障时自动切换备用节点。3. 关键配置参数详解3.1 etcd集群配置etcd作为Kubernetes的大脑其配置直接影响集群稳定性。关键参数包括# etcd配置示例 name: etcd1 listen-peer-urls: https://192.168.1.101:2380 listen-client-urls: https://192.168.1.101:2379,https://127.0.0.1:2379 advertise-client-urls: https://192.168.1.101:2379 initial-advertise-peer-urls: https://192.168.1.101:2380 initial-cluster: etcd1https://192.168.1.101:2380,etcd2https://192.168.1.102:2380,etcd3https://192.168.1.103:2380注意etcd节点间通信必须使用TLS加密生产环境严禁使用HTTP明文协议。3.2 kubeadm配置模板kubeadm-config.yaml是部署的核心配置文件apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.26.0 controlPlaneEndpoint: k8s-api.example.com:6443 apiServer: certSANs: - k8s-api.example.com - 192.168.1.100 # VIP地址 extraArgs: audit-log-path: /var/log/kubernetes/audit.log etcd: external: endpoints: - https://192.168.1.101:2379 - https://192.168.1.102:2379 - https://192.168.1.103:2379 caFile: /etc/kubernetes/pki/etcd/ca.crt certFile: /etc/kubernetes/pki/etcd/client.crt keyFile: /etc/kubernetes/pki/etcd/client.key networking: podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/124. 部署流程关键步骤4.1 前置准备阶段系统配置标准化# 所有节点执行 swapoff -a sed -i / swap / s/^/#/ /etc/fstab setenforce 0 sed -i s/^SELINUXenforcing$/SELINUXpermissive/ /etc/selinux/config内核参数调优cat /etc/sysctl.d/k8s.conf EOF net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 net.ipv4.ip_forward 1 EOF sysctl --system4.2 负载均衡器部署HAProxy配置示例frontend k8s-api bind *:6443 mode tcp option tcplog default_backend k8s-api-servers backend k8s-api-servers mode tcp balance roundrobin option tcp-check server k8s-master-1 192.168.1.101:6443 check fall 3 rise 2 server k8s-master-2 192.168.1.102:6443 check fall 3 rise 2 server k8s-master-3 192.168.1.103:6443 check fall 3 rise 2Keepalived配置要点vrrp_script chk_haproxy { script killall -0 haproxy interval 2 weight 2 } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 priority 101 virtual_ipaddress { 192.168.1.100/24 } track_script { chk_haproxy } }5. 常见问题与解决方案5.1 证书相关错误现象kubelet报错x509: certificate signed by unknown authority排查步骤检查各节点时间是否同步确认kubeadm init时使用的证书SAN列表包含所有访问地址验证证书有效期openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -A 2 Validity5.2 网络插件问题现象Pod间网络不通解决方案确认选择的CNI插件与内核版本兼容检查网络策略是否阻止通信验证iptables规则iptables-save | grep -i calico5.3 etcd集群健康检查定期监控etcd集群状态ETCDCTL_API3 etcdctl \ --endpointshttps://192.168.1.101:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ endpoint health预期输出https://192.168.1.101:2379 is healthy: successfully committed proposal: took 12.345678ms https://192.168.1.102:2379 is healthy: successfully committed proposal: took 11.234567ms https://192.168.1.103:2379 is healthy: successfully committed proposal: took 10.123456ms6. 性能优化建议6.1 API Server调优修改kube-apiserver配置apiServer: extraArgs: default-not-ready-toleration-seconds: 30 default-unreachable-toleration-seconds: 30 event-ttl: 1h max-mutating-requests-inflight: 500 max-requests-inflight: 15006.2 kubelet资源预留确保系统关键进程有足够资源kubelet: kubeReserved: cpu: 500m memory: 500Mi ephemeral-storage: 1Gi systemReserved: cpu: 500m memory: 500Mi ephemeral-storage: 1Gi7. 升级维护策略7.1 滚动升级流程隔离第一个Master节点kubectl drain master-node-1 --ignore-daemonsets --delete-emptydir-data升级kubeadmapt-get update apt-get install -y kubeadm1.26.0-00 kubeadm upgrade apply v1.26.0解除隔离kubectl uncordon master-node-17.2 备份恢复方案定期备份etcd数据ETCDCTL_API3 etcdctl snapshot save snapshot.db \ --endpointshttps://192.168.1.101:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key恢复etcd集群ETCDCTL_API3 etcdctl snapshot restore snapshot.db \ --initial-cluster-tokenetcd-cluster-1 \ --initial-advertise-peer-urlshttps://192.168.1.101:2380 \ --nameetcd1 \ --data-dir/var/lib/etcd-from-backup
1. 高可用Kubernetes集群架构解析在容器编排领域构建生产级高可用(HA)Kubernetes集群是每个DevOps工程师的必修课。今天我将分享基于kubeadm工具部署HA集群的完整架构设计这套方案已经在多个金融级生产环境稳定运行超过两年。高可用集群与单节点集群的核心区别在于控制平面的冗余设计。传统单Master架构存在单点故障风险当API Server不可用时整个集群的调度和管理功能将瘫痪。而HA架构通过多Master节点负载均衡的组合确保即使某个控制平面节点宕机集群仍能持续提供服务。2. 核心组件与拓扑设计2.1 基础架构组成典型的HA集群包含以下核心组件3个Master节点推荐奇数个以实现quorum2个Worker节点根据业务需求扩展负载均衡器通常采用KeepalivedHAProxy方案etcd集群可独立部署或与Master节点共存2.2 网络拓扑方案我们采用分层式网络设计[外部负载均衡器] | [Keepalived VIP] | [HAProxy Layer] → [Master Node 1 API Server] | → [Master Node 2 API Server] | → [Master Node 3 API Server] | [Worker Nodes]这种架构下客户端请求首先到达负载均衡层由HAProxy将流量分发到健康的API Server实例。Keepalived负责维护虚拟IP(VIP)的高可用当主负载均衡器故障时自动切换备用节点。3. 关键配置参数详解3.1 etcd集群配置etcd作为Kubernetes的大脑其配置直接影响集群稳定性。关键参数包括# etcd配置示例 name: etcd1 listen-peer-urls: https://192.168.1.101:2380 listen-client-urls: https://192.168.1.101:2379,https://127.0.0.1:2379 advertise-client-urls: https://192.168.1.101:2379 initial-advertise-peer-urls: https://192.168.1.101:2380 initial-cluster: etcd1https://192.168.1.101:2380,etcd2https://192.168.1.102:2380,etcd3https://192.168.1.103:2380注意etcd节点间通信必须使用TLS加密生产环境严禁使用HTTP明文协议。3.2 kubeadm配置模板kubeadm-config.yaml是部署的核心配置文件apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.26.0 controlPlaneEndpoint: k8s-api.example.com:6443 apiServer: certSANs: - k8s-api.example.com - 192.168.1.100 # VIP地址 extraArgs: audit-log-path: /var/log/kubernetes/audit.log etcd: external: endpoints: - https://192.168.1.101:2379 - https://192.168.1.102:2379 - https://192.168.1.103:2379 caFile: /etc/kubernetes/pki/etcd/ca.crt certFile: /etc/kubernetes/pki/etcd/client.crt keyFile: /etc/kubernetes/pki/etcd/client.key networking: podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/124. 部署流程关键步骤4.1 前置准备阶段系统配置标准化# 所有节点执行 swapoff -a sed -i / swap / s/^/#/ /etc/fstab setenforce 0 sed -i s/^SELINUXenforcing$/SELINUXpermissive/ /etc/selinux/config内核参数调优cat /etc/sysctl.d/k8s.conf EOF net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 net.ipv4.ip_forward 1 EOF sysctl --system4.2 负载均衡器部署HAProxy配置示例frontend k8s-api bind *:6443 mode tcp option tcplog default_backend k8s-api-servers backend k8s-api-servers mode tcp balance roundrobin option tcp-check server k8s-master-1 192.168.1.101:6443 check fall 3 rise 2 server k8s-master-2 192.168.1.102:6443 check fall 3 rise 2 server k8s-master-3 192.168.1.103:6443 check fall 3 rise 2Keepalived配置要点vrrp_script chk_haproxy { script killall -0 haproxy interval 2 weight 2 } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 priority 101 virtual_ipaddress { 192.168.1.100/24 } track_script { chk_haproxy } }5. 常见问题与解决方案5.1 证书相关错误现象kubelet报错x509: certificate signed by unknown authority排查步骤检查各节点时间是否同步确认kubeadm init时使用的证书SAN列表包含所有访问地址验证证书有效期openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -A 2 Validity5.2 网络插件问题现象Pod间网络不通解决方案确认选择的CNI插件与内核版本兼容检查网络策略是否阻止通信验证iptables规则iptables-save | grep -i calico5.3 etcd集群健康检查定期监控etcd集群状态ETCDCTL_API3 etcdctl \ --endpointshttps://192.168.1.101:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ endpoint health预期输出https://192.168.1.101:2379 is healthy: successfully committed proposal: took 12.345678ms https://192.168.1.102:2379 is healthy: successfully committed proposal: took 11.234567ms https://192.168.1.103:2379 is healthy: successfully committed proposal: took 10.123456ms6. 性能优化建议6.1 API Server调优修改kube-apiserver配置apiServer: extraArgs: default-not-ready-toleration-seconds: 30 default-unreachable-toleration-seconds: 30 event-ttl: 1h max-mutating-requests-inflight: 500 max-requests-inflight: 15006.2 kubelet资源预留确保系统关键进程有足够资源kubelet: kubeReserved: cpu: 500m memory: 500Mi ephemeral-storage: 1Gi systemReserved: cpu: 500m memory: 500Mi ephemeral-storage: 1Gi7. 升级维护策略7.1 滚动升级流程隔离第一个Master节点kubectl drain master-node-1 --ignore-daemonsets --delete-emptydir-data升级kubeadmapt-get update apt-get install -y kubeadm1.26.0-00 kubeadm upgrade apply v1.26.0解除隔离kubectl uncordon master-node-17.2 备份恢复方案定期备份etcd数据ETCDCTL_API3 etcdctl snapshot save snapshot.db \ --endpointshttps://192.168.1.101:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key恢复etcd集群ETCDCTL_API3 etcdctl snapshot restore snapshot.db \ --initial-cluster-tokenetcd-cluster-1 \ --initial-advertise-peer-urlshttps://192.168.1.101:2380 \ --nameetcd1 \ --data-dir/var/lib/etcd-from-backup
