1. 项目背景与核心价值去年在金融行业做容器化改造时我亲历了单节点Kubernetes集群故障导致的业务中断事故。这次经历让我深刻认识到生产环境必须实现控制平面和工作节点的双重高可用。本文将基于Ubuntu 22.04 LTS和Kubernetes 1.28版本完整演示如何构建支持自动故障转移的集群架构并通过Ingress实现七层流量调度。这个方案特别适合需要满足SLA 99.9%以上的在线业务场景。相比社区常见的单Master方案我们的架构具有三个关键改进点使用KeepalivedHaproxy实现API Server负载均衡采用etcd集群分离部署模式通过Ingress Controller实现智能流量分发2. 基础环境准备2.1 硬件资源配置建议生产环境最低配置要求节点类型CPU内存磁盘数量Control Plane4核8GB100GB3Worker Node8核16GB200GB至少2Load Balancer2核4GB20GB2实际测试中发现etcd节点务必使用SSD磁盘机械硬盘在选举时会出现明显的超时问题2.2 系统初始化配置在所有节点执行# 关闭swap sudo swapoff -a sudo sed -i / swap / s/^/#/ /etc/fstab # 设置内核参数 cat EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF sudo modprobe overlay sudo modprobe br_netfilter # 设置sysctl cat EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 EOF sudo sysctl --system3. 高可用控制平面部署3.1 KeepalivedHaproxy实现负载均衡在lb01和lb02节点安装配置sudo apt install -y keepalived haproxyHaproxy配置示例/etc/haproxy/haproxy.cfgfrontend k8s-api bind *:6443 mode tcp default_backend k8s-api backend k8s-api mode tcp balance roundrobin option tcp-check server k8s-master-1 192.168.1.101:6443 check fall 3 rise 2 server k8s-master-2 192.168.1.102:6443 check fall 3 rise 2 server k8s-master-3 192.168.1.103:6443 check fall 3 rise 2Keepalived配置关键点vrrp_script chk_haproxy { script killall -0 haproxy interval 2 weight 2 } vrrp_instance VI_1 { interface ens160 virtual_router_id 51 priority 100 # 另一节点设为90 virtual_ipaddress { 192.168.1.100/24 } }3.2 使用kubeadm初始化集群首个控制平面节点执行sudo kubeadm init \ --control-plane-endpoint 192.168.1.100:6443 \ --upload-certs \ --pod-network-cidr10.244.0.0/16 \ --service-cidr10.96.0.0/12 \ --image-repository registry.aliyuncs.com/google_containers其他控制平面节点加入命令sudo kubeadm join 192.168.1.100:6443 \ --token token \ --discovery-token-ca-cert-hash hash \ --control-plane \ --certificate-key key4. Ingress高可用实现4.1 部署Ingress Controller使用Helm安装Nginx Ingresshelm upgrade --install ingress-nginx ingress-nginx \ --repo https://kubernetes.github.io/ingress-nginx \ --namespace ingress-nginx --create-namespace \ --set controller.replicaCount3 \ --set controller.nodeSelector.kubernetes\.io/oslinux \ --set controller.admissionWebhooks.patch.nodeSelector.kubernetes\.io/oslinux4.2 配置外部负载均衡在公有云环境建议使用云厂商的LB服务本地数据中心可继续用Keepalivedfrontend http-https bind *:80 bind *:443 mode tcp default_backend ingress-nginx backend ingress-nginx mode tcp balance leastconn server ingress-1 10.244.1.10:80 check server ingress-2 10.244.2.10:80 check server ingress-3 10.244.3.10:80 check5. 关键验证与故障排查5.1 高可用测试验证API Server故障转移测试# 持续访问测试 while true; do kubectl get nodes; sleep 1; done # 随机关闭Master节点服务 sudo systemctl stop kube-apiserverIngress流量切换验证# 使用ab进行压力测试 ab -n 10000 -c 100 http://example.com/ # 模拟节点故障 kubectl cordon ingress-node kubectl drain ingress-node --ignore-daemonsets5.2 常见问题处理etcd集群健康状态检查ETCDCTL_API3 etcdctl \ --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ endpoint health证书过期问题预防# 检查证书有效期 kubeadm certs check-expiration # 更新证书 kubeadm certs renew all6. 生产环境优化建议网络性能调优# 调整conntrack参数 echo 1200000 /proc/sys/net/netfilter/nf_conntrack_max echo 600 /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_establishedIngress Controller优化配置controller: config: worker-processes: 4 upstream-keepalive-connections: 200 keep-alive: 75s resources: requests: cpu: 500m memory: 512Mi limits: cpu: 2000m memory: 2048Mi监控指标采集# 安装kube-prometheus-stack helm install prometheus prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValuesfalse在金融行业生产环境落地这个方案时我们发现Ingress Controller的HPA配置需要特别注意当QPS超过5000时默认的CPU指标扩容会滞后建议增加自定义的QPS指标触发扩容。另外etcd的定期碎片整理也必不可少建议每月通过etcdctl defrag命令维护一次。
1. 项目背景与核心价值去年在金融行业做容器化改造时我亲历了单节点Kubernetes集群故障导致的业务中断事故。这次经历让我深刻认识到生产环境必须实现控制平面和工作节点的双重高可用。本文将基于Ubuntu 22.04 LTS和Kubernetes 1.28版本完整演示如何构建支持自动故障转移的集群架构并通过Ingress实现七层流量调度。这个方案特别适合需要满足SLA 99.9%以上的在线业务场景。相比社区常见的单Master方案我们的架构具有三个关键改进点使用KeepalivedHaproxy实现API Server负载均衡采用etcd集群分离部署模式通过Ingress Controller实现智能流量分发2. 基础环境准备2.1 硬件资源配置建议生产环境最低配置要求节点类型CPU内存磁盘数量Control Plane4核8GB100GB3Worker Node8核16GB200GB至少2Load Balancer2核4GB20GB2实际测试中发现etcd节点务必使用SSD磁盘机械硬盘在选举时会出现明显的超时问题2.2 系统初始化配置在所有节点执行# 关闭swap sudo swapoff -a sudo sed -i / swap / s/^/#/ /etc/fstab # 设置内核参数 cat EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF sudo modprobe overlay sudo modprobe br_netfilter # 设置sysctl cat EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 EOF sudo sysctl --system3. 高可用控制平面部署3.1 KeepalivedHaproxy实现负载均衡在lb01和lb02节点安装配置sudo apt install -y keepalived haproxyHaproxy配置示例/etc/haproxy/haproxy.cfgfrontend k8s-api bind *:6443 mode tcp default_backend k8s-api backend k8s-api mode tcp balance roundrobin option tcp-check server k8s-master-1 192.168.1.101:6443 check fall 3 rise 2 server k8s-master-2 192.168.1.102:6443 check fall 3 rise 2 server k8s-master-3 192.168.1.103:6443 check fall 3 rise 2Keepalived配置关键点vrrp_script chk_haproxy { script killall -0 haproxy interval 2 weight 2 } vrrp_instance VI_1 { interface ens160 virtual_router_id 51 priority 100 # 另一节点设为90 virtual_ipaddress { 192.168.1.100/24 } }3.2 使用kubeadm初始化集群首个控制平面节点执行sudo kubeadm init \ --control-plane-endpoint 192.168.1.100:6443 \ --upload-certs \ --pod-network-cidr10.244.0.0/16 \ --service-cidr10.96.0.0/12 \ --image-repository registry.aliyuncs.com/google_containers其他控制平面节点加入命令sudo kubeadm join 192.168.1.100:6443 \ --token token \ --discovery-token-ca-cert-hash hash \ --control-plane \ --certificate-key key4. Ingress高可用实现4.1 部署Ingress Controller使用Helm安装Nginx Ingresshelm upgrade --install ingress-nginx ingress-nginx \ --repo https://kubernetes.github.io/ingress-nginx \ --namespace ingress-nginx --create-namespace \ --set controller.replicaCount3 \ --set controller.nodeSelector.kubernetes\.io/oslinux \ --set controller.admissionWebhooks.patch.nodeSelector.kubernetes\.io/oslinux4.2 配置外部负载均衡在公有云环境建议使用云厂商的LB服务本地数据中心可继续用Keepalivedfrontend http-https bind *:80 bind *:443 mode tcp default_backend ingress-nginx backend ingress-nginx mode tcp balance leastconn server ingress-1 10.244.1.10:80 check server ingress-2 10.244.2.10:80 check server ingress-3 10.244.3.10:80 check5. 关键验证与故障排查5.1 高可用测试验证API Server故障转移测试# 持续访问测试 while true; do kubectl get nodes; sleep 1; done # 随机关闭Master节点服务 sudo systemctl stop kube-apiserverIngress流量切换验证# 使用ab进行压力测试 ab -n 10000 -c 100 http://example.com/ # 模拟节点故障 kubectl cordon ingress-node kubectl drain ingress-node --ignore-daemonsets5.2 常见问题处理etcd集群健康状态检查ETCDCTL_API3 etcdctl \ --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ endpoint health证书过期问题预防# 检查证书有效期 kubeadm certs check-expiration # 更新证书 kubeadm certs renew all6. 生产环境优化建议网络性能调优# 调整conntrack参数 echo 1200000 /proc/sys/net/netfilter/nf_conntrack_max echo 600 /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_establishedIngress Controller优化配置controller: config: worker-processes: 4 upstream-keepalive-connections: 200 keep-alive: 75s resources: requests: cpu: 500m memory: 512Mi limits: cpu: 2000m memory: 2048Mi监控指标采集# 安装kube-prometheus-stack helm install prometheus prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValuesfalse在金融行业生产环境落地这个方案时我们发现Ingress Controller的HPA配置需要特别注意当QPS超过5000时默认的CPU指标扩容会滞后建议增加自定义的QPS指标触发扩容。另外etcd的定期碎片整理也必不可少建议每月通过etcdctl defrag命令维护一次。
