1. 项目概述在当今企业级IT基础设施中Kubernetes已成为容器编排的事实标准。而基于Rocky Linux的多Master高可用集群部署方案正逐渐成为金融、电信、制造业等关键业务场景的首选架构。这种部署模式不仅继承了CentOS的稳定基因还通过多Master节点实现了控制平面的冗余确保集群在单个甚至多个Master节点故障时仍能持续提供服务。我最近为一个中型电商平台完成了这样的生产环境部署整个集群需要支撑日均300万订单的业务流量。与常见的单Master或简易多节点部署不同生产级高可用Kubernetes对网络配置、证书管理、负载均衡等方面都有严格要求。本文将分享从零开始构建这样一个集群的完整过程包括我在实际部署中积累的各类避坑经验。2. 架构设计与核心组件2.1 高可用架构拓扑典型的多Master高可用Kubernetes集群包含以下核心组件3个Master节点推荐奇数个以实现etcd集群的选举共识2-3个Worker节点根据业务负载可横向扩展1个负载均衡器通常采用HAProxyKeepalived方案独立的etcd集群可与Master节点共置或独立部署[客户端] | [负载均衡器 VIP] | [Master1] [Master2] [Master3] |_______|_______| | [Worker1] [Worker2] [Worker3]2.2 关键组件选型操作系统Rocky Linux 8.6与RHEL完全兼容且提供长期支持容器运行时containerd 1.6比Docker更轻量且符合K8s标准网络插件Calico 3.24支持网络策略和BGP路由存储方案本地NVMe SSD Ceph RBD根据业务需求混合部署监控系统Prometheus-Operator Grafana内置K8s指标采集重要提示生产环境务必避免使用Docker作为运行时kubelet从1.24版本起已移除dockershim支持3. 系统准备与基础环境配置3.1 操作系统初始化在所有节点执行以下基础配置# 禁用SELinuxK8s组件兼容性问题主要来源 sed -i s/SELINUXenforcing/SELINUXdisabled/g /etc/selinux/config setenforce 0 # 关闭防火墙由网络插件管理规则 systemctl stop firewalld systemctl disable firewalld # 配置内核参数 cat EOF | tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 net.ipv4.ip_forward 1 vm.swappiness 0 EOF sysctl --system # 安装基础工具 dnf install -y yum-utils device-mapper-persistent-data lvm2 git curl conntrack-tools3.2 容器运行时安装配置containerd作为容器运行时# 安装containerd dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo dnf install -y containerd.io # 生成默认配置并启用SystemdCgroup containerd config default | tee /etc/containerd/config.toml sed -i s/SystemdCgroup false/SystemdCgroup true/g /etc/containerd/config.toml # 启动服务 systemctl enable --now containerd4. Kubernetes集群部署实战4.1 负载均衡器配置使用HAProxyKeepalived实现Master节点的高可用访问# 在LB节点安装软件包 dnf install -y haproxy keepalived # HAProxy配置示例/etc/haproxy/haproxy.cfg frontend k8s-api bind *:6443 mode tcp default_backend k8s-masters backend k8s-masters mode tcp balance roundrobin server master1 192.168.1.101:6443 check server master2 192.168.1.102:6443 check server master3 192.168.1.103:6443 check # Keepalived配置/etc/keepalived/keepalived.conf vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.1.100/24 } }4.2 使用kubeadm初始化集群在首个Master节点执行初始化# 安装kubeadm/kubelet/kubectl cat EOF | tee /etc/yum.repos.d/kubernetes.repo [kubernetes] nameKubernetes baseurlhttps://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled1 gpgcheck1 repo_gpgcheck1 gpgkeyhttps://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF dnf install -y kubelet kubeadm kubectl --disableexcludeskubernetes systemctl enable --now kubelet # 初始化控制平面 kubeadm init \ --control-plane-endpoint 192.168.1.100:6443 \ --upload-certs \ --pod-network-cidr172.16.0.0/16 \ --service-cidr10.96.0.0/12 \ --image-repository registry.aliyuncs.com/google_containers初始化完成后按照提示在其他Master节点执行join命令kubeadm join 192.168.1.100:6443 \ --token token \ --discovery-token-ca-cert-hash sha256:hash \ --control-plane \ --certificate-key key4.3 网络插件安装部署Calico网络插件kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml # 验证网络状态 kubectl get pods -n kube-system -l k8s-appcalico-node5. 生产环境关键配置5.1 证书管理高可用集群需要特别注意证书配置# 检查证书有效期 kubeadm certs check-expiration # 手动更新证书在证书到期前 kubeadm certs renew all5.2 集群备份策略使用etcdctl进行定期备份# 创建备份 ETCDCTL_API3 etcdctl \ --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ snapshot save /data/etcd-snapshot-$(date %Y%m%d).db # 恢复备份灾难恢复时 kubeadm reset ETCDCTL_API3 etcdctl snapshot restore /data/etcd-snapshot.db \ --data-dir /var/lib/etcd5.3 节点维护策略优雅驱逐Pod的两种方式# 方式1主动驱逐 kubectl drain node-name --ignore-daemonsets --delete-emptydir-data # 方式2节点维护标记 kubectl cordon node-name6. 常见问题排查6.1 节点NotReady状态排查检查流程确认kubelet服务状态systemctl status kubelet检查容器运行时crictl ps -a查看kubelet日志journalctl -xeu kubelet验证网络插件状态kubectl get pods -n kube-system6.2 API Server无法访问典型原因及解决负载均衡器故障检查HAProxy/Keepalived日志证书过期使用kubeadm certs check-expiration验证网络策略阻止临时禁用Calico网络策略测试6.3 etcd集群健康检查ETCDCTL_API3 etcdctl \ --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ endpoint health7. 性能优化建议7.1 内核参数调优# 增加连接跟踪表大小 echo net.netfilter.nf_conntrack_max1048576 /etc/sysctl.conf # 优化TCP协议栈 cat EOF /etc/sysctl.conf net.core.somaxconn 32768 net.ipv4.tcp_tw_reuse 1 net.ipv4.tcp_fin_timeout 30 EOF sysctl -p7.2 kubelet资源配置调整/var/lib/kubelet/config.yamlapiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration maxPods: 150 kubeReserved: cpu: 500m memory: 1Gi ephemeral-storage: 5Gi systemReserved: cpu: 500m memory: 1Gi ephemeral-storage: 5Gi evictionHard: memory.available: 500Mi nodefs.available: 10%7.3 监控与告警配置推荐监控指标API Server延迟apiserver_request_duration_secondsetcd写入性能etcd_disk_wal_fsync_duration_seconds节点内存压力node_memory_MemAvailable_bytes8. 安全加固措施8.1 RBAC权限控制创建最小权限ServiceAccount示例apiVersion: v1 kind: ServiceAccount metadata: name: ci-deployer namespace: dev --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: dev name: deployer-role rules: - apiGroups: [apps] resources: [deployments] verbs: [get, list, watch, create, update, patch] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: deployer-binding namespace: dev subjects: - kind: ServiceAccount name: ci-deployer namespace: dev roleRef: kind: Role name: deployer-role apiGroup: rbac.authorization.k8s.io8.2 Pod安全策略启用PodSecurity admissionapiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: PodSecurity configuration: apiVersion: pod-security.admission.config.k8s.io/v1 kind: PodSecurityConfiguration defaults: enforce: restricted enforce-version: latest exemptions: usernames: [] runtimeClasses: [] namespaces: [kube-system]8.3 网络策略示例限制default命名空间Pod的入口流量apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-ingress namespace: default spec: podSelector: {} policyTypes: - Ingress在实际部署过程中我发现很多问题都源于证书配置不当或网络插件冲突。特别是在Rocky Linux上需要特别注意SELinux策略与容器运行时的兼容性问题。建议在正式上线前至少进行两次完整的集群销毁和重建演练确保团队熟悉灾难恢复流程。
1. 项目概述在当今企业级IT基础设施中Kubernetes已成为容器编排的事实标准。而基于Rocky Linux的多Master高可用集群部署方案正逐渐成为金融、电信、制造业等关键业务场景的首选架构。这种部署模式不仅继承了CentOS的稳定基因还通过多Master节点实现了控制平面的冗余确保集群在单个甚至多个Master节点故障时仍能持续提供服务。我最近为一个中型电商平台完成了这样的生产环境部署整个集群需要支撑日均300万订单的业务流量。与常见的单Master或简易多节点部署不同生产级高可用Kubernetes对网络配置、证书管理、负载均衡等方面都有严格要求。本文将分享从零开始构建这样一个集群的完整过程包括我在实际部署中积累的各类避坑经验。2. 架构设计与核心组件2.1 高可用架构拓扑典型的多Master高可用Kubernetes集群包含以下核心组件3个Master节点推荐奇数个以实现etcd集群的选举共识2-3个Worker节点根据业务负载可横向扩展1个负载均衡器通常采用HAProxyKeepalived方案独立的etcd集群可与Master节点共置或独立部署[客户端] | [负载均衡器 VIP] | [Master1] [Master2] [Master3] |_______|_______| | [Worker1] [Worker2] [Worker3]2.2 关键组件选型操作系统Rocky Linux 8.6与RHEL完全兼容且提供长期支持容器运行时containerd 1.6比Docker更轻量且符合K8s标准网络插件Calico 3.24支持网络策略和BGP路由存储方案本地NVMe SSD Ceph RBD根据业务需求混合部署监控系统Prometheus-Operator Grafana内置K8s指标采集重要提示生产环境务必避免使用Docker作为运行时kubelet从1.24版本起已移除dockershim支持3. 系统准备与基础环境配置3.1 操作系统初始化在所有节点执行以下基础配置# 禁用SELinuxK8s组件兼容性问题主要来源 sed -i s/SELINUXenforcing/SELINUXdisabled/g /etc/selinux/config setenforce 0 # 关闭防火墙由网络插件管理规则 systemctl stop firewalld systemctl disable firewalld # 配置内核参数 cat EOF | tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 net.ipv4.ip_forward 1 vm.swappiness 0 EOF sysctl --system # 安装基础工具 dnf install -y yum-utils device-mapper-persistent-data lvm2 git curl conntrack-tools3.2 容器运行时安装配置containerd作为容器运行时# 安装containerd dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo dnf install -y containerd.io # 生成默认配置并启用SystemdCgroup containerd config default | tee /etc/containerd/config.toml sed -i s/SystemdCgroup false/SystemdCgroup true/g /etc/containerd/config.toml # 启动服务 systemctl enable --now containerd4. Kubernetes集群部署实战4.1 负载均衡器配置使用HAProxyKeepalived实现Master节点的高可用访问# 在LB节点安装软件包 dnf install -y haproxy keepalived # HAProxy配置示例/etc/haproxy/haproxy.cfg frontend k8s-api bind *:6443 mode tcp default_backend k8s-masters backend k8s-masters mode tcp balance roundrobin server master1 192.168.1.101:6443 check server master2 192.168.1.102:6443 check server master3 192.168.1.103:6443 check # Keepalived配置/etc/keepalived/keepalived.conf vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.1.100/24 } }4.2 使用kubeadm初始化集群在首个Master节点执行初始化# 安装kubeadm/kubelet/kubectl cat EOF | tee /etc/yum.repos.d/kubernetes.repo [kubernetes] nameKubernetes baseurlhttps://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled1 gpgcheck1 repo_gpgcheck1 gpgkeyhttps://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF dnf install -y kubelet kubeadm kubectl --disableexcludeskubernetes systemctl enable --now kubelet # 初始化控制平面 kubeadm init \ --control-plane-endpoint 192.168.1.100:6443 \ --upload-certs \ --pod-network-cidr172.16.0.0/16 \ --service-cidr10.96.0.0/12 \ --image-repository registry.aliyuncs.com/google_containers初始化完成后按照提示在其他Master节点执行join命令kubeadm join 192.168.1.100:6443 \ --token token \ --discovery-token-ca-cert-hash sha256:hash \ --control-plane \ --certificate-key key4.3 网络插件安装部署Calico网络插件kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml # 验证网络状态 kubectl get pods -n kube-system -l k8s-appcalico-node5. 生产环境关键配置5.1 证书管理高可用集群需要特别注意证书配置# 检查证书有效期 kubeadm certs check-expiration # 手动更新证书在证书到期前 kubeadm certs renew all5.2 集群备份策略使用etcdctl进行定期备份# 创建备份 ETCDCTL_API3 etcdctl \ --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ snapshot save /data/etcd-snapshot-$(date %Y%m%d).db # 恢复备份灾难恢复时 kubeadm reset ETCDCTL_API3 etcdctl snapshot restore /data/etcd-snapshot.db \ --data-dir /var/lib/etcd5.3 节点维护策略优雅驱逐Pod的两种方式# 方式1主动驱逐 kubectl drain node-name --ignore-daemonsets --delete-emptydir-data # 方式2节点维护标记 kubectl cordon node-name6. 常见问题排查6.1 节点NotReady状态排查检查流程确认kubelet服务状态systemctl status kubelet检查容器运行时crictl ps -a查看kubelet日志journalctl -xeu kubelet验证网络插件状态kubectl get pods -n kube-system6.2 API Server无法访问典型原因及解决负载均衡器故障检查HAProxy/Keepalived日志证书过期使用kubeadm certs check-expiration验证网络策略阻止临时禁用Calico网络策略测试6.3 etcd集群健康检查ETCDCTL_API3 etcdctl \ --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ endpoint health7. 性能优化建议7.1 内核参数调优# 增加连接跟踪表大小 echo net.netfilter.nf_conntrack_max1048576 /etc/sysctl.conf # 优化TCP协议栈 cat EOF /etc/sysctl.conf net.core.somaxconn 32768 net.ipv4.tcp_tw_reuse 1 net.ipv4.tcp_fin_timeout 30 EOF sysctl -p7.2 kubelet资源配置调整/var/lib/kubelet/config.yamlapiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration maxPods: 150 kubeReserved: cpu: 500m memory: 1Gi ephemeral-storage: 5Gi systemReserved: cpu: 500m memory: 1Gi ephemeral-storage: 5Gi evictionHard: memory.available: 500Mi nodefs.available: 10%7.3 监控与告警配置推荐监控指标API Server延迟apiserver_request_duration_secondsetcd写入性能etcd_disk_wal_fsync_duration_seconds节点内存压力node_memory_MemAvailable_bytes8. 安全加固措施8.1 RBAC权限控制创建最小权限ServiceAccount示例apiVersion: v1 kind: ServiceAccount metadata: name: ci-deployer namespace: dev --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: dev name: deployer-role rules: - apiGroups: [apps] resources: [deployments] verbs: [get, list, watch, create, update, patch] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: deployer-binding namespace: dev subjects: - kind: ServiceAccount name: ci-deployer namespace: dev roleRef: kind: Role name: deployer-role apiGroup: rbac.authorization.k8s.io8.2 Pod安全策略启用PodSecurity admissionapiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: PodSecurity configuration: apiVersion: pod-security.admission.config.k8s.io/v1 kind: PodSecurityConfiguration defaults: enforce: restricted enforce-version: latest exemptions: usernames: [] runtimeClasses: [] namespaces: [kube-system]8.3 网络策略示例限制default命名空间Pod的入口流量apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-ingress namespace: default spec: podSelector: {} policyTypes: - Ingress在实际部署过程中我发现很多问题都源于证书配置不当或网络插件冲突。特别是在Rocky Linux上需要特别注意SELinux策略与容器运行时的兼容性问题。建议在正式上线前至少进行两次完整的集群销毁和重建演练确保团队熟悉灾难恢复流程。
