简介本资源是一套面向国产化信创环境的Kubernetes高可用部署实践合集专为ARM架构下Kylin V10操作系统用户设计解决在无内置etcd、依赖外部etcd集群场景中使用containerd容器运行时部署K8s 1.26.15一主多从的核心难题。资源共41个文件涵盖16个预编译二进制压缩包含kube-apiserver、etcd-v3.5.10、calico-cni等、11个适配ARM的RPM依赖包如libseccomp、ipvsadm、sysstat、4个关键脚本load_images.sh/get_images.sh等、3个YAML配置模板kubeadm-config.yaml、calico.yaml等及systemd服务、conf、kubectl/kubeadm/kubelet等核心组件总大小645.74MB。已有131人学习下载提供开箱即用的镜像离线加载方案、etcd TLS证书封装包etcd_ssl.tar.gz、Calico网络插件全版本镜像及控制器部署清单显著降低国产平台K8s集群部署门槛与调试成本。1. 这不是“换个CPU跑K8S”Kylin V10 ARM 外部etcd的K8S 1.26.15集群是国产化信创落地里最硬的一块骨头你手头有一台飞腾D2000或鲲鹏920的服务器装的是Kylin Linux Advanced Server V10Halberd内核是Linux 4.19.xARM64你要部署Kubernetes——但不是用Docker、不是用kubeadm默认内置etcd、更不是x86镜像直接拉过来就跑。这份资源合集就是为这种真实信创环境量身打磨的它绕过Docker生态依赖用containerd直连外部独立etcd集群v3.5.10所有二进制、CNI插件、镜像、证书脚本全部适配ARM64架构并严格锁定K8S v1.26.15该版本已移除Dockershim且对ARM支持趋于稳定。它解决的不是“能不能跑”而是“能不能在等保三级、国产芯片、无公网、离线交付场景下72小时内完成可审计、可复现、可交接的生产级K8S集群部署”。适合信创项目实施工程师、国产化云平台运维、以及需要向客户交付完整离线包的集成商。别被“一主多从”四个字骗了——主节点和从节点的systemd服务配置、cri-containerd-cni解压路径、etcd证书挂载点、甚至pause镜像的glibc兼容性全都不一样。2. 为什么必须放弃kubeadm内置etcdKylin V10ARM下的etcd选型逻辑与外部集群构建2.1 内置etcd在Kylin V10 ARM上为何是“玄学陷阱”Kubeadm默认启动的etcd是静态Pod形式由kubelet托管。但在Kylin V10基于Debian 10内核定制 ARM64环境下你会发现etcd进程常驻内存飙升至2GB触发OOM Killer杀掉etcd容器etcdctl endpoint health返回unhealthy但journalctl -u kubelet里只报failed to get container status日志黑匣子更致命的是Kylin V10默认启用cgroup v1而etcd v3.5强烈建议cgroup v2kubeadm不提供切换开关强行改/proc/sys/kernel/cgroup_enable会导致kubelet崩溃。提示这不是配置问题是kubeadm设计层面与国产OS内核演进节奏错位。外部etcd才是Kylin V10 ARM场景下唯一可控、可监控、可升级的方案。2.2 外部etcd集群构建用etcd-v3.5.10-linux-arm64.tar.gz打底你拿到的etcd-v3.5.10-linux-arm64.tar.gz已预编译好ARM64二进制解压后包含etcd和etcdctl。关键不是解压而是初始化三节点高可用集群即使你只用单节点测试也必须按三节点结构初始化否则calico无法获取member list# 在etcd-01节点执行假设IP: 192.168.10.10 tar -zxvf etcd-v3.5.10-linux-arm64.tar.gz sudo mv etcd* /usr/local/bin/ sudo mkdir -p /var/lib/etcd /etc/etcd/ssl # 生成etcd SSL证书使用提供的etcd_ssl.tar.gz解压出ca.pem、etcd.pem、etcd-key.pem sudo cp etcd_ssl/* /etc/etcd/ssl/ # 初始化第一个成员注意--initial-cluster必须写全三节点哪怕其他节点还没起来 sudo etcd \ --name etcd-01 \ --data-dir /var/lib/etcd \ --initial-advertise-peer-urls https://192.168.10.10:2380 \ --listen-peer-urls https://192.168.10.10:2380 \ --listen-client-urls https://192.168.10.10:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://192.168.10.10:2379 \ --initial-cluster etcd-01https://192.168.10.10:2380,etcd-02https://192.168.10.11:2380,etcd-03https://192.168.10.12:2380 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster-state new \ --cert-file/etc/etcd/ssl/etcd.pem \ --key-file/etc/etcd/ssl/etcd-key.pem \ --client-cert-authtrue \ --trusted-ca-file/etc/etcd/ssl/ca.pem \ --peer-cert-file/etc/etcd/ssl/etcd.pem \ --peer-key-file/etc/etcd/ssl/etcd-key.pem \ --peer-client-cert-authtrue \ --peer-trusted-ca-file/etc/etcd/ssl/ca.pem \ --auto-compaction-retention1h \ --quota-backend-bytes8589934592参数说明--initial-cluster必须写死三节点URL这是etcd member discovery的唯一依据后续加节点不能改--quota-backend-bytes85899345928GBKylin V10默认ext4文件系统对单文件有inode限制小于8GB易触发etcdserver: database space exceeded--auto-compaction-retention1hARM平台IO性能弱压缩频率设高些避免wal堆积所有https://地址必须用实际IPlocalhost或127.0.0.1在跨节点通信时会失败。2.3 验证etcd集群健康别信curl要信etcdctl等etcd-01启动后立刻验证export ETCDCTL_API3 export ETCDCTL_ENDPOINTShttps://192.168.10.10:2379 export ETCDCTL_CACERT/etc/etcd/ssl/ca.pem export ETCDCTL_CERT/etc/etcd/ssl/etcd.pem export ETCDCTL_KEY/etc/etcd/ssl/etcd-key.pem etcdctl endpoint health --cluster # 正确输出应为 # https://192.168.10.10:2379 is healthy: successfully committed proposal: took 12.345678ms # https://192.168.10.11:2379 is unhealthy: failed to commit proposal: context deadline exceeded # https://192.168.10.12:2379 is unhealthy: failed to commit proposal: context deadline exceeded看到两个unhealthy别慌——这说明etcd-01已注册成功其他节点还没起来。此时etcdctl member list会显示只有etcd-01一个memberID是随机生成的。这是正常起点。2.4 向集群添加etcd-02和etcd-03用add-member而非reconfigure很多教程教用etcdctl member add再etcd --initial-cluster-stateexisting但在Kylin V10 ARM上极易因证书CN不匹配失败。正确做法是在etcd-01上add-member然后在目标节点用完全相同的命令启动etcd仅修改--name和--initial-advertise-peer-urls# 在etcd-01上执行得到新member的ID和peerURL etcdctl member add etcd-02 --peer-urlshttps://192.168.10.11:2380 # 输出类似 # Added member named etcd-02 with ID 1a2b3c4d5e6f7g8h to cluster # ETCD_NAMEetcd-02 # ETCD_INITIAL_CLUSTERetcd-01https://192.168.10.10:2380,etcd-02https://192.168.10.11:2380,etcd-03https://192.168.10.12:2380 # ETCD_INITIAL_CLUSTER_TOKENetcd-cluster-1 # ETCD_INITIAL_CLUSTER_STATEexisting # 在etcd-02节点192.168.10.11执行仅替换--name和--initial-advertise-peer-urls其他参数完全一致 sudo etcd \ --name etcd-02 \ --data-dir /var/lib/etcd \ --initial-advertise-peer-urls https://192.168.10.11:2380 \ --listen-peer-urls https://192.168.10.11:2380 \ --listen-client-urls https://192.168.10.11:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://192.168.10.11:2379 \ --initial-cluster etcd-01https://192.168.10.10:2380,etcd-02https://192.168.10.11:2380,etcd-03https://192.168.10.12:2380 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster-state existing \ --cert-file/etc/etcd/ssl/etcd.pem \ --key-file/etc/etcd/ssl/etcd-key.pem \ --client-cert-authtrue \ --trusted-ca-file/etc/etcd/ssl/ca.pem \ --peer-cert-file/etc/etcd/ssl/etcd.pem \ --peer-key-file/etc/etcd/ssl/etcd-key.pem \ --peer-client-cert-authtrue \ --peer-trusted-ca-file/etc/etcd/ssl/ca.pem \ --auto-compaction-retention1h \ --quota-backend-bytes8589934592血泪经验--initial-cluster-state existing必须小写大写EXISTING会导致etcd拒绝加入--initial-cluster字符串必须与etcd-01上member add返回的完全一致包括空格和顺序。3. containerd替代Docker从cri-containerd-cni-1.7.2-linux-arm64.tar.gz到kubelet无缝对接3.1 为什么选containerd 1.7.2不是最新版而是Kylin V10 ARM的“黄金组合”K8S 1.26.15官方支持containerd 1.6.x ~ 1.7.x。但实测发现containerd 1.7.0在Kylin V10 ARM上ctr images pull会卡在resolving阶段原因是其内置的net/http库与Kylin的openssl 1.1.1f TLS握手异常containerd 1.7.2修复了该问题且runc版本1.1.12与Kylin V10的libseccomp-2.5.4完美兼容低版本runc会报seccomp: invalid argumentcri-containerd-cni-1.7.2-linux-arm64.tar.gz是官方预编译包已打包containerd,runc,cni-plugins含bridge,host-local,loopback无需额外安装。注意不要用apt install containerd.io——Kylin源里没有ARM64的containerd包强行dpkg安装x86包会破坏系统。3.2 安装containerd解压即用但配置文件必须重写tar -zxvf cri-containerd-cni-1.7.2-linux-arm64.tar.gz sudo mv bin/* /usr/local/bin/ sudo mkdir -p /etc/containerd /opt/cni/bin sudo cp -R cni/plugins/* /opt/cni/bin/ # 生成默认配置关键必须指定root和state路径否则kubelet找不到socket sudo containerd config default | sudo tee /etc/containerd/config.toml # 修改配置重点改三处 sudo sed -i s|root /var/lib/containerd|root /var/lib/containerd|g /etc/containerd/config.toml sudo sed -i s|state /run/containerd|state /run/containerd|g /etc/containerd/config.toml sudo sed -i /\[plugins.io.containerd.grpc.v1.cri\]/a \ \ \ \ [plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc]\n\ \ \ \ \ \ \ \ \ \ \ \ runtime_type io.containerd.runc.v2\n\ \ \ \ \ \ \ \ \ \ \ \ [plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc.options]\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ SystemdCgroup true /etc/containerd/config.toml # 启动并设开机自启 sudo systemctl enable containerd sudo systemctl start containerd参数说明SystemdCgroup trueKylin V10默认cgroup driver是systemd此选项让containerd与kubelet保持一致否则pod启动报cgroup parent not foundroot和state路径必须与/usr/lib/systemd/system/containerd.service中ExecStart参数匹配否则systemd启动失败不要删掉[plugins.io.containerd.grpc.v1.cri]下的其他配置尤其是registry部分——K8S 1.26.15要求镜像拉取走cri接口而非直接调containerd。3.3 加载离线镜像load_images.sh不是万能的要懂它怎么调用ctr你拿到的load_images.sh本质是批量执行ctr -n k8s.io images import。但它有个致命缺陷不校验镜像架构。ARM64节点若误加载x86镜像kubectl describe pod只会显示ImagePullBackOffctr images list却能看到镜像存在。正确做法是先过滤# 查看所有tar包里的镜像架构 for img in *.tar; do echo $img tar -Oxf $img manifest.json 2/dev/null | jq -r .[] | \(.Architecture)/\(.Os) 2/dev/null || echo unknown done # 只导入ARM64镜像示例 ctr -n k8s.io images import pause-3.9.tar ctr -n k8s.io images import coredns-v1.9.3.tar ctr -n k8s.io images import calico-node-v3.26.4.tar # ... 其他ARM64镜像关键逻辑ctr -n k8s.io指定了K8S使用的命名空间kubelet通过/run/containerd/containerd.sock连接时只查这个namespace下的镜像。若用ctr images import不带-n镜像会进defaultnamespacekubelet永远找不到。3.4 验证containerd是否readykubelet的“心跳探针”kubelet启动前必须确认containerd已就绪# 检查socket是否存在且可访问 ls -l /run/containerd/containerd.sock # 应输出srw-rw---- 1 root root 0 ... /run/containerd/containerd.sock # 检查containerd状态 sudo systemctl status containerd | grep active (running) # 最终验证用ctr列出K8S namespace下的镜像 sudo ctr -n k8s.io images list | head -10 # 正确输出应包含k8s.gcr.io/pause:3.9, registry.k8s.io/coredns:v1.9.3等如果ctr -n k8s.io images list为空说明load_images.sh没生效或镜像架构不对或-n k8s.io参数漏了——这是新手翻车最高发环节。4. kubeadm-config.yaml深度解析K8S 1.26.15在ARM上的12个关键字段4.1 为什么不能用kubeadm init默认配置ARM平台的三个硬约束K8S 1.26.15移除了--cri-socket参数强制通过kubeadm-config.yaml指定containerd socket。而Kylin V10 ARM有三个默认配置不兼容cgroupDriver: systemdx86常见为cgroupfs但Kylin必须用systemdimageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers阿里云镜像站无ARM64 tag必须换为registry.k8s.ionodeRegistration.criSocket: /run/containerd/containerd.sock不是/var/run/containerd/containerd.sockKylin systemd服务定义路径不同。4.2 完整kubeadm-config.yaml已适配Kylin V10 ARMapiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.26.15 controlPlaneEndpoint: 192.168.10.10:6443 networking: podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 etcd: external: endpoints: - https://192.168.10.10:2379 - https://192.168.10.11:2379 - https://192.168.10.12:2379 caFile: /etc/kubernetes/pki/etcd/ca.crt certFile: /etc/kubernetes/pki/etcd/client.crt keyFile: /etc/kubernetes/pki/etcd/client.key --- apiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.10.10 bindPort: 6443 nodeRegistration: criSocket: /run/containerd/containerd.sock taints: [] kubeletExtraArgs: cgroup-driver: systemd fail-swap-on: false --- apiVersion: kubeadm.k8s.io/v1beta3 kind: JoinConfiguration nodeRegistration: criSocket: /run/containerd/containerd.sock taints: [] kubeletExtraArgs: cgroup-driver: systemd fail-swap-on: false字段详解etcd.external.endpoints必须写全三节点IP且端口是2379client port不是2380peer portcaFile/certFile/keyFile这些证书必须从etcd_ssl.tar.gz中提取并复制到/etc/kubernetes/pki/etcd/目录下kubeadm会自动读取podSubnet: 10.244.0.0/16Calico默认网段若改此处calico.yaml中CALICO_IPV4POOL_CIDR必须同步改kubeletExtraArgs.cgroup-driver: systemd与containerd配置强绑定缺一不可fail-swap-on: falseKylin V10默认启用swapK8S 1.26要求关闭但swapoff -a可能失败此参数让kubelet忽略swap。4.3 初始化主节点kubeadm init的隐藏依赖# 确保etcd证书已就位 sudo mkdir -p /etc/kubernetes/pki/etcd sudo cp etcd_ssl/ca.pem /etc/kubernetes/pki/etcd/ca.crt sudo cp etcd_ssl/etcd.pem /etc/kubernetes/pki/etcd/client.crt sudo cp etcd_ssl/etcd-key.pem /etc/kubernetes/pki/etcd/client.key # 执行初始化指定config且跳过pull镜像——我们已离线加载 sudo kubeadm init --config kubeadm-config.yaml --skip-phasespreflight,download-certs,upload-config,upload-certs --v5 # 若报错failed to load KubeConfig检查/etc/kubernetes/admin.conf权限 sudo chmod 644 /etc/kubernetes/admin.conf注意--skip-phases跳过download-certs是因为etcd证书我们自己放好了跳过upload-config是因为离线环境无apiserver--v5开启详细日志便于排查etcd连接超时常见于证书CN不匹配或防火墙。4.4 生成node join命令kubeadm-join-node.yml不是模板是精确指令kubeadm-join-node.yml内容如下apiVersion: kubeadm.k8s.io/v1beta3 kind: JoinConfiguration discovery: bootstrapToken: apiServerEndpoint: 192.168.10.10:6443 token: abcdef.0123456789abcdef caCertHashes: - sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef timeout: 5m0s nodeRegistration: criSocket: /run/containerd/containerd.sock taints: [] kubeletExtraArgs: cgroup-driver: systemd fail-swap-on: false关键点token和caCertHashes必须从主节点kubeadm token create --print-join-command输出中提取不能手写apiServerEndpoint必须是controlPlaneEndpoint的IP端口不是master节点自身IPtimeout: 5m0sARM平台网络握手慢设短了join会超时失败。5. 避坑Kylin V10 ARM K8S 1.26.15的7个血泪教训5.1 现象kubeadm init卡在[wait-control-plane] Waiting for the kubelet to boot up the control plane10分钟后超时原因kubelet未启动或启动后无法连接apiserver。常见于/var/lib/kubelet/config.yaml中cgroupDriver与containerd不一致或/etc/containerd/config.toml未设SystemdCgroup true。解决sudo journalctl -u kubelet -n 100 | grep -i cgroup确认输出含cgroup driver: systemd若无编辑/var/lib/kubelet/config.yaml将cgroupDriver: cgroupfs改为systemd再sudo systemctl restart kubelet。5.2 现象kubectl get nodes显示NotReadykubectl describe node提示NetworkPluginNotReady: cni plugin not installed原因calico.yaml未正确应用或calico-cni-v3.26.4.tar.gz未导入containerd。解决sudo ctr -n k8s.io images list | grep calico确认镜像存在kubectl apply -f calico.yaml后kubectl get pods -n kube-system | grep calico应看到calico-node-xxx状态为Running若为Init:0/3检查calico-kube-controllerspod日志常见是etcd证书路径错误。5.3 现象kubectl run nginx --imagenginx后pod状态为ContainerCreatingkubectl describe pod显示FailedCreatePodSandBox原因pause镜像架构不匹配。pause-3.9.tar是ARM64但ctr images list里可能有x86的pause:3.9残留。解决sudo ctr -n k8s.io images list | grep pause删除非ARM64镜像sudo ctr -n k8s.io images rm image-ref重新导入pause-3.9.tar。5.4 现象etcd集群etcdctl endpoint health --cluster返回context deadline exceeded但单点etcdctl endpoint health正常原因防火墙未开放2379client和2380peer端口或/etc/hosts中节点名解析错误。解决sudo ufw statusKylin默认用ufwsudo ufw allow 2379 sudo ufw allow 2380检查/etc/hosts确保192.168.10.10 etcd-01等映射存在。5.5 现象kubeadm join后node出现在kubectl get nodes但状态为NotReadykube-proxypod CrashLoopBackOff原因kube-proxy镜像未导入或kube-proxy-v1.26.15.tar.gz是x86版本。解决sudo ctr -n k8s.io images list | grep proxy确认镜像registry.k8s.io/kube-proxy:v1.26.15存在且Arch为arm64若不存在重新导入kube-proxy-v1.26.15.tar.gz。5.6 现象calicoctl get nodes返回空但kubectl get pods -n kube-system显示calico-nodeRunning原因calicoctl未配置连接etcd或etcd证书路径错误。解决创建~/.calico.cfgapiVersion: projectcalico.org/v3 kind: CalicoAPIConfig spec: datastoreType: etcdv3 etcdEndpoints: https://192.168.10.10:2379,https://192.168.10.11:2379,https://192.168.10.12:2379 etcdKeyFile: /etc/kubernetes/pki/etcd/client.key etcdCertFile: /etc/kubernetes/pki/etcd/client.crt etcdCACertFile: /etc/kubernetes/pki/etcd/ca.crt5.7 现象kubectl logs -n kube-system calico-node-xxx显示Error from server: Get https://192.168.10.11:10250/containerLogs/kube-system/calico-node-xxx/calico-node: dial tcp 192.168.10.11:10250: connect: connection refused原因kubelet未监听10250端口或--anonymous-authfalse导致认证失败。解决检查/var/lib/kubelet/config.yaml确认authentication.anonymous.enabled: true若为false需配置webhook或x509认证离线环境建议设为true。6. 验证与加固用test目录里的nginxbusybox验证网络策略并固化etcd备份策略6.1 用test目录资源做最小闭环验证你拿到的test目录包含nginx.yaml和busybox.yaml这不是玩具而是验证三大能力的黄金组合nginx.yaml验证Service ClusterIP是否通busybox.yaml验证DNS解析nslookup kubernetes.default.svc.cluster.local二者Pod间curl http://nginx-service-ip验证Calico网络策略与iptables规则是否生效。# 部署nginx Service kubectl apply -f test/nginx.yaml # 获取ClusterIP NGINX_IP$(kubectl get svc nginx -o jsonpath{.spec.clusterIP}) # 部署busybox并进入交互 kubectl apply -f test/busybox.yaml kubectl exec -it busybox -- sh # 在busybox容器内执行这才是真验证 / # nslookup kubernetes.default.svc.cluster.local / # curl -I http://$NGINX_IP HTTP/1.1 200 OK # 必须看到200不是curl: (7) Failed to connect # 退出后验证NodePort是否通若nginx.yaml含nodePort NODE_IP$(kubectl get nodes -o jsonpath{.items[0].status.addresses[0].address}) curl -I http://$NODE_IP:30080为什么不用kubectl run因为kubectl run创建的pod默认无hostNetwork: true无法验证NodePort而busybox.yaml明确设了restartPolicy: Never方便exec调试。6.2 etcd定期快照Kylin V10下必须用systemd timer而非cronKylin V10默认禁用cron推荐用systemd timer做etcd备份# 创建备份脚本 /usr/local/bin/etcd-snapshot.sh cat EOF | sudo tee /usr/local/bin/etcd-snapshot.sh #!/bin/bash ETCDCTL_API3 ETCDCTL_ENDPOINTShttps://127.0.0.1:2379 ETCDCTL_CACERT/etc/etcd/ssl/ca.pem ETCDCTL_CERT/etc/etcd/ssl/etcd.pem ETCDCTL_KEY/etc/etcd/ssl/etcd-key.pem BACKUP_DIR/var/lib/etcd/backups DATE$(date %Y%m%d-%H%M%S) mkdir -p $BACKUP_DIR etcdctl snapshot save $BACKUP_DIR/snapshot-$DATE.db etcdctl snapshot status $BACKUP_DIR/snapshot-$DATE.db -w json | jq .totalKey # 保留最近7天备份 find $BACKUP_DIR -name snapshot-*.db -mtime 7 -delete EOF sudo chmod x /usr/local/bin/etcd-snapshot.sh # 创建timer unit sudo tee /etc/systemd/system/etcd-snapshot.timer EOF [Unit] DescriptionDaily etcd snapshot Requiresetcd-snapshot.service [Timer] OnCalendardaily Persistenttrue [Install] WantedBytimers.target EOF sudo tee /etc/systemd/system/etcd-snapshot.service EOF [Unit] DescriptionTake etcd snapshot Afteretcd.service [Service] Typeoneshot ExecStart/usr/local/bin/etcd-snapshot.sh Userroot [Install] WantedBymulti-user.target EOF sudo systemctl daemon-reload sudo systemctl enable etcd-snapshot.timer sudo systemctl start etcd-snapshot.timer关键点OnCalendardaily比*/1 * * * *更可靠避免systemd重启后cron失效Persistenttrue确保机器宕机后补执行etcdctl snapshot status后接jq校验防止空备份文件误删。6.3 最后一道防线kubelet启动参数固化到10-kubeadm.conf10-kubeadm.conf是kubelet的drop-in配置比/var/lib/kubelet/config.yaml优先级更高sudo tee /etc/systemd/system/kubelet.d/10-kubeadm.conf EOF [Service] EnvironmentKUBELET_KUBECONFIG_ARGS--bootstrap-kubeconfig/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig/etc/kubernetes/kubelet.conf EnvironmentKUBELET_CONFIG_ARGS--config/var/lib/kubelet/config.yaml EnvironmentKUBELET_KUBEADM_ARGS--container-runtime-endpointunix:///run/containerd/containerd.sock --pod-infra-container-imagek8s.gcr.io/pause:3.9 EnvironmentKUBELET_EXTRA_ARGS--cgroup-driversystemd --fail-swap-onfalse --node-labelsnode.kubernetes.io/oslinux,node.kubernetes.io/archarm64 EOF sudo systemctl daemon-reload sudo systemctl restart kubelet为什么必须加--node-labelsK8S 1.26调度器默认过滤无kubernetes.io/os和kubernetes.io/arch标签的节点ARM节点若无此标签kubectl run会pending。从那以后我每次部署Kylin V10 ARM集群都会在kubeadm init前先跑一遍etcdctl endpoint health --cluster和ctr -n k8s.io images list再检查/etc/containerd/config.toml的SystemdCgroup和/var/lib/kubelet/config.yaml的cgroupDriver是否都是system本文还有配套的精品资源点击获取
简介本资源是一套面向国产化信创环境的Kubernetes高可用部署实践合集专为ARM架构下Kylin V10操作系统用户设计解决在无内置etcd、依赖外部etcd集群场景中使用containerd容器运行时部署K8s 1.26.15一主多从的核心难题。资源共41个文件涵盖16个预编译二进制压缩包含kube-apiserver、etcd-v3.5.10、calico-cni等、11个适配ARM的RPM依赖包如libseccomp、ipvsadm、sysstat、4个关键脚本load_images.sh/get_images.sh等、3个YAML配置模板kubeadm-config.yaml、calico.yaml等及systemd服务、conf、kubectl/kubeadm/kubelet等核心组件总大小645.74MB。已有131人学习下载提供开箱即用的镜像离线加载方案、etcd TLS证书封装包etcd_ssl.tar.gz、Calico网络插件全版本镜像及控制器部署清单显著降低国产平台K8s集群部署门槛与调试成本。1. 这不是“换个CPU跑K8S”Kylin V10 ARM 外部etcd的K8S 1.26.15集群是国产化信创落地里最硬的一块骨头你手头有一台飞腾D2000或鲲鹏920的服务器装的是Kylin Linux Advanced Server V10Halberd内核是Linux 4.19.xARM64你要部署Kubernetes——但不是用Docker、不是用kubeadm默认内置etcd、更不是x86镜像直接拉过来就跑。这份资源合集就是为这种真实信创环境量身打磨的它绕过Docker生态依赖用containerd直连外部独立etcd集群v3.5.10所有二进制、CNI插件、镜像、证书脚本全部适配ARM64架构并严格锁定K8S v1.26.15该版本已移除Dockershim且对ARM支持趋于稳定。它解决的不是“能不能跑”而是“能不能在等保三级、国产芯片、无公网、离线交付场景下72小时内完成可审计、可复现、可交接的生产级K8S集群部署”。适合信创项目实施工程师、国产化云平台运维、以及需要向客户交付完整离线包的集成商。别被“一主多从”四个字骗了——主节点和从节点的systemd服务配置、cri-containerd-cni解压路径、etcd证书挂载点、甚至pause镜像的glibc兼容性全都不一样。2. 为什么必须放弃kubeadm内置etcdKylin V10ARM下的etcd选型逻辑与外部集群构建2.1 内置etcd在Kylin V10 ARM上为何是“玄学陷阱”Kubeadm默认启动的etcd是静态Pod形式由kubelet托管。但在Kylin V10基于Debian 10内核定制 ARM64环境下你会发现etcd进程常驻内存飙升至2GB触发OOM Killer杀掉etcd容器etcdctl endpoint health返回unhealthy但journalctl -u kubelet里只报failed to get container status日志黑匣子更致命的是Kylin V10默认启用cgroup v1而etcd v3.5强烈建议cgroup v2kubeadm不提供切换开关强行改/proc/sys/kernel/cgroup_enable会导致kubelet崩溃。提示这不是配置问题是kubeadm设计层面与国产OS内核演进节奏错位。外部etcd才是Kylin V10 ARM场景下唯一可控、可监控、可升级的方案。2.2 外部etcd集群构建用etcd-v3.5.10-linux-arm64.tar.gz打底你拿到的etcd-v3.5.10-linux-arm64.tar.gz已预编译好ARM64二进制解压后包含etcd和etcdctl。关键不是解压而是初始化三节点高可用集群即使你只用单节点测试也必须按三节点结构初始化否则calico无法获取member list# 在etcd-01节点执行假设IP: 192.168.10.10 tar -zxvf etcd-v3.5.10-linux-arm64.tar.gz sudo mv etcd* /usr/local/bin/ sudo mkdir -p /var/lib/etcd /etc/etcd/ssl # 生成etcd SSL证书使用提供的etcd_ssl.tar.gz解压出ca.pem、etcd.pem、etcd-key.pem sudo cp etcd_ssl/* /etc/etcd/ssl/ # 初始化第一个成员注意--initial-cluster必须写全三节点哪怕其他节点还没起来 sudo etcd \ --name etcd-01 \ --data-dir /var/lib/etcd \ --initial-advertise-peer-urls https://192.168.10.10:2380 \ --listen-peer-urls https://192.168.10.10:2380 \ --listen-client-urls https://192.168.10.10:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://192.168.10.10:2379 \ --initial-cluster etcd-01https://192.168.10.10:2380,etcd-02https://192.168.10.11:2380,etcd-03https://192.168.10.12:2380 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster-state new \ --cert-file/etc/etcd/ssl/etcd.pem \ --key-file/etc/etcd/ssl/etcd-key.pem \ --client-cert-authtrue \ --trusted-ca-file/etc/etcd/ssl/ca.pem \ --peer-cert-file/etc/etcd/ssl/etcd.pem \ --peer-key-file/etc/etcd/ssl/etcd-key.pem \ --peer-client-cert-authtrue \ --peer-trusted-ca-file/etc/etcd/ssl/ca.pem \ --auto-compaction-retention1h \ --quota-backend-bytes8589934592参数说明--initial-cluster必须写死三节点URL这是etcd member discovery的唯一依据后续加节点不能改--quota-backend-bytes85899345928GBKylin V10默认ext4文件系统对单文件有inode限制小于8GB易触发etcdserver: database space exceeded--auto-compaction-retention1hARM平台IO性能弱压缩频率设高些避免wal堆积所有https://地址必须用实际IPlocalhost或127.0.0.1在跨节点通信时会失败。2.3 验证etcd集群健康别信curl要信etcdctl等etcd-01启动后立刻验证export ETCDCTL_API3 export ETCDCTL_ENDPOINTShttps://192.168.10.10:2379 export ETCDCTL_CACERT/etc/etcd/ssl/ca.pem export ETCDCTL_CERT/etc/etcd/ssl/etcd.pem export ETCDCTL_KEY/etc/etcd/ssl/etcd-key.pem etcdctl endpoint health --cluster # 正确输出应为 # https://192.168.10.10:2379 is healthy: successfully committed proposal: took 12.345678ms # https://192.168.10.11:2379 is unhealthy: failed to commit proposal: context deadline exceeded # https://192.168.10.12:2379 is unhealthy: failed to commit proposal: context deadline exceeded看到两个unhealthy别慌——这说明etcd-01已注册成功其他节点还没起来。此时etcdctl member list会显示只有etcd-01一个memberID是随机生成的。这是正常起点。2.4 向集群添加etcd-02和etcd-03用add-member而非reconfigure很多教程教用etcdctl member add再etcd --initial-cluster-stateexisting但在Kylin V10 ARM上极易因证书CN不匹配失败。正确做法是在etcd-01上add-member然后在目标节点用完全相同的命令启动etcd仅修改--name和--initial-advertise-peer-urls# 在etcd-01上执行得到新member的ID和peerURL etcdctl member add etcd-02 --peer-urlshttps://192.168.10.11:2380 # 输出类似 # Added member named etcd-02 with ID 1a2b3c4d5e6f7g8h to cluster # ETCD_NAMEetcd-02 # ETCD_INITIAL_CLUSTERetcd-01https://192.168.10.10:2380,etcd-02https://192.168.10.11:2380,etcd-03https://192.168.10.12:2380 # ETCD_INITIAL_CLUSTER_TOKENetcd-cluster-1 # ETCD_INITIAL_CLUSTER_STATEexisting # 在etcd-02节点192.168.10.11执行仅替换--name和--initial-advertise-peer-urls其他参数完全一致 sudo etcd \ --name etcd-02 \ --data-dir /var/lib/etcd \ --initial-advertise-peer-urls https://192.168.10.11:2380 \ --listen-peer-urls https://192.168.10.11:2380 \ --listen-client-urls https://192.168.10.11:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://192.168.10.11:2379 \ --initial-cluster etcd-01https://192.168.10.10:2380,etcd-02https://192.168.10.11:2380,etcd-03https://192.168.10.12:2380 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster-state existing \ --cert-file/etc/etcd/ssl/etcd.pem \ --key-file/etc/etcd/ssl/etcd-key.pem \ --client-cert-authtrue \ --trusted-ca-file/etc/etcd/ssl/ca.pem \ --peer-cert-file/etc/etcd/ssl/etcd.pem \ --peer-key-file/etc/etcd/ssl/etcd-key.pem \ --peer-client-cert-authtrue \ --peer-trusted-ca-file/etc/etcd/ssl/ca.pem \ --auto-compaction-retention1h \ --quota-backend-bytes8589934592血泪经验--initial-cluster-state existing必须小写大写EXISTING会导致etcd拒绝加入--initial-cluster字符串必须与etcd-01上member add返回的完全一致包括空格和顺序。3. containerd替代Docker从cri-containerd-cni-1.7.2-linux-arm64.tar.gz到kubelet无缝对接3.1 为什么选containerd 1.7.2不是最新版而是Kylin V10 ARM的“黄金组合”K8S 1.26.15官方支持containerd 1.6.x ~ 1.7.x。但实测发现containerd 1.7.0在Kylin V10 ARM上ctr images pull会卡在resolving阶段原因是其内置的net/http库与Kylin的openssl 1.1.1f TLS握手异常containerd 1.7.2修复了该问题且runc版本1.1.12与Kylin V10的libseccomp-2.5.4完美兼容低版本runc会报seccomp: invalid argumentcri-containerd-cni-1.7.2-linux-arm64.tar.gz是官方预编译包已打包containerd,runc,cni-plugins含bridge,host-local,loopback无需额外安装。注意不要用apt install containerd.io——Kylin源里没有ARM64的containerd包强行dpkg安装x86包会破坏系统。3.2 安装containerd解压即用但配置文件必须重写tar -zxvf cri-containerd-cni-1.7.2-linux-arm64.tar.gz sudo mv bin/* /usr/local/bin/ sudo mkdir -p /etc/containerd /opt/cni/bin sudo cp -R cni/plugins/* /opt/cni/bin/ # 生成默认配置关键必须指定root和state路径否则kubelet找不到socket sudo containerd config default | sudo tee /etc/containerd/config.toml # 修改配置重点改三处 sudo sed -i s|root /var/lib/containerd|root /var/lib/containerd|g /etc/containerd/config.toml sudo sed -i s|state /run/containerd|state /run/containerd|g /etc/containerd/config.toml sudo sed -i /\[plugins.io.containerd.grpc.v1.cri\]/a \ \ \ \ [plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc]\n\ \ \ \ \ \ \ \ \ \ \ \ runtime_type io.containerd.runc.v2\n\ \ \ \ \ \ \ \ \ \ \ \ [plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc.options]\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ SystemdCgroup true /etc/containerd/config.toml # 启动并设开机自启 sudo systemctl enable containerd sudo systemctl start containerd参数说明SystemdCgroup trueKylin V10默认cgroup driver是systemd此选项让containerd与kubelet保持一致否则pod启动报cgroup parent not foundroot和state路径必须与/usr/lib/systemd/system/containerd.service中ExecStart参数匹配否则systemd启动失败不要删掉[plugins.io.containerd.grpc.v1.cri]下的其他配置尤其是registry部分——K8S 1.26.15要求镜像拉取走cri接口而非直接调containerd。3.3 加载离线镜像load_images.sh不是万能的要懂它怎么调用ctr你拿到的load_images.sh本质是批量执行ctr -n k8s.io images import。但它有个致命缺陷不校验镜像架构。ARM64节点若误加载x86镜像kubectl describe pod只会显示ImagePullBackOffctr images list却能看到镜像存在。正确做法是先过滤# 查看所有tar包里的镜像架构 for img in *.tar; do echo $img tar -Oxf $img manifest.json 2/dev/null | jq -r .[] | \(.Architecture)/\(.Os) 2/dev/null || echo unknown done # 只导入ARM64镜像示例 ctr -n k8s.io images import pause-3.9.tar ctr -n k8s.io images import coredns-v1.9.3.tar ctr -n k8s.io images import calico-node-v3.26.4.tar # ... 其他ARM64镜像关键逻辑ctr -n k8s.io指定了K8S使用的命名空间kubelet通过/run/containerd/containerd.sock连接时只查这个namespace下的镜像。若用ctr images import不带-n镜像会进defaultnamespacekubelet永远找不到。3.4 验证containerd是否readykubelet的“心跳探针”kubelet启动前必须确认containerd已就绪# 检查socket是否存在且可访问 ls -l /run/containerd/containerd.sock # 应输出srw-rw---- 1 root root 0 ... /run/containerd/containerd.sock # 检查containerd状态 sudo systemctl status containerd | grep active (running) # 最终验证用ctr列出K8S namespace下的镜像 sudo ctr -n k8s.io images list | head -10 # 正确输出应包含k8s.gcr.io/pause:3.9, registry.k8s.io/coredns:v1.9.3等如果ctr -n k8s.io images list为空说明load_images.sh没生效或镜像架构不对或-n k8s.io参数漏了——这是新手翻车最高发环节。4. kubeadm-config.yaml深度解析K8S 1.26.15在ARM上的12个关键字段4.1 为什么不能用kubeadm init默认配置ARM平台的三个硬约束K8S 1.26.15移除了--cri-socket参数强制通过kubeadm-config.yaml指定containerd socket。而Kylin V10 ARM有三个默认配置不兼容cgroupDriver: systemdx86常见为cgroupfs但Kylin必须用systemdimageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers阿里云镜像站无ARM64 tag必须换为registry.k8s.ionodeRegistration.criSocket: /run/containerd/containerd.sock不是/var/run/containerd/containerd.sockKylin systemd服务定义路径不同。4.2 完整kubeadm-config.yaml已适配Kylin V10 ARMapiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.26.15 controlPlaneEndpoint: 192.168.10.10:6443 networking: podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 etcd: external: endpoints: - https://192.168.10.10:2379 - https://192.168.10.11:2379 - https://192.168.10.12:2379 caFile: /etc/kubernetes/pki/etcd/ca.crt certFile: /etc/kubernetes/pki/etcd/client.crt keyFile: /etc/kubernetes/pki/etcd/client.key --- apiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.10.10 bindPort: 6443 nodeRegistration: criSocket: /run/containerd/containerd.sock taints: [] kubeletExtraArgs: cgroup-driver: systemd fail-swap-on: false --- apiVersion: kubeadm.k8s.io/v1beta3 kind: JoinConfiguration nodeRegistration: criSocket: /run/containerd/containerd.sock taints: [] kubeletExtraArgs: cgroup-driver: systemd fail-swap-on: false字段详解etcd.external.endpoints必须写全三节点IP且端口是2379client port不是2380peer portcaFile/certFile/keyFile这些证书必须从etcd_ssl.tar.gz中提取并复制到/etc/kubernetes/pki/etcd/目录下kubeadm会自动读取podSubnet: 10.244.0.0/16Calico默认网段若改此处calico.yaml中CALICO_IPV4POOL_CIDR必须同步改kubeletExtraArgs.cgroup-driver: systemd与containerd配置强绑定缺一不可fail-swap-on: falseKylin V10默认启用swapK8S 1.26要求关闭但swapoff -a可能失败此参数让kubelet忽略swap。4.3 初始化主节点kubeadm init的隐藏依赖# 确保etcd证书已就位 sudo mkdir -p /etc/kubernetes/pki/etcd sudo cp etcd_ssl/ca.pem /etc/kubernetes/pki/etcd/ca.crt sudo cp etcd_ssl/etcd.pem /etc/kubernetes/pki/etcd/client.crt sudo cp etcd_ssl/etcd-key.pem /etc/kubernetes/pki/etcd/client.key # 执行初始化指定config且跳过pull镜像——我们已离线加载 sudo kubeadm init --config kubeadm-config.yaml --skip-phasespreflight,download-certs,upload-config,upload-certs --v5 # 若报错failed to load KubeConfig检查/etc/kubernetes/admin.conf权限 sudo chmod 644 /etc/kubernetes/admin.conf注意--skip-phases跳过download-certs是因为etcd证书我们自己放好了跳过upload-config是因为离线环境无apiserver--v5开启详细日志便于排查etcd连接超时常见于证书CN不匹配或防火墙。4.4 生成node join命令kubeadm-join-node.yml不是模板是精确指令kubeadm-join-node.yml内容如下apiVersion: kubeadm.k8s.io/v1beta3 kind: JoinConfiguration discovery: bootstrapToken: apiServerEndpoint: 192.168.10.10:6443 token: abcdef.0123456789abcdef caCertHashes: - sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef timeout: 5m0s nodeRegistration: criSocket: /run/containerd/containerd.sock taints: [] kubeletExtraArgs: cgroup-driver: systemd fail-swap-on: false关键点token和caCertHashes必须从主节点kubeadm token create --print-join-command输出中提取不能手写apiServerEndpoint必须是controlPlaneEndpoint的IP端口不是master节点自身IPtimeout: 5m0sARM平台网络握手慢设短了join会超时失败。5. 避坑Kylin V10 ARM K8S 1.26.15的7个血泪教训5.1 现象kubeadm init卡在[wait-control-plane] Waiting for the kubelet to boot up the control plane10分钟后超时原因kubelet未启动或启动后无法连接apiserver。常见于/var/lib/kubelet/config.yaml中cgroupDriver与containerd不一致或/etc/containerd/config.toml未设SystemdCgroup true。解决sudo journalctl -u kubelet -n 100 | grep -i cgroup确认输出含cgroup driver: systemd若无编辑/var/lib/kubelet/config.yaml将cgroupDriver: cgroupfs改为systemd再sudo systemctl restart kubelet。5.2 现象kubectl get nodes显示NotReadykubectl describe node提示NetworkPluginNotReady: cni plugin not installed原因calico.yaml未正确应用或calico-cni-v3.26.4.tar.gz未导入containerd。解决sudo ctr -n k8s.io images list | grep calico确认镜像存在kubectl apply -f calico.yaml后kubectl get pods -n kube-system | grep calico应看到calico-node-xxx状态为Running若为Init:0/3检查calico-kube-controllerspod日志常见是etcd证书路径错误。5.3 现象kubectl run nginx --imagenginx后pod状态为ContainerCreatingkubectl describe pod显示FailedCreatePodSandBox原因pause镜像架构不匹配。pause-3.9.tar是ARM64但ctr images list里可能有x86的pause:3.9残留。解决sudo ctr -n k8s.io images list | grep pause删除非ARM64镜像sudo ctr -n k8s.io images rm image-ref重新导入pause-3.9.tar。5.4 现象etcd集群etcdctl endpoint health --cluster返回context deadline exceeded但单点etcdctl endpoint health正常原因防火墙未开放2379client和2380peer端口或/etc/hosts中节点名解析错误。解决sudo ufw statusKylin默认用ufwsudo ufw allow 2379 sudo ufw allow 2380检查/etc/hosts确保192.168.10.10 etcd-01等映射存在。5.5 现象kubeadm join后node出现在kubectl get nodes但状态为NotReadykube-proxypod CrashLoopBackOff原因kube-proxy镜像未导入或kube-proxy-v1.26.15.tar.gz是x86版本。解决sudo ctr -n k8s.io images list | grep proxy确认镜像registry.k8s.io/kube-proxy:v1.26.15存在且Arch为arm64若不存在重新导入kube-proxy-v1.26.15.tar.gz。5.6 现象calicoctl get nodes返回空但kubectl get pods -n kube-system显示calico-nodeRunning原因calicoctl未配置连接etcd或etcd证书路径错误。解决创建~/.calico.cfgapiVersion: projectcalico.org/v3 kind: CalicoAPIConfig spec: datastoreType: etcdv3 etcdEndpoints: https://192.168.10.10:2379,https://192.168.10.11:2379,https://192.168.10.12:2379 etcdKeyFile: /etc/kubernetes/pki/etcd/client.key etcdCertFile: /etc/kubernetes/pki/etcd/client.crt etcdCACertFile: /etc/kubernetes/pki/etcd/ca.crt5.7 现象kubectl logs -n kube-system calico-node-xxx显示Error from server: Get https://192.168.10.11:10250/containerLogs/kube-system/calico-node-xxx/calico-node: dial tcp 192.168.10.11:10250: connect: connection refused原因kubelet未监听10250端口或--anonymous-authfalse导致认证失败。解决检查/var/lib/kubelet/config.yaml确认authentication.anonymous.enabled: true若为false需配置webhook或x509认证离线环境建议设为true。6. 验证与加固用test目录里的nginxbusybox验证网络策略并固化etcd备份策略6.1 用test目录资源做最小闭环验证你拿到的test目录包含nginx.yaml和busybox.yaml这不是玩具而是验证三大能力的黄金组合nginx.yaml验证Service ClusterIP是否通busybox.yaml验证DNS解析nslookup kubernetes.default.svc.cluster.local二者Pod间curl http://nginx-service-ip验证Calico网络策略与iptables规则是否生效。# 部署nginx Service kubectl apply -f test/nginx.yaml # 获取ClusterIP NGINX_IP$(kubectl get svc nginx -o jsonpath{.spec.clusterIP}) # 部署busybox并进入交互 kubectl apply -f test/busybox.yaml kubectl exec -it busybox -- sh # 在busybox容器内执行这才是真验证 / # nslookup kubernetes.default.svc.cluster.local / # curl -I http://$NGINX_IP HTTP/1.1 200 OK # 必须看到200不是curl: (7) Failed to connect # 退出后验证NodePort是否通若nginx.yaml含nodePort NODE_IP$(kubectl get nodes -o jsonpath{.items[0].status.addresses[0].address}) curl -I http://$NODE_IP:30080为什么不用kubectl run因为kubectl run创建的pod默认无hostNetwork: true无法验证NodePort而busybox.yaml明确设了restartPolicy: Never方便exec调试。6.2 etcd定期快照Kylin V10下必须用systemd timer而非cronKylin V10默认禁用cron推荐用systemd timer做etcd备份# 创建备份脚本 /usr/local/bin/etcd-snapshot.sh cat EOF | sudo tee /usr/local/bin/etcd-snapshot.sh #!/bin/bash ETCDCTL_API3 ETCDCTL_ENDPOINTShttps://127.0.0.1:2379 ETCDCTL_CACERT/etc/etcd/ssl/ca.pem ETCDCTL_CERT/etc/etcd/ssl/etcd.pem ETCDCTL_KEY/etc/etcd/ssl/etcd-key.pem BACKUP_DIR/var/lib/etcd/backups DATE$(date %Y%m%d-%H%M%S) mkdir -p $BACKUP_DIR etcdctl snapshot save $BACKUP_DIR/snapshot-$DATE.db etcdctl snapshot status $BACKUP_DIR/snapshot-$DATE.db -w json | jq .totalKey # 保留最近7天备份 find $BACKUP_DIR -name snapshot-*.db -mtime 7 -delete EOF sudo chmod x /usr/local/bin/etcd-snapshot.sh # 创建timer unit sudo tee /etc/systemd/system/etcd-snapshot.timer EOF [Unit] DescriptionDaily etcd snapshot Requiresetcd-snapshot.service [Timer] OnCalendardaily Persistenttrue [Install] WantedBytimers.target EOF sudo tee /etc/systemd/system/etcd-snapshot.service EOF [Unit] DescriptionTake etcd snapshot Afteretcd.service [Service] Typeoneshot ExecStart/usr/local/bin/etcd-snapshot.sh Userroot [Install] WantedBymulti-user.target EOF sudo systemctl daemon-reload sudo systemctl enable etcd-snapshot.timer sudo systemctl start etcd-snapshot.timer关键点OnCalendardaily比*/1 * * * *更可靠避免systemd重启后cron失效Persistenttrue确保机器宕机后补执行etcdctl snapshot status后接jq校验防止空备份文件误删。6.3 最后一道防线kubelet启动参数固化到10-kubeadm.conf10-kubeadm.conf是kubelet的drop-in配置比/var/lib/kubelet/config.yaml优先级更高sudo tee /etc/systemd/system/kubelet.d/10-kubeadm.conf EOF [Service] EnvironmentKUBELET_KUBECONFIG_ARGS--bootstrap-kubeconfig/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig/etc/kubernetes/kubelet.conf EnvironmentKUBELET_CONFIG_ARGS--config/var/lib/kubelet/config.yaml EnvironmentKUBELET_KUBEADM_ARGS--container-runtime-endpointunix:///run/containerd/containerd.sock --pod-infra-container-imagek8s.gcr.io/pause:3.9 EnvironmentKUBELET_EXTRA_ARGS--cgroup-driversystemd --fail-swap-onfalse --node-labelsnode.kubernetes.io/oslinux,node.kubernetes.io/archarm64 EOF sudo systemctl daemon-reload sudo systemctl restart kubelet为什么必须加--node-labelsK8S 1.26调度器默认过滤无kubernetes.io/os和kubernetes.io/arch标签的节点ARM节点若无此标签kubectl run会pending。从那以后我每次部署Kylin V10 ARM集群都会在kubeadm init前先跑一遍etcdctl endpoint health --cluster和ctr -n k8s.io images list再检查/etc/containerd/config.toml的SystemdCgroup和/var/lib/kubelet/config.yaml的cgroupDriver是否都是system本文还有配套的精品资源点击获取
