【Prometheus·告警篇】告警规则：Recording Rules 与 Alerting Rules 最佳实践

前言Prometheus 的规则分为两类Recording Rules预计算指标和 Alerting Rules告警规则。写好规则是做监控告警的核心技能。本篇详解两类规则的语法、设计原则和常见模式。一、Recording Rules预计算指标为什么需要 Recording Rules# 复杂查询 sum(rate(http_requests_total{methodGET}[5m])) by (service, status) / sum(rate(http_requests_total[5m])) by (service) * 100 # 问题 # 1. 每次查询都要重新计算 # 2. Dashboard 频繁查询 → Prometheus 负载高 # 3. 范围查询更慢要逐点计算# Recording Rule 预计算 # http_request_error_rate:precomputed → 存入 TSDB # 查询时直接读预计算结果O(1)语法# rules/recording.yml groups: - name: http_metrics interval: 30s # 每 30 秒评估一次 rules: - record: job:http_requests:rate5m # 指标名 expr: sum by (job)(rate(http_requests_total[5m])) # 可选添加标签 labels: team: platform - record: job:http_requests:error_rate expr: | sum(rate(http_requests_total{status~5..}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job) * 100 - record: instance:cpu_usage:ratio expr: | 1 - (avg(rate(node_cpu_seconds_total{modeidle}[5m])) by (instance))命名约定规则名格式 scope:metric:operation 示例 job:http_requests:rate5m # 范围:指标:操作 instance:cpu_usage:ratio # 实例级 cluster:memory_usage:percentage # 集群级使用预计算指标# 查询预计算结果 job:http_requests:rate5m # Dashboard 查询更快 job:http_requests:error_rate 5 # 告警规则用预计算指标 expr: job:http_requests:error_rate 5完整示例# rules/recording_rules.yml groups: # HTTP 指标 - name: http_recording interval: 30s rules: - record: job:http_requests:rate5m expr: sum by (job, method)(rate(http_requests_total[5m])) - record: job:http_requests:rate1m expr: sum by (job, method)(rate(http_requests_total[1m])) - record: job:http_requests:error_rate5m expr: | sum by (job)(rate(http_requests_total{status~5..}[5m])) / sum by (job)(rate(http_requests_total[5m])) * 100 - record: job:http_request_duration:p99 expr: | histogram_quantile(0.99, sum by (job, le)(rate(http_request_duration_seconds_bucket[5m]))) - record: job:http_request_duration:p95 expr: | histogram_quantile(0.95, sum by (job, le)(rate(http_request_duration_seconds_bucket[5m]))) # 资源指标 - name: resource_recording interval: 30s rules: - record: instance:cpu_usage:ratio expr: 1 - (avg by(instance)(rate(node_cpu_seconds_total{modeidle}[5m]))) - record: instance:memory_usage:ratio expr: 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) - record: instance:disk_usage:ratio expr: 1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes) - record: instance:network_rx:rate5m expr: rate(node_network_receive_bytes_total{device!~lo|docker.*|veth.*}[5m]) * 8 - record: instance:network_tx:rate5m expr: rate(node_network_transmit_bytes_total{device!~lo|docker.*|veth.*}[5m]) * 8 # Kubernetes 指标 - name: k8s_recording interval: 30s rules: - record: namespace:pod_count expr: count by(namespace)(kube_pod_status_phase{phaseRunning}) - record: namespace:cpu_request:total expr: sum by(namespace)(kube_pod_container_resource_requests{resourcecpu}) - record: namespace:memory_usage:total expr: sum by(namespace)(container_memory_working_set_bytes{container!})二、Alerting Rules告警规则语法# rules/alerting.yml groups: - name: alerting_rules interval: 15s rules: - alert: HighCPUUsage # 告警名唯一 expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{modeidle}[5m])) * 100) 80 for: 5m # 持续 5 分钟才告警 labels: severity: warning # 告警标签 annotations: summary: CPU usage 80% on {{ $labels.instance }} description: CPU usage is {{ $value | printf \%.2f\ }}% (threshold: 80%) runbook_url: https://wiki.example.com/runbooks/high-cpu关键字段字段说明alert告警名称唯一标识exprPromQL 表达式返回非空则触发for持续时间防止瞬时波动误告警labels附加标签用于路由annotations描述信息支持模板for 的三阶段告警生命周期 Pending等待中→ Firing触发中→ Resolved已恢复 expr 成立 → Pending → for 时间后仍成立 → Firing → 发送告警 expr 不成立 → Resolved → 发送恢复通知send_resolvedtrue示例for: 5m 10:00 expr 成立 → Pending 10:03 expr 仍成立 → 继续 Pending 10:05 仍成立 → Firing → 发送告警 10:08 expr 不成立 → Resolved → 发送恢复通知三、告警规则最佳实践1. 分级告警groups: - name: severity-based rules: # Critical立即处理 - alert: ServiceDown expr: up 0 for: 1m labels: severity: critical annotations: summary: Service {{ $labels.job }} is down # Warning关注但不紧急 - alert: HighCPU expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{modeidle}[5m])) * 100) 80 for: 10m labels: severity: warning annotations: summary: High CPU on {{ $labels.instance }} # Info仅记录 - alert: DiskSpaceLow expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 70 for: 30m labels: severity: info annotations: summary: Disk space low on {{ $labels.instance }}2. 使用预计算指标# ❌ 直接复杂查询慢 - alert: HighErrorRate expr: | sum(rate(http_requests_total{status~5..}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service) * 100 5 for: 5m # ✅ 用 Recording Rule快 - alert: HighErrorRate expr: job:http_requests:error_rate5m 5 for: 5m3. 注解模板- alert: HighMemory expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 90 for: 5m labels: severity: critical annotations: summary: Memory usage 90% on {{ $labels.instance }} description: Memory usage: {{ $value | printf \%.1f\ }}%, available: {{ with printf \node_memory_MemAvailable_bytes{instance\\\%s\\\}\ $labels.instance | query }}{{ . | first | value | humanize1024 }}{{ end }} dashboard: https://grafana.example.com/d/node?var-instance{{ $labels.instance }} runbook: https://wiki.example.com/runbooks/high-memory4. 避免告警风暴# 使用 for 防止瞬时波动 - alert: PodCrashLooping expr: rate(kube_pod_container_status_restarts_total[15m]) 0 for: 15m # 持续 15 分钟才告警 labels: severity: warning # 使用 count 控制批量告警 - alert: ManyServicesDown expr: count(up 0) 3 # 超过 3 个服务宕机才告警 for: 2m labels: severity: critical四、完整告警规则集# rules/alerting_rules.yml groups: # 基础设施告警 - name: infrastructure rules: # 主机宕机 - alert: NodeDown expr: up{jobnode-exporter} 0 for: 2m labels: severity: critical annotations: summary: Node {{ $labels.instance }} is down # CPU 高 - alert: HighCPU expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{modeidle}[5m])) * 100) 80 for: 10m labels: severity: warning annotations: summary: CPU 80% on {{ $labels.instance }} # 内存高 - alert: HighMemory expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 90 for: 5m labels: severity: critical annotations: summary: Memory 90% on {{ $labels.instance }} # 磁盘空间不足 - alert: DiskSpaceLow expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 85 for: 5m labels: severity: warning annotations: summary: Disk 85% on {{ $labels.instance }} {{ $labels.mountpoint }} # 磁盘满预测 - alert: DiskWillFillIn4h expr: predict_linear(node_filesystem_avail_bytes[1h], 4*3600) 0 for: 10m labels: severity: warning annotations: summary: Disk will fill in 4h on {{ $labels.instance }} # 系统负载高 - alert: HighLoad expr: node_load1 count by(instance)(node_cpu_seconds_total{modeidle}) * 1.5 for: 10m labels: severity: warning # MTRMean Time to Recover监控 - alert: NetworkErrors expr: rate(node_network_receive_errs_total[5m]) 0 for: 5m labels: severity: warning # Kubernetes 告警 - name: kubernetes rules: # Pod CrashLoopBackOff - alert: PodCrashLooping expr: increase(kube_pod_container_status_restarts_total[15m]) 5 for: 5m labels: severity: warning # Pod Not Ready - alert: PodNotReady expr: kube_pod_status_ready{conditiontrue} 0 for: 10m labels: severity: warning # Deployment 副本不足 - alert: DeploymentReplicasMismatch expr: kube_deployment_spec_replicas ! kube_deployment_status_replicas for: 10m labels: severity: warning # Node Not Ready - alert: KubernetesNodeNotReady expr: kube_node_status_condition{conditionReady,status!true} 1 for: 5m labels: severity: critical # 应用告警 - name: application rules: # 高错误率 - alert: HighErrorRate expr: | sum(rate(http_requests_total{status~5..}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service) * 100 5 for: 5m labels: severity: critical # 高延迟 P99 - alert: HighLatency expr: | histogram_quantile(0.99, sum by (service, le)(rate(http_request_duration_seconds_bucket[5m]))) 1 for: 5m labels: severity: warning # 服务不可达 - alert: ServiceUnavailable expr: up{jobapp} 0 for: 2m labels: severity: critical # 数据库告警 - name: database rules: - alert: MySQLDown expr: mysql_up 0 for: 2m labels: severity: critical - alert: MySQLReplicationLag expr: mysql_slave_status_seconds_behind_master 300 for: 5m labels: severity: warning - alert: MySQLTooManyConnections expr: mysql_global_status_threads_connected / mysql_global_variables_max_connections * 100 80 for: 5m labels: severity: warning - alert: RedisDown expr: redis_up 0 for: 2m labels: severity: critical - alert: RedisMemoryFull expr: redis_memory_used_bytes / redis_memory_max_bytes * 100 90 for: 5m labels: severity: critical五、配置与管理Prometheus 配置# prometheus.yml rule_files: - rules/recording/*.yml - rules/alerting/*.yml alerting: alertmanagers: - static_configs: - targets: [alertmanager:9093]验证规则# 检查规则语法 promtool check rules rules/recording/*.yml rules/alerting/*.yml # 检查配置 promtool check config prometheus.yml # 查看已加载的规则 curl http://localhost:9090/api/v1/rules | jq # 查看当前告警 curl http://localhost:9090/api/v1/alerts | jq热重载# 修改规则文件后热重载 curl -X POST http://localhost:9090/-/reload⚠️踩坑提示- 告警名称必须全局唯一-for时间不要太短推荐至少 1-5 分钟- 避免for: 0瞬时告警容易误报- Recording Rule 指标名不要和原始指标名冲突- 每条规则都要写 annotations方便接手人理解要点回顾规则类型作用使用场景Recording Rules预计算存储指标复杂查询、频繁查询Alerting Rules触发告警所有需要告警的场景Recording Rule 命名scope:metric:operationAlerting Rule 三阶段Pending → Firing → Resolvedfor防瞬时波动至少 1-5 分钟告警分级critical / warning / infoannotations 写清 summary description runbook验证用 promtool check rules下一篇预告告警规则搞定了下一篇【Prometheus·可视化篇】Grafana 集成数据源配置与 Dashboard 设计原则将讲解如何用 Grafana 做可视化。