前言前面讲的 Exporter 都是白盒监控——需要在被监控方安装 Agent。但有些场景你无法在目标上安装任何东西外部 API、第三方服务、网络设备。Blackbox Exporter 从外部探测目标属于黑盒监控。一、白盒 vs 黑盒监控白盒监控 目标内部 → 安装 Exporter → 暴露内部指标 → CPU、内存、QPS、错误率 → 回答为什么出问题了 黑盒监控 外部 → 探测目标 → 验证可达性 → HTTP 状态码、响应时间、证书有效期 → 回答是否出问题了维度白盒黑盒部署位置目标内部外部可见性内部细节外部行为告警含义即将出问题已经出问题适用自己的服务外部/不可控目标二、安装 Blackbox Exporter二进制wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.25.0/blackbox_exporter-0.25.0.linux-amd64.tar.gz tar xzf blackbox_exporter-0.25.0.linux-amd64.tar.gz cd blackbox_exporter-0.25.0.linux-amd64 sudo cp blackbox_exporter /usr/local/bin/配置文件# /etc/blackbox/blackbox.yml modules: # HTTP 探测 http_2xx: prober: http timeout: 5s http: preferred_ip_protocol: ip4 ip_protocol_fallback: false valid_http_versions: [HTTP/1.1, HTTP/2.0] valid_status_codes: [200, 301, 302] method: GET follow_redirects: true headers: User-Agent: Prometheus-Blackbox-Exporter # TLS 配置 tls_config: insecure_skip_verify: false # HTTPS 探测关注证书 http_2xx_check_tls: prober: http timeout: 5s http: valid_status_codes: [200] method: GET tls_config: insecure_skip_verify: false # POST 请求探测 http_post_2xx: prober: http timeout: 5s http: method: POST headers: Content-Type: application/json body: {key:value} # TCP 端口探测 tcp_connect: prober: tcp timeout: 5s # SSH 端口探测banner 检查 ssh_banner: prober: tcp timeout: 5s tcp: query_response: - expect: ^SSH-2.0- # ICMP ping icmp: prober: icmp timeout: 5s icmp: preferred_ip_protocol: ip4 # DNS 解析 dns_check: prober: dns timeout: 5s dns: query_name: example.com query_type: A valid_rcodes: [NOERROR]Systemd 服务# /etc/systemd/system/blackbox_exporter.service [Unit] DescriptionBlackbox Exporter Afternetwork.target [Service] Typesimple ExecStart/usr/local/bin/blackbox_exporter \ --config.file/etc/blackbox/blackbox.yml \ --web.listen-address0.0.0.0:9115 \ --log.levelinfo Restarton-failure [Install] WantedBymulti-user.targetDocker 部署docker run -d \ --name blackbox-exporter \ -p 9115:9115 \ -v /etc/blackbox/blackbox.yml:/etc/blackbox/blackbox.yml \ prom/blackbox-exporter:v0.25.0 \ --config.file/etc/blackbox/blackbox.yml三、HTTP 探测手动探测# 探测一个 URL curl http://localhost:9115/probe?targethttps://example.commodulehttp_2xx # 查看探测结果 probe_success 1 probe_duration_seconds 0.234 probe_http_status_code 200 probe_ssl_earliest_cert_expiry 1.738688e09Prometheus 配置scrape_configs: - job_name: blackbox-http metrics_path: /probe params: module: [http_2xx] static_configs: - targets: - https://www.example.com - https://api.example.com/health - http://internal-app:8080/health relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115关键指标# 1. 探测成功0失败1成功 probe_success # 2. 响应时间 probe_duration_seconds # 3. HTTP 状态码 probe_http_status_code # 4. SSL 证书剩余天数 (probe_ssl_earliest_cert_expiry - time()) / 86400 # 5. DNS 解析时间 probe_dns_lookup_time_seconds # 6. TLS 握手时间 probe_tls_handshake_seconds # 7. HTTP 版本 probe_http_version四、TCP 端口探测scrape_configs: - job_name: blackbox-tcp metrics_path: /probe params: module: [tcp_connect] static_configs: - targets: - mysql:3306 - redis:6379 - kafka:9092 - postgres:5432 labels: probe_type: tcp relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115五、ICMP Ping 探测scrape_configs: - job_name: blackbox-icmp metrics_path: /probe params: module: [icmp] static_configs: - targets: - 8.8.8.8 - 8.8.4.4 - 114.114.114.114 - 192.168.1.1 - 10.0.0.1 labels: probe_type: icmp relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115⚠️踩坑提示ICMP 探测需要 root 权限或setcap cap_net_rawep /usr/local/bin/blackbox_exporter。# 设置 cap_net_raw 权限 sudo setcap cap_net_rawep /usr/local/bin/blackbox_exporter六、DNS 解析探测scrape_configs: - job_name: blackbox-dns metrics_path: /probe params: module: [dns_check] static_configs: - targets: - 8.8.8.8 # DNS 服务器 labels: dns_query: example.com relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115七、SSL 证书监控scrape_configs: - job_name: ssl-cert-expiry metrics_path: /probe params: module: [http_2xx_check_tls] static_configs: - targets: - https://www.example.com - https://api.example.com - https://admin.example.com relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115# 证书剩余天数 (probe_ssl_earliest_cert_expiry - time()) / 86400 # 证书过期告警30 天 ((probe_ssl_earliest_cert_expiry - time()) / 86400) 30八、告警规则groups: - name: blackbox-alerts rules: # 服务不可达 - alert: ServiceDown expr: probe_success 0 for: 2m labels: severity: critical annotations: summary: Service {{ $labels.instance }} is down description: Probe to {{ $labels.instance }} has been failing for 2 minutes # 响应时间过长 - alert: SlowResponse expr: probe_duration_seconds 5 for: 5m labels: severity: warning annotations: summary: Slow response from {{ $labels.instance }} description: Response time is {{ $value }}s (threshold: 5s) # SSL 证书即将过期 - alert: SSLCertExpiringSoon expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 30 for: 1h labels: severity: warning annotations: summary: SSL cert for {{ $labels.instance }} expires soon description: Certificate expires in {{ $value | printf %.0f }} days # SSL 证书即将过期紧急 - alert: SSLCertExpiringCritical expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 7 for: 1h labels: severity: critical annotations: summary: SSL cert for {{ $labels.instance }} expires in 7 days! # HTTP 状态码异常 - alert: HTTPStatusError expr: probe_http_status_code 500 for: 2m labels: severity: critical annotations: summary: HTTP {{ $labels.instance }} returning {{ $value }} # DNS 解析失败 - alert: DNSResolutionFailed expr: probe_dns_rcode ! 0 for: 5m labels: severity: warning annotations: summary: DNS resolution failed for {{ $labels.instance }}九、动态目标配置从 Consul 发现scrape_configs: - job_name: blackbox-consul metrics_path: /probe params: module: [http_2xx] consul_sd_configs: - server: consul:8500 services: [web, api] relabel_configs: - source_labels: [__address__] target_label: __param_target regex: (.):(?:\d) replacement: http://$1/ - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115从文件发现scrape_configs: - job_name: blackbox-file metrics_path: /probe params: module: [http_2xx] file_sd_configs: - files: [/etc/prometheus/targets/blackbox/*.yml] relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115# /etc/prometheus/targets/blackbox/services.yml - targets: - https://service1.example.com - https://service2.example.com labels: team: payments env: production - targets: - https://service3.example.com labels: team: orders env: production要点回顾探测类型module场景HTTPhttp_2xxWeb 服务可用性HTTPSTLShttp_2xx_check_tls证书监控TCPtcp_connect数据库/中间件端口SSHssh_bannerSSH 服务检查ICMPicmp主机存活检查DNSdns_checkDNS 解析监控Blackbox 是外部探测无需在目标安装 Agent通过relabel_configs将 target 转为参数传给 BlackboxHTTP 探测可监控状态码、响应时间、SSL 证书ICMP 探测需要 cap_net_raw 权限SSL 证书过期告警是 Blackbox 最常用场景下一篇预告现有 Exporter 覆盖了常见场景但有时需要监控自定义业务指标。下一篇【Prometheus·Exporter 篇】自定义 Exporter 开发Client Library 实战将手把手开发自己的 Exporter。
前言前面讲的 Exporter 都是白盒监控——需要在被监控方安装 Agent。但有些场景你无法在目标上安装任何东西外部 API、第三方服务、网络设备。Blackbox Exporter 从外部探测目标属于黑盒监控。一、白盒 vs 黑盒监控白盒监控 目标内部 → 安装 Exporter → 暴露内部指标 → CPU、内存、QPS、错误率 → 回答为什么出问题了 黑盒监控 外部 → 探测目标 → 验证可达性 → HTTP 状态码、响应时间、证书有效期 → 回答是否出问题了维度白盒黑盒部署位置目标内部外部可见性内部细节外部行为告警含义即将出问题已经出问题适用自己的服务外部/不可控目标二、安装 Blackbox Exporter二进制wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.25.0/blackbox_exporter-0.25.0.linux-amd64.tar.gz tar xzf blackbox_exporter-0.25.0.linux-amd64.tar.gz cd blackbox_exporter-0.25.0.linux-amd64 sudo cp blackbox_exporter /usr/local/bin/配置文件# /etc/blackbox/blackbox.yml modules: # HTTP 探测 http_2xx: prober: http timeout: 5s http: preferred_ip_protocol: ip4 ip_protocol_fallback: false valid_http_versions: [HTTP/1.1, HTTP/2.0] valid_status_codes: [200, 301, 302] method: GET follow_redirects: true headers: User-Agent: Prometheus-Blackbox-Exporter # TLS 配置 tls_config: insecure_skip_verify: false # HTTPS 探测关注证书 http_2xx_check_tls: prober: http timeout: 5s http: valid_status_codes: [200] method: GET tls_config: insecure_skip_verify: false # POST 请求探测 http_post_2xx: prober: http timeout: 5s http: method: POST headers: Content-Type: application/json body: {key:value} # TCP 端口探测 tcp_connect: prober: tcp timeout: 5s # SSH 端口探测banner 检查 ssh_banner: prober: tcp timeout: 5s tcp: query_response: - expect: ^SSH-2.0- # ICMP ping icmp: prober: icmp timeout: 5s icmp: preferred_ip_protocol: ip4 # DNS 解析 dns_check: prober: dns timeout: 5s dns: query_name: example.com query_type: A valid_rcodes: [NOERROR]Systemd 服务# /etc/systemd/system/blackbox_exporter.service [Unit] DescriptionBlackbox Exporter Afternetwork.target [Service] Typesimple ExecStart/usr/local/bin/blackbox_exporter \ --config.file/etc/blackbox/blackbox.yml \ --web.listen-address0.0.0.0:9115 \ --log.levelinfo Restarton-failure [Install] WantedBymulti-user.targetDocker 部署docker run -d \ --name blackbox-exporter \ -p 9115:9115 \ -v /etc/blackbox/blackbox.yml:/etc/blackbox/blackbox.yml \ prom/blackbox-exporter:v0.25.0 \ --config.file/etc/blackbox/blackbox.yml三、HTTP 探测手动探测# 探测一个 URL curl http://localhost:9115/probe?targethttps://example.commodulehttp_2xx # 查看探测结果 probe_success 1 probe_duration_seconds 0.234 probe_http_status_code 200 probe_ssl_earliest_cert_expiry 1.738688e09Prometheus 配置scrape_configs: - job_name: blackbox-http metrics_path: /probe params: module: [http_2xx] static_configs: - targets: - https://www.example.com - https://api.example.com/health - http://internal-app:8080/health relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115关键指标# 1. 探测成功0失败1成功 probe_success # 2. 响应时间 probe_duration_seconds # 3. HTTP 状态码 probe_http_status_code # 4. SSL 证书剩余天数 (probe_ssl_earliest_cert_expiry - time()) / 86400 # 5. DNS 解析时间 probe_dns_lookup_time_seconds # 6. TLS 握手时间 probe_tls_handshake_seconds # 7. HTTP 版本 probe_http_version四、TCP 端口探测scrape_configs: - job_name: blackbox-tcp metrics_path: /probe params: module: [tcp_connect] static_configs: - targets: - mysql:3306 - redis:6379 - kafka:9092 - postgres:5432 labels: probe_type: tcp relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115五、ICMP Ping 探测scrape_configs: - job_name: blackbox-icmp metrics_path: /probe params: module: [icmp] static_configs: - targets: - 8.8.8.8 - 8.8.4.4 - 114.114.114.114 - 192.168.1.1 - 10.0.0.1 labels: probe_type: icmp relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115⚠️踩坑提示ICMP 探测需要 root 权限或setcap cap_net_rawep /usr/local/bin/blackbox_exporter。# 设置 cap_net_raw 权限 sudo setcap cap_net_rawep /usr/local/bin/blackbox_exporter六、DNS 解析探测scrape_configs: - job_name: blackbox-dns metrics_path: /probe params: module: [dns_check] static_configs: - targets: - 8.8.8.8 # DNS 服务器 labels: dns_query: example.com relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115七、SSL 证书监控scrape_configs: - job_name: ssl-cert-expiry metrics_path: /probe params: module: [http_2xx_check_tls] static_configs: - targets: - https://www.example.com - https://api.example.com - https://admin.example.com relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115# 证书剩余天数 (probe_ssl_earliest_cert_expiry - time()) / 86400 # 证书过期告警30 天 ((probe_ssl_earliest_cert_expiry - time()) / 86400) 30八、告警规则groups: - name: blackbox-alerts rules: # 服务不可达 - alert: ServiceDown expr: probe_success 0 for: 2m labels: severity: critical annotations: summary: Service {{ $labels.instance }} is down description: Probe to {{ $labels.instance }} has been failing for 2 minutes # 响应时间过长 - alert: SlowResponse expr: probe_duration_seconds 5 for: 5m labels: severity: warning annotations: summary: Slow response from {{ $labels.instance }} description: Response time is {{ $value }}s (threshold: 5s) # SSL 证书即将过期 - alert: SSLCertExpiringSoon expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 30 for: 1h labels: severity: warning annotations: summary: SSL cert for {{ $labels.instance }} expires soon description: Certificate expires in {{ $value | printf %.0f }} days # SSL 证书即将过期紧急 - alert: SSLCertExpiringCritical expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 7 for: 1h labels: severity: critical annotations: summary: SSL cert for {{ $labels.instance }} expires in 7 days! # HTTP 状态码异常 - alert: HTTPStatusError expr: probe_http_status_code 500 for: 2m labels: severity: critical annotations: summary: HTTP {{ $labels.instance }} returning {{ $value }} # DNS 解析失败 - alert: DNSResolutionFailed expr: probe_dns_rcode ! 0 for: 5m labels: severity: warning annotations: summary: DNS resolution failed for {{ $labels.instance }}九、动态目标配置从 Consul 发现scrape_configs: - job_name: blackbox-consul metrics_path: /probe params: module: [http_2xx] consul_sd_configs: - server: consul:8500 services: [web, api] relabel_configs: - source_labels: [__address__] target_label: __param_target regex: (.):(?:\d) replacement: http://$1/ - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115从文件发现scrape_configs: - job_name: blackbox-file metrics_path: /probe params: module: [http_2xx] file_sd_configs: - files: [/etc/prometheus/targets/blackbox/*.yml] relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115# /etc/prometheus/targets/blackbox/services.yml - targets: - https://service1.example.com - https://service2.example.com labels: team: payments env: production - targets: - https://service3.example.com labels: team: orders env: production要点回顾探测类型module场景HTTPhttp_2xxWeb 服务可用性HTTPSTLShttp_2xx_check_tls证书监控TCPtcp_connect数据库/中间件端口SSHssh_bannerSSH 服务检查ICMPicmp主机存活检查DNSdns_checkDNS 解析监控Blackbox 是外部探测无需在目标安装 Agent通过relabel_configs将 target 转为参数传给 BlackboxHTTP 探测可监控状态码、响应时间、SSL 证书ICMP 探测需要 cap_net_raw 权限SSL 证书过期告警是 Blackbox 最常用场景下一篇预告现有 Exporter 覆盖了常见场景但有时需要监控自定义业务指标。下一篇【Prometheus·Exporter 篇】自定义 Exporter 开发Client Library 实战将手把手开发自己的 Exporter。
