【Prometheus】第三篇:配置alertmamager
监控系统中非常重要的一环,就是告警,系统得在故障发生的第一时间将事件发送出来,通知干系人,prometheus提供了alertmanager来实现这个功能。
第一步:prometheus.yml配置文件,配置alertmanager地址
第二步:编写触发器,也就是在什么情况下产生告警。
Prometheus.yml填写触发器配置文件路径
alert_rule.yml内容
groups: - name: node rules: - alert: node_cpu>80% expr: (1-rate(node_cpu_seconds_total{mode="idle"}[1m]))*100 > 80 labels: severity: 3 - alert: node_mem_availble<20% expr: node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes*100 < 20 labels: severity: 3 - alert: node_cpu_load>10 expr: node_load1 > 10 labels: severity: 3 - alert: node_disk<20% expr: node_filesystem_avail_bytes{device!='nsfs'}/node_filesystem_size_bytes{device!='nsfs'}*100 < 20 labels: severity: 3 - name: docker rules: - alert: docker_cpu>50% expr: rate(container_cpu_usage_seconds_total{image!=''}[1m])*100 > 50 labels: severity: 3 - alert: docker_restarted expr: changes(container_start_time_seconds[1m]) != 0 labels: severity: 4
其中expr就是产生告警的条件,即当这个语句条件成立时,触发告警,下面的labels是告警内容中的标签,这里添加了一个标签,即告警等级severity,可以自定义1-5,来区分不同级别的告警。
第三步:产生的告警怎么处理,是发消息?发送给谁?通过什么发送?都是在这里配置。alertmanager.yml配置文件
内容如下:
global:
resolve_timeout: 5m
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
repeat_interval: 1h
receiver: 'wechat'
routes:
- match_re:
severity: 1|2|3|4|5
receiver: 'wechat'
continue: true
- match:
severity: 5
receiver: 'message'
continue: true
- match:
severity: 5
receiver: 'call'
continue: true
receivers:
- name: 'wechat'
webhook_configs:
- url: 'http://localhost/alert_wechat'
- name: 'message'
webhook_configs:
- url: 'http://localhost/alert_message'
- name: 'call'
webhook_configs:
- url: 'http://localhost/alert_call'
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']
这里用了一个receiver,即web_hook,Prometheus会把告警内容post到指定的url地址。