③prometheus报警

通过在Prometheus中定义AlertRule(告警规则),Prometheus会周期性的对告警规则进行计算，如果满足告警触发条件就会向Alertmanager发送告警信息。

prometheus发送邮件报警需要alertmanager组件支持

安装运行alertmanager

wget https://github.com/prometheus/alertmanager/releases/download/v0.21.0/alertmanager-0.21.0.linux-amd64.tar.gz
tar zxvf alertmanager-0.21.0.linux-amd64.tar.gz -C /usr/local/
cd /usr/local/
ln -sv alertmanager-0.21.0.linux-amd64/ alertmanager
cd /alertmanager
./alertmanager

在监控服务器上写一个简单的告警规则

cat /usr/local/prometheus/alert_config/alert_rules/instance_down.yaml

groups:
- name: AllInstances
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 10s
    annotations:
      title: 'Instance down'
      description: Instance has been down for more than 1 minute.'
    labels:
      severity: 'critical'

编写alertmanager邮件告警

vi /usr/local/alertmanager/alertmanager.yml

global:
  resolve_timeout: 5m
  smtp_from: 'XXXXXXXX@163.com'
  smtp_smarthost: 'smtp.163.com:465'
  smtp_auth_username: 'XXXXXXXX@163.com'
  smtp_auth_password: 'XXXXXXXXXXXXX'
  smtp_require_tls: false
  smtp_hello: 'qq.com'
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 10s
  receiver: 'email-me'
receivers:
- name: 'email-me'
  email_configs:
  - to: 'XXXXXXXX@163.com'
    send_resolved: true

邮件告警规则写好了，alertmanager邮件告警也配置好了，这时候在修改下prometheus的配置文件，使报警生效

......
# 这里是安装alertmanager组件的主机和端口，触发报警规则之后，prometheus向alertmanager发起请求
# Alertmanager configuration 
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - 192.168.32.130:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "alert_config/alert_rules/*.yaml"
......

之后重启prometheus服务，还有node_exporter服务，再次进入192.168.32.130:9090可以看到告警规则

此时停掉一个node_exporter,可以看到告警规则已经生效

此时查看邮箱也发现了alertmanager发来的告警邮件

自定义模板

vi /usr/local/alertmanager/email.tmpl

{{ define "email.from" }}XXXXXXXX@163.com{{ end }}
{{ define "email.to" }}XXXXXXX@163.com{{ end }}
{{ define "email.to.html" }}
{{ range .Alerts }}
=========start==========

告警程序: prometheus_alert 

告警级别: {{ .Labels.severity }} 级 

告警类型: {{ .Labels.alertname }} 

故障主机: {{ .Labels.instance }} 

告警主题: {{ .Annotations.summary }} 

告警详情: {{ .Annotations.description }} 

触发时间: {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }} 

=========end==========

{{ end }}
{{ end }}

定义好模板之后在alertmanager的配置文件中调用

global:
  ......

# 模板路径
templates:
  - '/usr/local/alertmanager/email.tmpl'

receivers:
- name: 'email-me'
  email_configs:
  - to: 'xxxxxxxx@163.com'
    html: '{{ template "email.to.html" . }}' # 调用模板
    send_resolved: true

告警抑制

如果问题不处理，alertmanager会一直发送报警邮件，这个时候我们要用到抑制机制， Alertmanager的抑制机制可以避免当某种问题告警产生之后用户接收到大量由此问题导致的一系列的其它告警通知。

vi /usr/local/alertmanager/alertmanager.yml

global:
  resolve_timeout: 5m
  smtp_from: 'XXXXXXXX@163.com'
  smtp_smarthost: 'smtp.163.com:465'
  smtp_auth_username: 'XXXXXXXX918@163.com'
  smtp_auth_password: 'XXXXXXXX'
  smtp_require_tls: false
  smtp_hello: 'qq.com'
route: 
  # 注释掉组选项，避免对消息抑制造成影响
  #group_by: ['alertname']
  #group_wait: 10s
  #group_interval: 10s
  repeat_interval: 10s
  receiver: 'email-me'
receivers:
- name: 'email-me'
  email_configs:
  - to: 'XXXXXXXX@163.com'
    send_resolved: true

# 告警抑制
inhibit_rules:
- source_match:
    alertname: InstanceDown
    severity: critical
  target_match:
    severity: critical
  equal:
   - instance

prometheus如果要使用钉钉告警，需要dingtalk插件的支持，下载安装dingtalk

wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v1.4.0/prometheus-webhook-dingtalk-1.4.0.linux-amd64.tar.gz
tar -zxvf prometheus-webhook-dingtalk-1.4.0.linux-amd64.tar.gz -C /usr/local/
cd /usr/local/
ln -sv prometheus-webhook-dingtalk-1.4.0.linux-amd64/ dingtalk

钉钉建一个群，里边添加一个自定义机器人，复制下来它的webhook和加签信息，修改配置文件

vi /usr/local/dingtalk/config.yml

...
targets:
  webhook1:
    # 钉钉机器人接口
    url: https://oapi.dingtalk.com/robot/send?access_token=9773c2bc976a813481e0c8f6c6668b944e5cc8478ae5206ff02cdb467e751dcb
    # 钉钉加签
    secret: SECdc47fe6d66575724514eafb13388ab19c452ccb15d11aeeb70b3bfabaa4e3bbc

修改dingtalk为系统服务，方便管理

vi /etc/systemd/system/dingtalk.service

[Unit]
Description=prometheus-webhook-dingtalk
After=network-online.target

[Service]
Restart=on-failure
ExecStart=/usr/local/dingtalk/prometheus-webhook-dingtalk --config.file=/usr/local/dingtalk/config.yml

[Install]
WantedBy=multi-user.target

启动dingtalk

systemctl start dingtalk    # 启动之后监听8060端口

修改alertmanager.yml

global:
  resolve_timeout: 5m
route:
  #group_by: ['alertname', 'severity', 'namespace']
  #group_wait: 10s
  #group_interval: 10s
  repeat_interval: 10s
  receiver: 'dingding.webhook1'
  routes:
  - receiver: 'dingding.webhook1'
    match:
      team: DevOps
    group_wait: 10s
    group_interval: 15s
    repeat_interval: 3h
  - receiver: 'dingding.webhook.all'
    match:
      team: SRE
    group_wait: 10s
    group_interval: 15s
    repeat_interval: 3h
receivers:
- name: 'dingding.webhook1'
  webhook_configs:
  - url: 'http://192.168.1.1:8060/dingtalk/webhook1/send'
    send_resolved: true
- name: 'dingding.webhook.all'
  webhook_configs:
  - url: 'http://192.168.1.1:8060/dingtalk/webhook_mention_all/send'
    send_resolved: true

inhibit_rules:
- source_match:
    alertname: InstanceDown
    severity: critical
  target_match:
    severity: critical
  equal:
   - instance

重启alertmanager。停掉node_exporter测试，成功收到告警信息