Prometheus实战之配置汇总

　　导航:这里主要是列出一个prometheus一些系统的学习过程，最后按照章节顺序查看，由于写作该文档经历了不同时期，所以在文中有时出现

的云环境不统一，但是学习具体使用方法即可，在最后的篇章，有一个完整的腾讯云的实战案例。

　　1.

　　2.

　　3.

　　4.

　　5.

　　6.

　　7.

　　8.

　　9.

　　10.

　　11.

　　12.

　　13.

　　14.

　　15.

　　参考:

　　https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

　　https://yunlzheng.gitbook.io/prometheus-book/part-iii-prometheus-shi-zhan/readmd/use-prometheus-monitor-kubernetes

　　https://www.bookstack.cn/read/prometheus_practice/introduction-README.md

　　https://www.kancloud.cn/huyipow/prometheus/521184

　　https://www.qikqiak.com/k8s-book/docs/

　　因为记录此文档经历了几次云环境的更换，且在各篇章讲解的过程中配置列出不够详细，所以这里做一个汇总，更容易查阅。（代码中敏感信息已经隐藏）

1.主prometheus配置

global:
  scrape_interval:     15s
  evaluation_interval: 15s
rule_files:
  - /opt/prometheus/rule/*.rules

#remote_write:
#  - url: "http://influxdb:8086/api/v1/prom/write?u=root&p=huawei&db=prometheus"
    #basic_auth:
      #username: root
      #password: huawei
#remote_read:
  #- url: "http://influxdb:8086/api/v1/prom/read?u=root&p=huawei&db=prometheus"
    #basic_auth:
      #username: root
      #password: huawei

alerting:
  alertmanagers:
    - static_configs:
      - targets: ["10.10.1.10:9093","10.10.1.5:9093"]

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['prometheus01:9100','prometheus02:9100','influxdb:9100']
      labels:
        project: prometheus

    relabel_configs:
    - source_labels: [__address__]
      regex: ([a-zA-Z0-9-.]+):(\d{1,4})
      target_label: hostname
      replacement: ${1}


  - job_name: "cn-lcm-prod"
    scrape_interval: 20s
    scrape_timeout: 20s
    honor_labels: true
    metrics_path: '/federate'

    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'
        - '{job="cn-lcm-prod-kubernetes-kubelet"}'
        - '{job="cn-lcm-prod-kubernetes-cadvisor"}'
        - '{job="cn-lcm-prod-kubernetes-pods"}'
        - '{job="cn-lcm-prod-kubernetes-apiservers"}'
        - '{job="cn-lcm-prod-kubernetes-services"}'
        - '{job="cn-lcm-prod-kubernetes-service-endpoints"}'

    static_configs:
      - targets:
        - 'x.x.x.x:9090'
        labels:
          project: cn-lcm-prod-k8s



  - job_name: "cn-gc"

    scrape_interval: 2s
    scrape_timeout: 2s
    honor_labels: true
    metrics_path: '/federate'

    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'
        - '{job="cn-gc-consul-node"}'
        - '{job="cn-gc-consul-process"}'
        - '{job="cn-gc-nginx01"}'
        - '{job="cn-gc-nginx02"}'

    static_configs:
      - targets:
        - 'x.x.x.x:9090'
        labels:
          project: cn-gc
主prometheus配置

2.主prometheus告警模版样例

cat cn-lcm-prod-kubernetes.rules
groups:
- name: cn-lcm-prod
  rules:
  - alert: PodCpu
    expr: (sum(rate(container_cpu_usage_seconds_total{job=~'cn-lcm-prod.*',container_name!=''}[3m])) by(job,cluster,namespace,container_name,pod_name))/(sum(container_spec_cpu_quota) by(job,cluster,namespace,container_name,pod_name) /100000)*100 > 70
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD PodCpu CPU使用率已达到告警阈值"
      description: "{{ $labels.pod_name }} 告警阈值为70%, 当前值为: {{ $value }}"

  - alert: PodMem
    expr: sum(container_memory_rss{image!="",job=~'cn-lcm-prod.*'}) by(job,cluster,namespace,container_name,pod_name) / sum(container_spec_memory_limit_bytes{image!="",job=~'cn-lcm-prod.*'}) by(job,cluster,namespace,container_name,pod_name) * 100 != +inf  > 90
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD PodMem 内存使用率已达到告警阈值"
      description: "{{ $labels.pod_name }} 告警阈值为90%, 当前值为: {{ $value }}"

  - alert: PodNetworkIn
    expr: sum by (namespace,job,pod_name) (irate(container_network_receive_bytes_total{image!="",job=~'cn-lcm-prod.*'}[1m]))  / 1024 /1024 > 30
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD PodNetworkIn NETWORK入带宽使用率已达到告警阈值"
      description: "{{ $labels.pod_name }} 告警阈值为20MB, 当前值为: {{ $value }} MB"

  - alert: PodNetworkOut
    expr: sum by (namespace,job,pod_name) (irate(container_network_transmit_bytes_total{image!="",job=~'cn-lcm-prod.*'}[1m]))  / 1024 /1024 > 30
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD PodNetworkOut NETWORK出带宽使用率已达到告警阈值"
      description: "{{ $labels.pod_name }} 告警阈值为10MB, 当前值为: {{ $value }} MB"

  - alert: PodRestart
    expr: changes(kube_pod_container_status_restarts_total{job=~"cn-lcm-prod.*"}[3m]) > 0
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD PodRestart POD重启"
      description: "{{ $labels.pod }} 有重启记录"

  - alert: PodCpu
    expr: (sum(rate(container_cpu_usage_seconds_total{job=~'cn-lcm-prod.*',container_name!=""}[3m])) by(job,cluster,namespace,container_name,pod_name))/(sum(container_spec_cpu_quota) by(job,cluster,namespace,container_name,pod_name) /100000)*100 > 90
    for: 1m
    labels:
      severity: high
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD PodCpu CPU使用率已达到告警阈值"
      description: "{{ $labels.pod_name }} 告警阈值为90%, 当前值为: {{ $value }}"

  - alert: PodMem
    expr: sum(container_memory_rss{image!="",job=~'cn-lcm-prod.*'}) by(job,cluster,namespace,container_name,pod_name) / sum(container_spec_memory_limit_bytes{image!="",job=~'cn-lcm-prod.*'}) by(job,cluster,namespace,container_name,pod_name) * 100 != +inf  > 90
    for: 1m
    labels:
      severity: high
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD PodMem 内存使用率已达到告警阈值"
      description: "{{ $labels.pod_name }} 告警阈值为90%, 当前值为: {{ $value }}"



cat cn-lcm-prod-node.rules
groups:
- name: cn-lcm-prod
  rules:
  - alert: NodeCpu
    expr: (1-((sum(increase(node_cpu_seconds_total{mode="idle",job=~'cn-lcm-prod.*'}[1m])) by (instance,job,hostname)) / (sum(increase(node_cpu_seconds_total[1m])) by (instance,job,hostname)))) * 100 > 70
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeCpu  使用率已达到告警阈值"
      description: "{{ $labels.hostname }} 告警阈值为70%, 当前值为: {{ $value }}"

  - alert: NodeMem
    expr: (1-((node_memory_Buffers_bytes + node_memory_Cached_bytes + node_memory_MemFree_bytes) / node_memory_MemTotal_bytes {job=~'cn-lcm-prod.*'}))  * 100 > 80
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeMem 使用率已达到告警阈值"
      description: "{{ $labels.instance }} 告警阈值为80%, 当前值为: {{ $value }}"

  - alert: NodeNetworkIn
    expr: irate(node_network_receive_bytes_total{job=~"cn-lcm-prod.*",device='eth0'}[5m]) /1024 /1024  > 10
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeNetworkIn 网卡 ETH0 使用率已达到告警阈值"
      description: "{{ $labels.instance }} 告警阈值为10MB, 当前值为: {{ $value }} MB"

  - alert: NodeNetworkOut
    expr: irate(node_network_transmit_bytes_total{job=~"cn-lcm-prod.*",device='eth0'}[5m]) /1024 /1024  > 5
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeNetworkOut 网卡 ETH0 使用率已达到告警阈值"
      description: "{{ $labels.instance }} 告警阈值为5MB, 当前值为: {{ $value }} MB"

  - alert: NodeDiskUsed
    expr: (sum(node_filesystem_size_bytes{job=~'cn-lcm-prod.*'}) by(instance,device,hostname) - sum(node_filesystem_free_bytes{job=~'cn-lcm-prod.*'}) by(instance,device,hostname)) / sum(node_filesystem_size_bytes{job=~'cn-lcm-prod.*'}) by(instance,device,hostname) * 100 > 70
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeDiskUsed 磁盘使用率已达到告警阈值"
      description: "{{ $labels.instance }} 告警阈值为70%, 当前值为: {{ $value }}"

  - alert: NodeDiskIopsRead
    expr: irate(node_disk_reads_completed_total{job=~"cn-lcm-prod.*"}[1m]) > 500
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeDiskIopsRead 磁盘读取IOPS已达到告警阈值"
      description: "{{ $labels.instance }} 告警阈值为500, 当前值为: {{ $value }}"

  - alert: NodeDiskIopsWrite
    expr: irate(node_disk_writes_completed_total{job=~"cn-lcm-prod.*"}[1m]) > 500
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeDiskIopsWrite 磁盘写入IOPS已达到告警阈值"
      description: "{{ $labels.instance }} 告警阈值为500, 当前值为: {{ $value }}"

  - alert: NodeDiskReadTime
    expr: irate(node_disk_read_time_seconds_total{job=~"cn-lcm-prod.*"}[1m]) > 2
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeDiskReadTime 磁盘读取时间已达到告警阈值"
      description: "{{ $labels.instance }} 告警阈值为2s, 当前值为: {{ $value }} s"

  - alert: NodeDiskWriteTime
    expr: irate(node_disk_write_time_seconds_total{job=~"cn-lcm-prod.*"}[1m]) > 2
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeDiskWriteTime 磁盘写入时间已达到告警阈值"
      description: "{{ $labels.instance }} 告警阈值为2s, 当前值为: {{ $value }} s"

  - alert: NodeTcpCurrEstab
    expr: node_netstat_Tcp_CurrEstab{job=~"cn-lcm-prod.*"} > 4000
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeTcpCurrEstab TCP当前连接已达到告警阈值"
      description: "{{ $labels.instance }} 告警阈值为500, 当前值为: {{ $value }}"

  - alert: NodeSystemLoad
    expr: node_load5{hostname=~"$node"} > 10
    for: 1m
    labels:
      severity: warning
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeSystemLoad Node 5分钟内负载已达到告警阈值"
      description: "{{ $labels.instance }} 5分钟内负载告警阈值为10, 当前值为: {{ $value }}"


###################high#####################

  - alert: NodeCpu
    expr: (1-((sum(increase(node_cpu_seconds_total{mode="idle",job=~'cn-lcm-prod.*'}[1m])) by (instance,job,hostname)) / (sum(increase(node_cpu_seconds_total[1m])) by (instance,job,hostname)))) * 100 > 90
    for: 1m
    labels:
      severity: high
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeCpu  使用率已达到告???阈值(严重)"
      description: "{{ $labels.hostname }} 告警阈值为90%, 当前值为: {{ $value }}"

  - alert: NodeMem
    expr: (1-((node_memory_Buffers_bytes + node_memory_Cached_bytes + node_memory_MemFree_bytes) / node_memory_MemTotal_bytes {job=~'cn-lcm-prod.*'}))  * 100 > 90
    for: 1m
    labels:
      severity: high
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeMem 使用率已达到告警阈值(严重)"
      description: "{{ $labels.instance }} 告警阈值为90%, 当前值为: {{ $value }}"

  - alert: NodeDiskUsed
    expr: (sum(node_filesystem_size_bytes{job=~'cn-lcm-prod.*'}) by(instance,device,hostname) - sum(node_filesystem_free_bytes{job=~'cn-lcm-prod.*'}) by(instance,device,hostname)) / sum(node_filesystem_size_bytes{job=~'cn-lcm-prod.*'}) by(instance,device,hostname) * 100  > 90
    for: 1m
    labels:
      severity: high
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeDiskUsed 磁盘使用率已达???告警阈值(严重)"
      description: "{{ $labels.instance }} 告警阈值为90%, 当前值为: {{ $value }}"

  - alert: NodeDiskIopsRead
    expr: irate(node_disk_reads_completed_total{job=~"cn-lcm-prod.*"}[1m]) > 2000
    for: 1m
    labels:
      severity: high
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeDiskIopsRead 磁盘读取IOPS已达到告警阈值(严重)"
      description: "{{ $labels.instance }} 告警阈值为2000, 当前值为: {{ $value }}"

  - alert: NodeDiskIopsWrite
    expr: irate(node_disk_writes_completed_total{job=~"cn-lcm-prod.*"}[1m]) > 2000
    for: 1m
    labels:
      severity: high
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeDiskIopsWrite 磁盘写入IOPS已达到告警阈值(严重)"
      description: "{{ $labels.instance }} 告警阈值为2000, 当前值为: {{ $value }}"



  - alert: NodeTcpCurrEstab
    expr: node_netstat_Tcp_CurrEstab{job=~"cn-lcm-prod.*"} > 6000
    for: 1m
    labels:
      severity: high
      project: cn-lcm-prod
    annotations:
      summary:  "CN-LCM-PROD NodeTcpCurrEstab TCP当前连接已达到告警阈值(严重)"
      description: "{{ $labels.instance }} 告警阈值为6000, 当前值为: {{ $value }}"
主prometheus告警模版样例

3.备prometheus配置以及告警模版样例

　　和主配置，一点都不需要更改

 

4.主备alertmanager配置

　　alertmanager主备配置都一样，只是启动的时候命令不相同。

global:
  #该参数定义了当Alertmanager持续多长时间未接收到告警后标记告警状态为resolved（已解决）。该参数的定义可能会影响到告警恢复通知的接收时间，读者可根据自己的实际场景进行定义，其默认值为5分钟
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.qiye.163.com:465'
  smtp_from: 'monitor@em.denachina.com'
  smtp_auth_username: 'monitor@em.denachina.com'
  smtp_auth_password: '3tkvGD8G4giGmAu'
  smtp_require_tls: false

  #wechat_api_url: 'https://qyapi.weixin.qq.com/cgi-bin/'
  #wechat_api_secret: 'CahXkrW-B12Il05HDqB79DIO8yRjLF8jhym0_hgZNUg'
  #wechat_api_corp_id: 'wwd30f74a2a3aca657'


templates:
-  "/opt/alertmanager/templates/*.tmpl"

route:
  receiver: 'default-receiver'
  #这里是被监控的标签key,比如一个alertname标签的信息为一封邮件发送
  group_by: ['alertname','project','severity']
  #两组告警的间隔时间
  group_interval: 1m
  #为了分组发送,在多少秒之内的告警合为一条
  group_wait: 20s
  #如果已经发送了通知，还要等多久才能再次发送
  repeat_interval: 120m
  routes:
  - receiver: cn-lcm-prod
    group_wait: 20s
    match_re:
      project: cn-lcm-prod
  - receiver: cn-gc
    group_wait: 20s
    match_re:
      project: cn-gc


receivers:
- name: 'default-receiver'
  email_configs:
  - to: "zexiong.li@xx.com,zhaohe.ji@xx.com,xin.a.yao@xx.com,lizexiong@huawei.com"
    send_resolved: true
    #html: '{{ template "email.html" . }}'
    #headers: { Subject: " {{ .CommonLabels.alertname }}  {{  .CommonAnnotations.summary }}" }
    headers: { Subject: "{{  .CommonAnnotations.summary }}" }
- name: 'cn-lcm-prod'
  email_configs:
  - to: "{{ template "lizexiongmail" }}"
    send_resolved: true
    headers: { Subject: "{{  .CommonAnnotations.summary }}" }
  webhook_configs:
  - url: "http://10.10.1.16:5000"
    send_resolved: true


- name: 'cn-gc'
  email_configs:
  - to: "{{ template "lizexiongmail" }}"
    send_resolved: true
    headers: { Subject: "  {{ .CommonAnnotations.summary }}" }
  webhook_configs:
  - url: "http://10.10.1.16:5000"
    send_resolved: true
  #wechat_configs:
  #- corp_id: 'wwd30f74a2a3aca657'
    #to_party: '1'
    #send_resolved: true
    #agent_id: '1000002'
    #to_user: '@all'
    #api_secret: 'CahXkrW-B12Il05HDqB79DIO8yRjLF8jhym0_hgZNUg'
    #message: '{{ template "wechat.default.message" . }}'


inhibit_rules:
  #source_match: 匹配当前告警发生后其他告警抑制掉
  - source_match:
      #severity: 指定告警级别
      severity: 'high'
      #target_match:抑制告警
    target_match:
      #severity: 指定抑制告警级别
      severity: 'warning'
      #equal: 只有包含指定标签才可成立规则
    equal: ['alertname',  'instance']
主备alertmanager配置

5.Alertmanager模版示例

　　可以看到alertmanager用了lizexiongmail模版变量，这里看看模版变量怎么写的。

cat email.tmpl 
{{ define "lizexiongmail" }} zexiong.li@dena.com {{ end }}
{{ define "email.html" }}
"5">
    
    {{ range $i, $alert := .Alerts }}
        
    {{ end }}

报警项
        磁盘
        报警阀值
        开始时间
    
{{ index $alert.Labels "alertname" }}
            {{ index $alert.Labels "instance" }}
            {{ index $alert.Labels "value" }}
            {{ $alert.StartsAt }}
        

{{ end }}

　　还有官方自带的模版

cat email.tmpl 
{{ define "lizexiongmail" }} zexiong.li@dena.com {{ end }}
{{ define "email.html" }}
"5">
    
    {{ range $i, $alert := .Alerts }}
        
    {{ end }}

报警项
        磁盘
        报警阀值
        开始时间
    
{{ index $alert.Labels "alertname" }}
            {{ index $alert.Labels "instance" }}
            {{ index $alert.Labels "value" }}
            {{ $alert.StartsAt }}
        

{{ end }}

还有官方自带的模版
cat default.tmpl 
{{ define "__alertmanager" }}Alertmanager{{ end }}
{{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver | urlquery }}{{ end }}

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}
{{ define "__description" }}{{ end }}

{{ define "__text_alert_list" }}{{ range . }}Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}Source: {{ .GeneratorURL }}
{{ end }}{{ end }}


{{ define "slack.default.title" }}{{ template "__subject" . }}{{ end }}
{{ define "slack.default.username" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "slack.default.fallback" }}{{ template "slack.default.title" . }} | {{ template "slack.default.titlelink" . }}{{ end }}
{{ define "slack.default.callbackid" }}{{ end }}
{{ define "slack.default.pretext" }}{{ end }}
{{ define "slack.default.titlelink" }}{{ template "__alertmanagerURL" . }}{{ end }}
{{ define "slack.default.iconemoji" }}{{ end }}
{{ define "slack.default.iconurl" }}{{ end }}
{{ define "slack.default.text" }}{{ end }}
{{ define "slack.default.footer" }}{{ end }}


{{ define "pagerduty.default.description" }}{{ template "__subject" . }}{{ end }}
{{ define "pagerduty.default.client" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "pagerduty.default.clientURL" }}{{ template "__alertmanagerURL" . }}{{ end }}
{{ define "pagerduty.default.instances" }}{{ template "__text_alert_list" . }}{{ end }}


{{ define "opsgenie.default.message" }}{{ template "__subject" . }}{{ end }}
{{ define "opsgenie.default.description" }}{{ .CommonAnnotations.SortedPairs.Values | join " " }}
{{ if gt (len .Alerts.Firing) 0 -}}
Alerts Firing:
{{ template "__text_alert_list" .Alerts.Firing }}
{{- end }}
{{ if gt (len .Alerts.Resolved) 0 -}}
Alerts Resolved:
{{ template "__text_alert_list" .Alerts.Resolved }}
{{- end }}
{{- end }}
{{ define "opsgenie.default.source" }}{{ template "__alertmanagerURL" . }}{{ end }}


{{ define "wechat.default.message" }}{{ template "__subject" . }}
{{ .CommonAnnotations.SortedPairs.Values | join " " }}
{{ if gt (len .Alerts.Firing) 0 -}}
Alerts Firing:
{{ template "__text_alert_list" .Alerts.Firing }}
{{- end }}
{{ if gt (len .Alerts.Resolved) 0 -}}
Alerts Resolved:
{{ template "__text_alert_list" .Alerts.Resolved }}
{{- end }}
AlertmanagerUrl:
{{ template "__alertmanagerURL" . }}
{{- end }}
{{ define "wechat.default.to_user" }}{{ end }}
{{ define "wechat.default.to_party" }}{{ end }}
{{ define "wechat.default.to_tag" }}{{ end }}
{{ define "wechat.default.agent_id" }}{{ end }}



{{ define "victorops.default.state_message" }}{{ .CommonAnnotations.SortedPairs.Values | join " " }}
{{ if gt (len .Alerts.Firing) 0 -}}
Alerts Firing:
{{ template "__text_alert_list" .Alerts.Firing }}
{{- end }}
{{ if gt (len .Alerts.Resolved) 0 -}}
Alerts Resolved:
{{ template "__text_alert_list" .Alerts.Resolved }}
{{- end }}
{{- end }}
{{ define "victorops.default.entity_display_name" }}{{ template "__subject" . }}{{ end }}
{{ define "victorops.default.monitoring_tool" }}{{ template "__alertmanager" . }}{{ end }}

{{ define "email.default.subject" }}{{ template "__subject" . }}{{ end }}
{{ define "email.default.html" }}
"-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

"http://www.w3.org/1999/xhtml" xmlns="http://www.w3.org/1999/xhtml" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
"viewport" content="width=device-width" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
"Content-Type" content="text/html; charset=UTF-8" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
"</span><span style="color: rgba(128, 0, 0, 1)">font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;</span><span style="color: rgba(128, 0, 0, 1)">"</span>>{{ template <span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">__subject</span><span style="color: rgba(128, 0, 0, 1)">"</span> . }}



"" itemtype="http://schema.org/EmailMessage" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; -webkit-font-smoothing: antialiased; -webkit-text-size-adjust: none; height: 100%; line-height: 1.6em; width: 100% !important; background-color: #f6f6f6; margin: 0; padding: 0;" bgcolor="#f6f6f6">

"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; width: 100%; background-color: #f6f6f6; margin: 0;" bgcolor="#f6f6f6">
  "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
    
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0;" valign="top">
    "600" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; display: block !important; max-width: 600px !important; clear: both !important; width: 100% !important; margin: 0 auto; padding: 0;" valign="top">
      
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; max-width: 600px; display: block; margin: 0 auto; padding: 0;">
        "100%" cellpadding="0" cellspacing="0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; border-radius: 3px; background-color: #fff; margin: 0; border: 1px solid #e9e9e9;" bgcolor="#fff">
          "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
            "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
            
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 16px; vertical-align: top; color: #fff; font-weight: 500; text-align: center; border-radius: 3px 3px 0 0; background-color: #E6522C; margin: 0; padding: 20px;" align="center" bgcolor="#E6522C" valign="top">
              {{ .Alerts | len }} alert{{ if gt (len .Alerts) 1 }}s{{ end }} for {{ range .GroupLabels.SortedPairs }}
                {{ .Name }}={{ .Value }}
              {{ end }}
            
          

          
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 10px;" valign="top">
              "100%" cellpadding="0" cellspacing="0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  
                {{ if gt (len .Alerts.Firing) 0 }}
                "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  
                {{ end }}
                {{ range .Alerts.Firing }}
                "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  
                {{ end }}

                {{ if gt (len .Alerts.Resolved) 0 }}
                  {{ if gt (len .Alerts.Firing) 0 }}
                "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  
                  {{ end }}
                "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  
                {{ end }}
                {{ range .Alerts.Resolved }}
                "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  
                {{ end }}
              
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                    "{{ template "__alertmanagerURL" . }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; color: #FFF; text-decoration: none; line-height: 2em; font-weight: bold; text-align: center; cursor: pointer; display: inline-block; border-radius: 5px; text-transform: capitalize; background-color: #348eda; margin: 0; border-color: #348eda; border-style: solid; border-width: 10px 20px;">View in {{ template "__alertmanager" . }}
                  
                
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                    "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">[{{ .Alerts.Firing | len }}] Firing
                  
                
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                    "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Labels
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                    {{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                    {{ if gt (len .Annotations) 0 }}"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Annotations
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                    {{ range .Annotations.SortedPairs }}{{ .Name }} = {{ .Value }}
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                    "{{ .GeneratorURL }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; color: #348eda; text-decoration: underline; margin: 0;">Source
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                  		
                
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                    
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                    
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                    
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                  		
                
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                    "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">[{{ .Alerts.Resolved | len }}] Resolved
                  
                
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                    "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Labels
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                    {{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                    {{ if gt (len .Annotations) 0 }}"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Annotations
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                    {{ range .Annotations.SortedPairs }}{{ .Name }} = {{ .Value }}
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                    "{{ .GeneratorURL }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; color: #348eda; text-decoration: underline; margin: 0;">Source
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                  		
                

            		
          

        


        
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; width: 100%; clear: both; color: #999; margin: 0; padding: 20px;">
          "100%" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
            "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
              
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 12px; vertical-align: top; text-align: center; color: #999; margin: 0; padding: 0 0 20px;" align="center" valign="top">"{{ .ExternalURL }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 12px; color: #999; text-decoration: underline; margin: 0;">Sent by {{ template "__alertmanager" . }}
            

          

        

    		
    "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0;" valign="top">
  







{{ end }}

{{ define "pushover.default.title" }}{{ template "__subject" . }}{{ end }}
{{ define "pushover.default.message" }}{{ .CommonAnnotations.SortedPairs.Values | join " " }}
{{ if gt (len .Alerts.Firing) 0 }}
Alerts Firing:
{{ template "__text_alert_list" .Alerts.Firing }}
{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}
Alerts Resolved:
{{ template "__text_alert_list" .Alerts.Resolved }}
{{ end }}
{{ end }}
{{ define "pushover.default.url" }}{{ template "__alertmanagerURL" . }}{{ end }}

{{ define "sns.default.subject" }}{{ template "__subject" . }}{{ end }}
{{ define "sns.default.message" }}{{ .CommonAnnotations.SortedPairs.Values | join " " }}
{{ if gt (len .Alerts.Firing) 0 }}
Alerts Firing:
{{ template "__text_alert_list" .Alerts.Firing }}
{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}
Alerts Resolved:
{{ template "__text_alert_list" .Alerts.Resolved }}
{{ end }}
{{ end }}
[root@prometheus01 templates]# cat default.tmpl ^C
[root@prometheus01 templates]# vim default.tmpl 
[root@prometheus01 templates]# cat default.tmpl | wc -l
231
[root@prometheus01 templates]# vim default.tmpl ^C
[root@prometheus01 templates]# cat default.tmpl 
{{ define "__alertmanager" }}Alertmanager{{ end }}
{{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver | urlquery }}{{ end }}

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}
{{ define "__description" }}{{ end }}

{{ define "__text_alert_list" }}{{ range . }}Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}Source: {{ .GeneratorURL }}
{{ end }}{{ end }}


{{ define "slack.default.title" }}{{ template "__subject" . }}{{ end }}
{{ define "slack.default.username" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "slack.default.fallback" }}{{ template "slack.default.title" . }} | {{ template "slack.default.titlelink" . }}{{ end }}
{{ define "slack.default.callbackid" }}{{ end }}
{{ define "slack.default.pretext" }}{{ end }}
{{ define "slack.default.titlelink" }}{{ template "__alertmanagerURL" . }}{{ end }}
{{ define "slack.default.iconemoji" }}{{ end }}
{{ define "slack.default.iconurl" }}{{ end }}
{{ define "slack.default.text" }}{{ end }}
{{ define "slack.default.footer" }}{{ end }}


{{ define "pagerduty.default.description" }}{{ template "__subject" . }}{{ end }}
{{ define "pagerduty.default.client" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "pagerduty.default.clientURL" }}{{ template "__alertmanagerURL" . }}{{ end }}
{{ define "pagerduty.default.instances" }}{{ template "__text_alert_list" . }}{{ end }}


{{ define "opsgenie.default.message" }}{{ template "__subject" . }}{{ end }}
{{ define "opsgenie.default.description" }}{{ .CommonAnnotations.SortedPairs.Values | join " " }}
{{ if gt (len .Alerts.Firing) 0 -}}
Alerts Firing:
{{ template "__text_alert_list" .Alerts.Firing }}
{{- end }}
{{ if gt (len .Alerts.Resolved) 0 -}}
Alerts Resolved:
{{ template "__text_alert_list" .Alerts.Resolved }}
{{- end }}
{{- end }}
{{ define "opsgenie.default.source" }}{{ template "__alertmanagerURL" . }}{{ end }}


{{ define "wechat.default.message" }}{{ template "__subject" . }}
{{ .CommonAnnotations.SortedPairs.Values | join " " }}
{{ if gt (len .Alerts.Firing) 0 -}}
Alerts Firing:
{{ template "__text_alert_list" .Alerts.Firing }}
{{- end }}
{{ if gt (len .Alerts.Resolved) 0 -}}
Alerts Resolved:
{{ template "__text_alert_list" .Alerts.Resolved }}
{{- end }}
AlertmanagerUrl:
{{ template "__alertmanagerURL" . }}
{{- end }}
{{ define "wechat.default.to_user" }}{{ end }}
{{ define "wechat.default.to_party" }}{{ end }}
{{ define "wechat.default.to_tag" }}{{ end }}
{{ define "wechat.default.agent_id" }}{{ end }}



{{ define "victorops.default.state_message" }}{{ .CommonAnnotations.SortedPairs.Values | join " " }}
{{ if gt (len .Alerts.Firing) 0 -}}
Alerts Firing:
{{ template "__text_alert_list" .Alerts.Firing }}
{{- end }}
{{ if gt (len .Alerts.Resolved) 0 -}}
Alerts Resolved:
{{ template "__text_alert_list" .Alerts.Resolved }}
{{- end }}
{{- end }}
{{ define "victorops.default.entity_display_name" }}{{ template "__subject" . }}{{ end }}
{{ define "victorops.default.monitoring_tool" }}{{ template "__alertmanager" . }}{{ end }}

{{ define "email.default.subject" }}{{ template "__subject" . }}{{ end }}
{{ define "email.default.html" }}
"-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

"http://www.w3.org/1999/xhtml" xmlns="http://www.w3.org/1999/xhtml" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
"viewport" content="width=device-width" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
"Content-Type" content="text/html; charset=UTF-8" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
"</span><span style="color: rgba(128, 0, 0, 1)">font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;</span><span style="color: rgba(128, 0, 0, 1)">"</span>>{{ template <span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">__subject</span><span style="color: rgba(128, 0, 0, 1)">"</span> . }}



"" itemtype="http://schema.org/EmailMessage" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; -webkit-font-smoothing: antialiased; -webkit-text-size-adjust: none; height: 100%; line-height: 1.6em; width: 100% !important; background-color: #f6f6f6; margin: 0; padding: 0;" bgcolor="#f6f6f6">

"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; width: 100%; background-color: #f6f6f6; margin: 0;" bgcolor="#f6f6f6">
  "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
    
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0;" valign="top">
    "600" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; display: block !important; max-width: 600px !important; clear: both !important; width: 100% !important; margin: 0 auto; padding: 0;" valign="top">
      
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; max-width: 600px; display: block; margin: 0 auto; padding: 0;">
        "100%" cellpadding="0" cellspacing="0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; border-radius: 3px; background-color: #fff; margin: 0; border: 1px solid #e9e9e9;" bgcolor="#fff">
          "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
            "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
            
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 16px; vertical-align: top; color: #fff; font-weight: 500; text-align: center; border-radius: 3px 3px 0 0; background-color: #E6522C; margin: 0; padding: 20px;" align="center" bgcolor="#E6522C" valign="top">
              {{ .Alerts | len }} alert{{ if gt (len .Alerts) 1 }}s{{ end }} for {{ range .GroupLabels.SortedPairs }}
                {{ .Name }}={{ .Value }}
              {{ end }}
            
          

          
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 10px;" valign="top">
              "100%" cellpadding="0" cellspacing="0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  
                {{ if gt (len .Alerts.Firing) 0 }}
                "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  
                {{ end }}
                {{ range .Alerts.Firing }}
                "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  
                {{ end }}

                {{ if gt (len .Alerts.Resolved) 0 }}
                  {{ if gt (len .Alerts.Firing) 0 }}
                "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  
                  {{ end }}
                "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  
                {{ end }}
                {{ range .Alerts.Resolved }}
                "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
                  
                {{ end }}
              
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                    "{{ template "__alertmanagerURL" . }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; color: #FFF; text-decoration: none; line-height: 2em; font-weight: bold; text-align: center; cursor: pointer; display: inline-block; border-radius: 5px; text-transform: capitalize; background-color: #348eda; margin: 0; border-color: #348eda; border-style: solid; border-width: 10px 20px;">View in {{ template "__alertmanager" . }}
                  
                
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                    "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">[{{ .Alerts.Firing | len }}] Firing
                  
                
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                    "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Labels
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                    {{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                    {{ if gt (len .Annotations) 0 }}"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Annotations
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                    {{ range .Annotations.SortedPairs }}{{ .Name }} = {{ .Value }}
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                    "{{ .GeneratorURL }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; color: #348eda; text-decoration: underline; margin: 0;">Source
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                  		
                
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                    
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                    
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                    
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                  		
                
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                    "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">[{{ .Alerts.Resolved | len }}] Resolved
                  
                
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
                    "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Labels
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                    {{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                    {{ if gt (len .Annotations) 0 }}"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Annotations
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                    {{ range .Annotations.SortedPairs }}{{ .Name }} = {{ .Value }}
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}
                    "{{ .GeneratorURL }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; color: #348eda; text-decoration: underline; margin: 0;">Source
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
                  		
                

            		
          

        


        
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; width: 100%; clear: both; color: #999; margin: 0; padding: 20px;">
          "100%" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
            "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
              
"font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 12px; vertical-align: top; text-align: center; color: #999; margin: 0; padding: 0 0 20px;" align="center" valign="top">"{{ .ExternalURL }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 12px; color: #999; text-decoration: underline; margin: 0;">Sent by {{ template "__alertmanager" . }}
            

          

        

    		
    "font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0;" valign="top">
  







{{ end }}

{{ define "pushover.default.title" }}{{ template "__subject" . }}{{ end }}
{{ define "pushover.default.message" }}{{ .CommonAnnotations.SortedPairs.Values | join " " }}
{{ if gt (len .Alerts.Firing) 0 }}
Alerts Firing:
{{ template "__text_alert_list" .Alerts.Firing }}
{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}
Alerts Resolved:
{{ template "__text_alert_list" .Alerts.Resolved }}
{{ end }}
{{ end }}
{{ define "pushover.default.url" }}{{ template "__alertmanagerURL" . }}{{ end }}

{{ define "sns.default.subject" }}{{ template "__subject" . }}{{ end }}
{{ define "sns.default.message" }}{{ .CommonAnnotations.SortedPairs.Values | join " " }}
{{ if gt (len .Alerts.Firing) 0 }}
Alerts Firing:
{{ template "__text_alert_list" .Alerts.Firing }}
{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}
Alerts Resolved:
{{ template "__text_alert_list" .Alerts.Resolved }}
{{ end }}
{{ end }}
官方自带模版
prometheus

相关

Prometheus安装

Prometheus之node exporter使用帮助

Prometheus之Alertmanager介绍

Prometheus之PromQL进阶

Prometheus之部署blackbox_exporter

prometheus 生产环境部署

Prometheus（三）：Prometheus监控交换机（snmp）

Prometheus部署（一）

使用thanos管理Prometheus持久化数据

使用thanos管理Prometheus持久化数据

Prometheus+alertmanage配置邮件报警

prometheus-operator 安装部署

标签