The following are all different examples of alerts and corresponding Alertmanager configuration file setups (alertmanager.yml). Each use the system.
In this example we've customised our Slack notification to send a URL to our organisation's wiki on how to deal with the particular alert that's been sent.
global:
# Also possible to place this URL in a file.
# Ex: `slack_api_url_file: '/etc/alertmanager/slack_url'`
slack_api_url: '<slack_webhook_url>'
route:
receiver: 'slack-notifications'
group_by: [alertname, datacenter, app]
receivers:
- name: 'slack-notifications'
slack_configs:
- channel: '#alerts'
text: 'https://internal.myorg.net/wiki/alerts/{{ .GroupLabels.app }}/{{ .GroupLabels.alertname }}'
In this example we again customize the text sent to our Slack receiver accessing the summary
and description
stored in the CommonAnnotations
of the data sent by the Alertmanager.
Alert
groups:
- name: Instances
rules:
- alert: InstanceDown
expr: up == 0
for: 5m
labels:
severity: page
# Prometheus templates apply here in the annotation and label fields of the alert.
annotations:
description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
summary: 'Instance {{ $labels.instance }} down'
Receiver
- name: 'team-x'
slack_configs:
- channel: '#alerts'
# Alertmanager templates apply here.
text: "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}"
Finally, assuming the same alert as the previous example, we customize our receiver to range over all of the alerts received from the Alertmanager, printing their respective annotation summaries and descriptions on new lines.
Receiver
- name: 'default-receiver'
slack_configs:
- channel: '#alerts'
title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"
Going back to our first example, we can also provide a file containing named templates which are then loaded by Alertmanager in order to avoid complex templates that span many lines.
Create a file under /alertmanager/template/myorg.tmpl
and create a template in it named "slack.myorg.text":
{{ define "slack.myorg.text" }}https://internal.myorg.net/wiki/alerts/{{ .GroupLabels.app }}/{{ .GroupLabels.alertname }}{{ end}}
The configuration now loads the template with the given name for the "text" field and we provide a path to our custom template file:
global:
slack_api_url: '<slack_webhook_url>'
route:
receiver: 'slack-notifications'
group_by: [alertname, datacenter, app]
receivers:
- name: 'slack-notifications'
slack_configs:
- channel: '#alerts'
text: '{{ template "slack.myorg.text" . }}'
templates:
- '/etc/alertmanager/templates/myorg.tmpl'
This example is explained in further detail in this blogpost.
This documentation is . Please help improve it by filing issues or pull requests.