Prometheus is an open-source system monitoring and alerting toolkit. It was originally developed at SoundCloud. Prometheus scraps metrices from the assigned jobs. It stores all scrapped samples locally and runs rules over this data to generate alerts. In this blog post, I will describe my own experience of setting up email alerts in Prometheus.
Define Rules and include this to Prometheus
Open prometheus config file located at /etc/prometheus/prometheus.yml
Add rule file location to it, as defined below.
rule_files: - 'prometheus.rules.yml'
If you are unaware about prometheus config file then please check prometheus demo config here.
Check Rules using promtool
After adding rules to prometheus.yml file, use promtool to check rules.
For promotheus config file
promtool check rules prometheus.yml
For prometheus rules file
promtool check rules prometheus.rules.yml
Install Alert Manager and configure it as a service
The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.
There are several ways for installing Alert Manager, though installing from precompiled binaries are recommended way.
Step -1: Get precompiled binaries from download section of prometheus.io
Step -2: untar this file using tar -xvf command
tar -xvf alertmanager-0.15.2.linux-amd64.tar.gz
Step-3: Move alertmanager binaries to /usr/local/bin/ folder
mv alertmanager /usr/local/bin
Step-4: Add a config file to /etc/alertmanager/config.yml ( If you are unable to find any folder /etc/alertmanager create it and then create config.yml file )
Example Alertmanager config
global: # The smarthost and SMTP sender used for mail notifications. smtp_smarthost: 'localhost:25' smtp_from: '[email protected]' smtp_auth_username: 'alertmanager' smtp_auth_password: 'password' # The auth token for Hipchat. hipchat_auth_token: '1234556789' # Alternative host for Hipchat. hipchat_api_url: 'https://hipchat.foobar.org/' # The directory from which notification templates are read. templates: - '/etc/alertmanager/template/*.tmpl' # The root route on which each incoming alert enters. route: # The labels by which incoming alerts are grouped together. For example, # multiple alerts coming in for cluster=A and alertname=LatencyHigh would # be batched into a single group. # # To aggregate by all possible labels use '...' as the sole label name. # This effectively disables aggregation entirely, passing through all # alerts as-is. This is unlikely to be what you want, unless you have # a very low alert volume or your upstream notification system performs # its own grouping. Example: group_by: [...] group_by: ['alertname', 'cluster', 'service'] # When a new group of alerts is created by an incoming alert, wait at # least 'group_wait' to send the initial notification. # This way ensures that you get multiple alerts for the same group that start # firing shortly after another are batched together on the first # notification. group_wait: 30s # When the first notification was sent, wait 'group_interval' to send a batch # of new alerts that started firing for that group. group_interval: 5m # If an alert has successfully been sent, wait 'repeat_interval' to # resend them. repeat_interval: 3h # A default receiver receiver: team-X-mails # All the above attributes are inherited by all child routes and can # overwritten on each. # The child route trees. routes: # This routes performs a regular expression match on alert labels to # catch alerts that are related to a list of services. - match_re: service: ^(foo1|foo2|baz)$ receiver: team-X-mails # The service has a sub-route for critical alerts, any alerts # that do not match, i.e. severity != critical, fall-back to the # parent node and are sent to 'team-X-mails' routes: - match: severity: critical receiver: team-X-pager - match: service: files receiver: team-Y-mails routes: - match: severity: critical receiver: team-Y-pager # This route handles all alerts coming from a database service. If there's # no team to handle it, it defaults to the DB team. - match: service: database receiver: team-DB-pager # Also group alerts by affected database. group_by: [alertname, cluster, database] routes: - match: owner: team-X receiver: team-X-pager continue: true - match: owner: team-Y receiver: team-Y-pager # Inhibition rules allow to mute a set of alerts given that another alert is # firing. # We use this to mute any warning-level notifications if the same alert is # already critical. inhibit_rules: - source_match: severity: 'critical' target_match: severity: 'warning' # Apply inhibition if the alertname is the same. equal: ['alertname', 'cluster', 'service'] receivers: - name: 'team-X-mails' email_configs: - to: '[email protected]' - name: 'team-X-pager' email_configs: - to: '[email protected]' pagerduty_configs: - service_key: <team-X-key> - name: 'team-Y-mails' email_configs: - to: '[email protected]' - name: 'team-Y-pager' pagerduty_configs: - service_key: <team-Y-key> - name: 'team-DB-pager' pagerduty_configs: - service_key: <team-DB-key> - name: 'team-X-hipchat' hipchat_configs: - auth_token: <auth_token> room_id: 85 message_format: html notify: true
Define Alert Manager as a service
Go to /etc/systemd/systme and create a file called alertmanager.service
add following code to alertmanager.service
[Unit] Description=Alert Manager for Prometheus Wants=network-online.target After=network-online.target [Service] User=root Group=root Type=simple ExecStart=/usr/local/bin/alertmanager --config.file /etc/alertmanager/config/config.yml [Install] WantedBy=multi-user.target
Reload Service daemon for new changes to take effect.
Start Alert Manager service
sudo service alertmanager start
Sudo service prometheus restart
Check Status of both the services i.e, Prometheus and Alert Manager using command
sudo service alertmanager status
sudo service prometheus status
If these services are working fine and there is no issue. Now all you have to verify your alerts.
To verify alerts please visit to prometehus installation url like prometheus.exampledomain.com/alerts
and you should see loaded alerts like this.
Congratulations! You have successfully installed email alerts at your Prometheus server.