MySQL and MongoDB Alerting with Percona Monitoring and Management and Grafana

Percona is the only company that delivers enterprise-class support, consulting, managed services and software for MySQL®, MariaDB®, MongoDB®
Published:
This post contains step-by-step instructions for setting up alerting in Percona Monitoring and Management (PMM) using Grafana.

As a Solutions Engineer at Percona, I get to speak with a lot of the early adopters of Percona’s new Percona Monitoring and Management software. One of the frequent feature requests from them was the need for alerts at certain MySQL and MongoDB thresholds. Percona’s philosophy on alerting is that we don’t want to reinvent the wheel. We integrated Consul, Grafana and Prometheus into PMM because they are fantastic, free and open-source products. PMM packages them together in an easy to deploy and maintain format.

With the release of Grafana 4.0 came Alerting! The Grafana team excitedly shared this feature with its users (including Percona). Percona quickly followed suit by releasing a new PMM 1.0.7 package that contained Grafana 4.0.


Configuring Grafana for Alerting

The instructions below assume that you have a running instance of PMM. Percona’s website contains detailed installation instructions if you don’t.


There are several ways to receive alerts with Grafana – email, Slack, Webhook and PagerDuty. For this blog post, I demonstrate alerting through email.Please note that for this to work correctly, SMTP settings must be configured in the Grafana config file, which is located inside your docker container. To enter the Docker container, use the ID of the running container found with the command “docker ps”. Once you have the container ID, enter the Docker shell by using the commanddocker exec -it CONTAINER_ID /bin/bash. From this terminal, you can access the /etc/grafana/grafana.ini configuration file. Documentation for the configuration file can be found on Grafana’s website.


Below is a basic, default configuration setting for the SMTP/email section for testing purposes that works for Gmail:

SMTP Config

[smtp]
enabled = true
host = smtp.gmail.com:465
user = your.gmail.account@gmail.com
password = yourpassword
cert_file =
key_file =
skip_verify = false
from_address = admin@grafana.localhost


There are also instructions for setting up other alerting methods on Grafana’s website. Once you have made changes to the config file, you can restart pmm-server to pick up the changes with adocker restart pmm-server command.


Once you restart the docker container, access the Alert Notification panel by selecting the Grafana icon dropdown in the upper left corner of any Grafana dashboard:

Alerting Tab

To set up alerting, you need to create an alert notification on this page. For this test, I am only entering one email address. You can enter more than one by separating emails with commas.

Testing the Configuration

You can test your server’s configuration by clicking the “Send Test” button.

Alert Notification Page

You’ll know that you configured this correctly when you receive the following email:

Test Email Confirmation

Enabling Alerting for a Variable

Now that you’ve updated your Grafana config file and tested to ensure your SMTP settings are correct, it’s time to set up an alert for a specific variable. To do that, go the dashboard for which you’re interested in setting up alerting: 

  1. Click the title [1]
  2. Select “Edit” [2]
  3. Choose the Alert tab [3]
  4. Select the “Create Alert” button. In the Alert Config section, you can modify the frequency at which the alert checks for a trigger [4] and the conditions to trigger the alert [5].

Configuring Alert

Currently, a known limitation exists with Grafana. Alerting doesn’t support template variables yet. However, there is a feature request on Github to add this functionality in the future. Because of this, template variables must be removed from the Metrics tab [6] of the dashboard you are editing. 

Grafana sends an alert email whenever the configured threshold gets passed:

Alert Email

Once an alert clears, you will get an all clear email.

All Clear Email

Summary

Proactive monitoring is vital to a highly-available environment. Hopefully, this blog post gives you the confidence to move forward with integrating alerting into your PMM platform. I also wrote this blog post to highlight the fact that our PMM team continues to add functionality requested by users with regards to monitoring and maintaining their environment. If you’d like to ask questions or give feedback, the PMM forums on Percona’s website are a good place to go.

0
1,662 Views
Percona is the only company that delivers enterprise-class support, consulting, managed services and software for MySQL®, MariaDB®, MongoDB®

Comments (0)

Have a question about something in this article? You can receive help directly from the article author. Sign up for a free trial to get started.