Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.
As a Solutions Engineer at Percona, I get to speak with a lot of the early adopters of Percona’s new Percona Monitoring and Management software. One of the frequent feature requests from them was the need for alerts at certain MySQL and MongoDB thresholds. Percona’s philosophy on alerting is that we don’t want to reinvent the wheel. We integrated Consul, Grafana and Prometheus into PMM because they are fantastic, free and open-source products. PMM packages them together in an easy to deploy and maintain format.
With the release of Grafana 4.0 came Alerting! The Grafana team excitedly shared this feature with its users (including Percona). Percona quickly followed suit by releasing a new PMM 1.0.7 package that contained Grafana 4.0.
The instructions below assume that you have a running instance of PMM. Percona’s website contains detailed installation instructions if you don’t.
There are several ways to receive alerts with Grafana – email, Slack, Webhook and PagerDuty. For this blog post, I demonstrate alerting through email.Please note that for this to work correctly, SMTP settings must be configured in the Grafana config file, which is located inside your docker container. To enter the Docker container, use the ID of the running container found with the command “docker ps”. Once you have the container ID, enter the Docker shell by using the commanddocker exec -it CONTAINER_ID /bin/bash. From this terminal, you can access the /etc/grafana/grafana.ini configuration file. Documentation for the configuration file can be found on Grafana’s website.
Below is a basic, default configuration setting for the SMTP/email section for testing purposes that works for Gmail:
[smtp] enabled = true host = smtp.gmail.com:465 user = email@example.com password = yourpassword cert_file = key_file = skip_verify = false from_address = firstname.lastname@example.org
There are also instructions for setting up other alerting methods on Grafana’s website. Once you have made changes to the config file, you can restart pmm-server to pick up the changes with adocker restart pmm-server command.
Once you restart the docker container, access the Alert Notification panel by selecting the Grafana icon dropdown in the upper left corner of any Grafana dashboard:
To set up alerting, you need to create an alert notification on this page. For this test, I am only entering one email address. You can enter more than one by separating emails with commas.
You can test your server’s configuration by clicking the “Send Test” button.
You’ll know that you configured this correctly when you receive the following email:
Now that you’ve updated your Grafana config file and tested to ensure your SMTP settings are correct, it’s time to set up an alert for a specific variable. To do that, go the dashboard for which you’re interested in setting up alerting:
Currently, a known limitation exists with Grafana. Alerting doesn’t support template variables yet. However, there is a feature request on Github to add this functionality in the future. Because of this, template variables must be removed from the Metrics tab  of the dashboard you are editing.
Grafana sends an alert email whenever the configured threshold gets passed:
Once an alert clears, you will get an all clear email.
Proactive monitoring is vital to a highly-available environment. Hopefully, this blog post gives you the confidence to move forward with integrating alerting into your PMM platform. I also wrote this blog post to highlight the fact that our PMM team continues to add functionality requested by users with regards to monitoring and maintaining their environment. If you’d like to ask questions or give feedback, the PMM forums on Percona’s website are a good place to go.