Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 318
  • Last Modified:

Large number of Nagios alerts after a host comes up from being down.

Hi,

We are running Nagios 4.0.7 and whenever a host goes down (ping results time out) we get an alert that the host is down and nothing else, which is great. However, when the host comes back up, all of the other service checks immediately time out and start sending a massive amount of alerts about each service. Then, as soon as the services come back up, we get another massive amount of alerts stating that the services are recovered.

Is there a way to delay service alerts after a host goes down and comes back up? For instance, a host goes down, we get an alert regarding the down'd host. Host comes up, and we get an alert that the host is up. If the services aren't okay after the host has been recoered for, say, 5 minutes THEN we start to get service alerts. Is this possible?

Thank you
0
OAC Technology
Asked:
OAC Technology
1 Solution
 
Seth SimmonsSr. Systems AdministratorCommented:
you could increase your service check frequency or the number of soft alerts before triggering a hard alert
if either of those are too small, the hard alert would be triggered faster causing that to happen
0
 
Sanga CollinsSystems AdminCommented:
You can also use host a service dependencies so when a host goes down, any hosts or services that are dependencies will suppress their alerts. When the host comes back up the dependencies will follow the same process. If the services goes down on its own it will alert you as configured.
0
 
OAC TechnologyProfessional NerdsAuthor Commented:
Seth, this increase would delay service alerts across the board and not just if a host went down and came back up, correct? My hope was that there was a way to tell service alerts to hold off for a while only if the host went down and came back up. Otherwise we'll be waiting 5 minutes to be alerted if a service just decides to die

Sanga, That's how we have it set now. If a host goes down, the services don't report that they are down, but the problem is when the host comes back up, all of the services are still marked as down so we get a flood of alerts


Thanks for the help
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
Seth SimmonsSr. Systems AdministratorCommented:
depends how you have things configured
you might have templates that all hosts follow or some might be customized
is it that important that you need to be notified that soon?  do you need service check intervals that short?

as far as dependencies go, services associated with a host are automatically dependent of a host
using dependencies is more for something in between to prevent false positives
for example, a remote site goes down, a system there could be reported down when it isn't.  having the gateway/router as dependency will make that system 'unknown' because the parent is down and not the system itself
0
 
Jan SpringerCommented:
What Sanga said.  Do you have dependencies configured?
0
 
OAC TechnologyProfessional NerdsAuthor Commented:
How do I check to make sure I have dependencies setup/configured?
0
 
Seth SimmonsSr. Systems AdministratorCommented:
could you post your configuration file(s) to review?
0
 
OAC TechnologyProfessional NerdsAuthor Commented:
I've posted the configuration file for one of the servers I am monitoring (with details scrubbed). Are there any other files you need me to upload?
0
 
Seth SimmonsSr. Systems AdministratorCommented:
there is nothing attached
0
 
OAC TechnologyProfessional NerdsAuthor Commented:
Not sure why the attachment didn't show up, but I was able to find a solution that works for us. We are using NAN (https://www.monitoringexchange.org/inventory/Utilities/AddOn-Projects/Notifications/NAN---Nagios-Notification-Daemon) to consolidate all of our alerts and it has been working great. It takes our flood of 200 messages within a 3 minute period and consolidates them into 1 for alerts and 1 for recoveries.

Thank you
0
 
OAC TechnologyProfessional NerdsAuthor Commented:
Found solution
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Tackle projects and never again get stuck behind a technical roadblock.
Join Now