Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Large number of Nagios alerts after a host comes up from being down.

Posted on 2014-11-07
11
Medium Priority
?
312 Views
Last Modified: 2014-11-22
Hi,

We are running Nagios 4.0.7 and whenever a host goes down (ping results time out) we get an alert that the host is down and nothing else, which is great. However, when the host comes back up, all of the other service checks immediately time out and start sending a massive amount of alerts about each service. Then, as soon as the services come back up, we get another massive amount of alerts stating that the services are recovered.

Is there a way to delay service alerts after a host goes down and comes back up? For instance, a host goes down, we get an alert regarding the down'd host. Host comes up, and we get an alert that the host is up. If the services aren't okay after the host has been recoered for, say, 5 minutes THEN we start to get service alerts. Is this possible?

Thank you
0
Comment
Question by:OAC Technology
11 Comments
 
LVL 36

Expert Comment

by:Seth Simmons
ID: 40429095
you could increase your service check frequency or the number of soft alerts before triggering a hard alert
if either of those are too small, the hard alert would be triggered faster causing that to happen
0
 
LVL 18

Expert Comment

by:Sanga Collins
ID: 40429116
You can also use host a service dependencies so when a host goes down, any hosts or services that are dependencies will suppress their alerts. When the host comes back up the dependencies will follow the same process. If the services goes down on its own it will alert you as configured.
0
 
LVL 2

Author Comment

by:OAC Technology
ID: 40429163
Seth, this increase would delay service alerts across the board and not just if a host went down and came back up, correct? My hope was that there was a way to tell service alerts to hold off for a while only if the host went down and came back up. Otherwise we'll be waiting 5 minutes to be alerted if a service just decides to die

Sanga, That's how we have it set now. If a host goes down, the services don't report that they are down, but the problem is when the host comes back up, all of the services are still marked as down so we get a flood of alerts


Thanks for the help
0
Automating Your MSP Business

The road to profitability.
Delivering superior services is key to ensuring customer satisfaction and the consequent long-term relationships that enable MSPs to lock in predictable, recurring revenue. What's the best way to deliver superior service? One word: automation.

 
LVL 36

Expert Comment

by:Seth Simmons
ID: 40429170
depends how you have things configured
you might have templates that all hosts follow or some might be customized
is it that important that you need to be notified that soon?  do you need service check intervals that short?

as far as dependencies go, services associated with a host are automatically dependent of a host
using dependencies is more for something in between to prevent false positives
for example, a remote site goes down, a system there could be reported down when it isn't.  having the gateway/router as dependency will make that system 'unknown' because the parent is down and not the system itself
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 40429171
What Sanga said.  Do you have dependencies configured?
0
 
LVL 2

Author Comment

by:OAC Technology
ID: 40429286
How do I check to make sure I have dependencies setup/configured?
0
 
LVL 36

Expert Comment

by:Seth Simmons
ID: 40429294
could you post your configuration file(s) to review?
0
 
LVL 2

Author Comment

by:OAC Technology
ID: 40429375
I've posted the configuration file for one of the servers I am monitoring (with details scrubbed). Are there any other files you need me to upload?
0
 
LVL 36

Expert Comment

by:Seth Simmons
ID: 40439876
there is nothing attached
0
 
LVL 2

Accepted Solution

by:
OAC Technology earned 0 total points
ID: 40447997
Not sure why the attachment didn't show up, but I was able to find a solution that works for us. We are using NAN (https://www.monitoringexchange.org/inventory/Utilities/AddOn-Projects/Notifications/NAN---Nagios-Notification-Daemon) to consolidate all of our alerts and it has been working great. It takes our flood of 200 messages within a 3 minute period and consolidates them into 1 for alerts and 1 for recoveries.

Thank you
0
 
LVL 2

Author Closing Comment

by:OAC Technology
ID: 40459161
Found solution
0

Featured Post

WatchGuard Case Study: Museum of Flight

“With limited money and limited staffing, we didn’t have a lot of choices in terms of what we could do to bring efficiency. WatchGuard played a central part in changing that.” To provide strong, secure Wi-Fi access within the museum, Hunter chose to deploy WatchGuard’s AP120 APs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Quality of Service (QoS) options are nearly endless when it comes to networks today. This article is merely one example of how it can be handled in a hub-n-spoke design using a 3-tier configuration.
In this article, we’ll look at how to deploy ProxySQL.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
There's a multitude of different network monitoring solutions out there, and you're probably wondering what makes NetCrunch so special. It's completely agentless, but does let you create an agent, if you desire. It offers powerful scalability …
Suggested Courses
Course of the Month12 days, 9 hours left to enroll

972 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question