[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

DNS failover not working as expected query

Posted on 2007-07-31
8
Medium Priority
?
1,118 Views
Last Modified: 2013-12-08
Hi all,

Thanks in advance. I have been working on a really strange issue today and last night. Here's the run down. Basically we have two HPUX 11.i DNS servers lets call them 192.168.1.03 and 192.168.1.04. Each of our UNIX servers had the resolv.conf file configured as: -
search domain.local
namesserver 192.168.1.04
nameserver 192.168.1.03

We then had a major hardware failure for 192.168.1.04 and were not able to bring it back online. We were then seeing some DNS delays when trying to resolve hostnames but it was working to the secondary name server 192.168.1.03. In order to fix the delay we edited the resolv.conf file and changed the order of the name servers to be: -
search domain.local
namesserver 192.168.1.03
nameserver 192.168.1.04

This fixed the delay when doing nslookups but we still had issues with some applications including SAP. Our application servers were giving unknown host errors when trying to communicate with each other even though ping and nslookups from the OS were working without any problem.

We then commented out the 192.168.1.04 box from one of the server with the issue. This still did not help. Finally the only thing we could do to work around the issue was to reboot the server with the 192.168.1.04 line commented in the resolv.conf file and this fixed the issue. We were then later able to get the 192.168.1.04 DNS server back online and this resolved the issue on the remaining servers that had not been rebooted but had the resolv.conf file configured as follows: -

search domain.local
namesserver 192.168.1.03
nameserver 192.168.1.04

I guess what I am wondering is why the servers/applications seemed unable to failover to the secondary DNS server even though at the OS level the failover was working correctly. Does anyone have any thoughts/comments on how to configure this so that failover would work as we would like and prevent any outages in case of a single DNS failure?

Please accept my apologies for the length of this post and thanks for taking the time to read it.

PS. All local host files only contain the localhost and all the nsswitch.conf files were set to dns (noservercontinue) files.
0
Comment
Question by:mass2612
  • 4
  • 3
8 Comments
 
LVL 28

Expert Comment

by:peakpeak
ID: 19606816
Sounds like a cache issue. Does the server reread the resolve.conf file after it's been changed? Maybe tou only need to restart the DNS daemon, not the machine.
0
 
LVL 29

Author Comment

by:mass2612
ID: 19606941
That's the real question. I've read throughout the HP ITRC forums that you should simply be able to update the resolv.conf file and the file will be automatically re-read but it seems that its not. I'm not sure if its particular processes that read the file only when they start and therefore don't see the update or if there is something else I am missing.

0
 
LVL 28

Expert Comment

by:peakpeak
ID: 19607037
Do a test with your secondary DNS server, change the conf file and se if it responds to the new value, if not, restart the daemon and test again
0
Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

 
LVL 29

Author Comment

by:mass2612
ID: 19607055
Yes testing from the OS level works straight away but the applications seem to not read the change until after a reboot. If I modify the resolv.conf and then run nslookup hostname the update to the resolv.conf file is seen right away but any running application or new aplication fails.
0
 
LVL 28

Expert Comment

by:peakpeak
ID: 19607099
Well, nslookup is an application, albeit small. What applications are not working?
0
 
LVL 63

Accepted Solution

by:
SysExpert earned 1600 total points
ID: 19611830
Unfortunately a  lot of applications cache the DNS of relevant servers adn do not flush or update them frequently.

Not a lot you can do except learn how to flush the local caching manually per application, or to shut down and restart the application it self.

I hope this helps !
0
 
LVL 28

Assisted Solution

by:peakpeak
peakpeak earned 400 total points
ID: 19611984
Yep, caching is at times good, at others it's not time saving as it was meant to be. Microsoft use a LOT of caching, most of it probably good but, like you, I've met cases where you need to "push a button"
0
 
LVL 29

Author Comment

by:mass2612
ID: 19636327
Hi both - I agree pretty much with what you say. I think the only way for me to have this failover correctly would be to place dns into a clustered package using the clustered IP on the clients so that the apps that cache the dns config at startup would not require a restart.

Thanks again.
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Salesforce.com’s Console is a great tool to view activities, leads, contacts, accounts and opportunities all in one screen. It is particularly effective during call blocks and working numerous activities at a time in a quick, repetitive fashion (suc…
BIND is the most widely used Name Server. A Name Server is the one that translates a site name to it's IP address. There is a new bug in BIND (https://kb.isc.org/article/AA-01272), affecting all versions of BIND 9 from BIND 9.1.0 (inclusive) thro…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses
Course of the Month19 days, 7 hours left to enroll

872 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question