Solved

DNS Queries for external domains timeout

Posted on 2011-03-23
22
746 Views
Last Modified: 2012-05-11
Hi,

We are a W2K8 R2 AD environment; most clients are windows XP Pro using DHCP to obtain their IP information.

Our AD domain name is providence.on.ca.  Our DNS servers are 10.1.0.150 & 10.1.0.151.

When I query an external domain (ie google.com), there is a delay.  Usually the query fails.  However, if I re-query the same domain, the query returns successfully.  

I've ran wireshark, and when I query www.google.com, the domain that is actually being queried is www.google.com.providence.on.ca.  Once that fails, it then attempts to query www.google.com.on.ca.  Once that fails it queries the www.google.com, and eventually returns a result.
 Screenshot of Wireshark
If I attempt to resolve the name of an internal host, it also automatically adds the suffix to the name.  It also adds the suffix if I also place the suffix in the query.

 Screenshot of wireshark - internal name
Here is an NSLookup in Debug mode:

 
H:\>nslookup
Default Server:  phcdc01.providence.on.ca
Address:  10.1.0.150

> set debug
> www.google.com
Server:  phcdc01.providence.on.ca
Address:  10.1.0.150

------------
Got answer:
    HEADER:
        opcode = QUERY, id = 2, rcode = NXDOMAIN
        header flags:  response, auth. answer, want recursion, recursion avail.
        questions = 1,  answers = 0,  authority records = 1,  additional = 0

    QUESTIONS:
        www.google.com.providence.on.ca, type = A, class = IN
    AUTHORITY RECORDS:
    ->  providence.on.ca
        ttl = 3600 (1 hour)
        primary name server = phcdc01.providence.on.ca
        responsible mail addr = hostmaster
        serial  = 56877
        refresh = 900 (15 mins)
        retry   = 600 (10 mins)
        expire  = 86400 (1 day)
        default TTL = 3600 (1 hour)

------------
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 3, rcode = NXDOMAIN
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 0,  authority records = 1,  additional = 0

    QUESTIONS:
        www.google.com.on.ca, type = A, class = IN
    AUTHORITY RECORDS:
    ->  ca
        ttl = 820 (13 mins 40 secs)
        primary name server = jbq01.prd.cira.ca
        responsible mail addr = admin-dns.cira.ca
        serial  = 2011032316
        refresh = 1800 (30 mins)
        retry   = 900 (15 mins)
        expire  = 604800 (7 days)
        default TTL = 3600 (1 hour)

------------
DNS request timed out.
    timeout was 2 seconds.
timeout (2 secs)
*** Request to phcdc01.providence.on.ca timed-out
>

Open in new window


and trying again after a few seconds:

 
>
> www.google.com
Server:  phcdc01.providence.on.ca
Address:  10.1.0.150

------------
Got answer:
    HEADER:
        opcode = QUERY, id = 5, rcode = NXDOMAIN
        header flags:  response, auth. answer, want recursion, recursion avail.
        questions = 1,  answers = 0,  authority records = 1,  additional = 0

    QUESTIONS:
        www.google.com.providence.on.ca, type = A, class = IN
    AUTHORITY RECORDS:
    ->  providence.on.ca
        ttl = 3600 (1 hour)
        primary name server = phcdc01.providence.on.ca
        responsible mail addr = hostmaster
        serial  = 56877
        refresh = 900 (15 mins)
        retry   = 600 (10 mins)
        expire  = 86400 (1 day)
        default TTL = 3600 (1 hour)

------------
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 6, rcode = NXDOMAIN
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 0,  authority records = 1,  additional = 0

    QUESTIONS:
        www.google.com.on.ca, type = A, class = IN
    AUTHORITY RECORDS:
    ->  ca
        ttl = 787 (13 mins 7 secs)
        primary name server = jbq01.prd.cira.ca
        responsible mail addr = admin-dns.cira.ca
        serial  = 2011032316
        refresh = 1800 (30 mins)
        retry   = 900 (15 mins)
        expire  = 604800 (7 days)
        default TTL = 3600 (1 hour)

------------
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 7, rcode = NOERROR
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 6,  authority records = 0,  additional = 0

    QUESTIONS:
        www.google.com, type = A, class = IN
    ANSWERS:
    ->  www.google.com
        canonical name = www.l.google.com
        ttl = 35933 (9 hours 58 mins 53 secs)
    ->  www.l.google.com
        internet address = 74.125.225.19
        ttl = 247 (4 mins 7 secs)
    ->  www.l.google.com
        internet address = 74.125.225.20
        ttl = 247 (4 mins 7 secs)
    ->  www.l.google.com
        internet address = 74.125.225.16
        ttl = 247 (4 mins 7 secs)
    ->  www.l.google.com
        internet address = 74.125.225.17
        ttl = 247 (4 mins 7 secs)
    ->  www.l.google.com
        internet address = 74.125.225.18
        ttl = 247 (4 mins 7 secs)

------------
Non-authoritative answer:
Name:    www.l.google.com
Addresses:  74.125.225.19, 74.125.225.20, 74.125.225.16, 74.125.225.17
          74.125.225.18
Aliases:  www.google.com

>

Open in new window


Thanks

0
Comment
Question by:Providence_Healthcare
  • 8
  • 8
  • 6
22 Comments
 
LVL 38

Expert Comment

by:Hypercat (Deb)
ID: 35202364
This behavior is by design (it's called "devolution"), so I'm suspicious that it's not the actual cause of the name resolution problems you're experiencing. Here's a quick and simple article explaining it:

http://technet.microsoft.com/en-us/library/ee683928(WS.10).aspx

Are you using forwarders on this DNS server?
0
 

Author Comment

by:Providence_Healthcare
ID: 35206409
We are using forwarders on the DNS servers.  I have verified that the servers are valid & functioning.

Using the Monitoring tab in DNS, both the simple & recursive queries pass.


If it is an issue with DNS Suffix Devolution, what would I need to set for the Devolution & Devolution Level?

0
 
LVL 20

Expert Comment

by:Iain MacMillan
ID: 35207379
you should internal and external DNS forwarders set such as your ISP DNS server, or better yet, Open DNS servers (with free basic content filtering).  I have the same setup, 2008 R2 DC's, with users systems on XP, except for 2 IT systems testing Win 7 64bit.

OpenDNS nameservers are 208.67.222.222 and 208.67.220.220.
0
 
LVL 20

Expert Comment

by:Iain MacMillan
ID: 35207441
ah apologies, i left my desk for a few hours, and didn't post what i had typed.

do you have your DC's only listening on their own IP addresses (.150 and .151)?
does this happen with systems such as servers with static IP & DNS settings - might be a DHCP scope issue?
have you also checked your firewall settings, to make sure it also has just the 2 named DC's listed?
0
 

Author Comment

by:Providence_Healthcare
ID: 35207562
Each DC has itself and the other DC listed in its IP configuration.

Systems that are assigned a static IP address exhibit the same symptom (but they have the same DNS settings as what is being assigned via DHCP).

I'm not sure which firewall you are asking me to verify - the permiter firewalls allow both UDP & TCP outbound on port 53 from both domain controllers.
0
 
LVL 20

Expert Comment

by:Iain MacMillan
ID: 35207943
the listening setup is done under the server properties in DNS management, of the IP address of each server, you would have it point to each other as primary and secondary would be 127.0.0.x (where X would be 150 & 151 in your case, last digit of internal IP), as I found if you entered the DNS servers own real IP, on restart sometimes the DNS server service won't be ready, and I get auth errors on the 2008 R2 DC's, since using the 127 trick, i don't unless i'm daft enough to reboot both DC's at the same time!!

On your DHCP Scope options, how many items do you have listed??  I have 4 - 003 Router., 006 DNS Server, 015 DNS domain name & 042 NTP Servers.
0
 

Author Comment

by:Providence_Healthcare
ID: 35208015
I have changed it so that each DC has it's partners IP as the primary DNS server, and the loopback address as the secondary DNS server.

I have the following DHCP Scope options:

03 - router
06 - dns server - set to 10.1.0.151 and 10.1.0.150
15 - DNS Domain name - providence.on.ca
42 - NTP Server - set to the DC's
44 - WINS Server - set to the DC's (they are running WINS)
46 - WINS Node Type (set to 0x8).

0
 
LVL 38

Expert Comment

by:Hypercat (Deb)
ID: 35208277
We obviously have experienced different results with different settings. I have always used the DNS server's own IP address (or you could use the loopback) as the primary DNS server in the TCP/IP settings and either left the secondary out or used another DNS server as secondary. This is based on having an AD-integrated DNS zone. There are pros and cons to listing a different DC first, one of the cons being that it generates more DNS traffic on your network, so you might want to reconsider that change.  

Your DHCP scope options look fine.

Here's an article that explains an update to DNS involving devolution (which you will want to install if you haven't already), and some workarounds for your situation:

http://www.microsoft.com/technet/security/advisory/971888.mspx

One of the recommended steps which may resolve your issue is to set a specific DNS suffix search list.  Here's an article that includes instructions about setting a DNS suffix search list through group policy:

http://support.microsoft.com/kb/294785/
0
 
LVL 20

Expert Comment

by:Iain MacMillan
ID: 35213492
all our domain systems have a domain suffix applied when they are added to the domain.  in your case it would be PROVIDENCE.ON.CA, set under the TCP/IP v4 properties and the Adv, DNS tab.  Also tick the 2 boxes under that option if not already done.
0
 

Author Comment

by:Providence_Healthcare
ID: 35214647
See attached for my current settings, where the problem is occurring.

 My computer settings
0
 
LVL 38

Expert Comment

by:Hypercat (Deb)
ID: 35215621
I think the gist of the articles that I posted above is that you should try using a specific DNS suffix list instead of using the checkboxes on the DNS tab.  Unless you need to have the "on.ca" domain searched separately, you should click the Append these DNS suffixes box instead and enter "providence.on.ca" in the list. That way, ONLY the providence.on.ca domain would be used and devolution (the process that causes it to search "providence.on.ca" and then "on.ca") would be turned off.  This can be done through group policy as well, under Computer Configuration/Policies/Administrative Templates/Network/DNS Client/DNS Suffix Search List.
0
6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

 

Author Comment

by:Providence_Healthcare
ID: 35218745
Hi,

I've tried adding providence.on.ca into the 'Append these DNS suffixes (in order):' box.

When I add providence.on.ca, and NSLookup www.google.com, it 1st queries www.google.com.providence.on.ca, followed by www.google.com.  The response from the DNS server does not come until over 5 seconds later.


From my workstation if I run "nslookup www.google.us 205.42.210.205" it queries www.google.us.providence.on.ca.  It then queries www.google.us and seems to not return a response.

From my workstation if I run "nslookup www.google.us 208.67.222.222"  it queries www.google.us.providence.on.ca and returns a response in 0.57 seconds.  It's the wrong answer as OpenDNS tries to redirect to itself.

From my workstation if I run "nslookup www.google.us" it queries www.google.us.providence.on.ca followed by www.google.us.  A response was returned almost instantly (0.005 seconds).


From the Domain Controller (and DNS server):

I'm not going to install wireshark on the DC on a Friday afternoon.

nslookup www.google.com times out 3 times, followed by an answer.
nslookup www.google.com 208.67.222.222 answers quickly, but it's the wrong answer (since www.google.com.providence.on.ca is what is queried).
nslookup www.google.com 205.42.210.205 times out 3 times followed by an answer.


On my workstation if I add a '.' above the providence.on.ca into the 'Append these DNS suffixes (in order):' box, nslookup is still flakey, but the response with PING is as expected.


0
 
LVL 20

Expert Comment

by:Iain MacMillan
ID: 35230222
you need to have your workstations set a DNS suffix of PROVIDENCE.ON.CA ( i have ALL my systems set with a suffix domain), then with OpenDNS set as the external forwarders on the DC's, you should be seeing correct/fast results from your tests.

IPCONFIG for your systems should return only the DC's as DNS servers.
0
 
LVL 38

Expert Comment

by:Hypercat (Deb)
ID: 35232767
IainNIX's answer is one approach. The other is to remove the forwarders from your DNS servers and try running the same queries using the root servers only.  This should work, but if it doesn't it might point us to something else that might be wrong with your DNS server or the configuration.
0
 

Author Comment

by:Providence_Healthcare
ID: 35233014
IainNix: I have taken a workstation, and set a manual suffix of providence.on.ca.  Name resolution is still slow / unreliable.

hypercat: on one of the DC's I've removed the forwarders.  From the DC when running the command 'nslookup www.google.com 127.0.0.1' this is the result:

C:\Users\mynamehere>nslookup www.google.com 127.0.0.1
Server:  localhost
Address:  127.0.0.1

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** Request to localhost timed-out

(I've put the forwarders back in place for the time being).

0
 
LVL 38

Expert Comment

by:Hypercat (Deb)
ID: 35233191
If a DNS query to the localhost IP address times out, that would seem to indicate that the DNS server or client service on the local machine is not running, or for some reason not listening on 127.0.0.1.  Have you checked the status of the services and also the DNS event log to see if there are any errors or problems there?
0
 
LVL 38

Expert Comment

by:Hypercat (Deb)
ID: 35233244
You could also run "netstat -a" and send it to a text file so that you can examine it to see whether the IP address of the server and/or the loopback address is listening on port 53.
0
 

Author Comment

by:Providence_Healthcare
ID: 35233297
See attached.

I've removed the ESTABLISHED and TIME_WAIT connections.

 netstat-output.txt
0
 
LVL 20

Expert Comment

by:Iain MacMillan
ID: 35233329
the DC's primary DNS setting should normally be the other internal IP of the other DNS server and the second IP would be 127.0.0.x (where X is the last number of the actual internal IP of that server).

for example, if your DNS server was set to 10.1.0.150, then its loopback or secondary DNS setting would be 127.0.0.150 - so the server you tried this on should have DNS1=10.1.0.151 and DNS2=127.0.0.150.  See if you can redo the test and get a positive result.  As Hypercat says, you may have other issues adding to the problem.

make sure there is not any security software or firewalls blocking or delaying TCP port 53 access.
0
 
LVL 38

Accepted Solution

by:
Hypercat (Deb) earned 125 total points
ID: 35233411
That sure looks normal to me. Try the nslookup without specifying the "127.0.0.1" and see what results you get.  Also try the nslookup and specify an external DNS server, such as Google's own DNS server 8.8.8.8.

What else is running on this server? Is it at all possible that there is a misbehaving router somewhere on your network that is blocking port 53 outgoing?
0
 

Author Comment

by:Providence_Healthcare
ID: 35233622
Hypercat: I've changed the DNS forwarders to 8.8.8.8 (and 8.8.4.4).  DNS requests now come back very quickly.

I will test further.
0
 
LVL 38

Expert Comment

by:Hypercat (Deb)
ID: 35233685
OK - that seems to indicate to me two things.  One, the DNS forwarders you were using were not responding properly.  And B, your server may not have a correct (or any) root hints list on it. I would look at both of those things.
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Occasionally you run into the website or two that will not resolve properly using your own DNS servers.  Some people simply set up global forwarders for their DNS server.  I don’t recommend doing this because it can cause problems resolving addresse…
I've written instructions for one router type, but this principle may be useful for others of the same brand and even other brands of router. Problem: I had an issue especially with mobile devices that refused to use DNS information supplied via…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now