DNS resolution really slow on Win 2012 R2 with forwarders

To ensure an apples to apples comparison, you are doing your tests on the same network, using the same gateway device, and your DNS server(s) are also using the same Google servers as their only forwarders as your test laptop?

Have you tried removing your forwarder entries and just use the root hint servers. I have seen problems with forwarders and slow response in the past.

Scott Gorcester

@Cliff I am on the same network using the same router with 8.8.8.8 as my DNS (which is being used as the forwarder on the DNS server)

@Scott I didn't realize that would work.  I thought you had to specify DNS servers to forward to.  So, if I check the "use root hints if forwarders not available" box and remove the forwarders then DNS for public sites will still work?  It may even fix the speed issue?

You do not require a forwarder as root hints can provide all of the necessary information. I would also look for bad forwarder addresses as this can slow things down considerably. I also look to see if my root hints are properly listed and resolving.

Scott

I removed the forwarders and checked that I can still resolve names.  So far so good.  It was pretty fast too.  However, the problem is intermittent so I'll keep this open and see how tomorrow goes.  If we don't have DNS resolution problems tomorrow then I think we've found the culprit.

Bummer, that didn't fix it.  I'm still getting slow DNS responses.  A page will timeout and then about 5 seconds later load just fine.  I was able to get a screen cap of the error before the page successfully loads.  See attached.
DNS-Slow.PNG

Lets try putting a forwarder back in, use 4.2.2.1 for now, also have a look at this post.

https://support.opendns.com/entries/21684834-getting-slow-response-to-my-dns-lookups-

Scott

Since we are using AT&T Uverse, I updated our forwarders to be AT&T's DNS servers.  Maybe that'll help since they are on their network?

DNS1: 68.94.156.1
DNS2: 68.94.157.1

I noticed from the link you sent that some people fixed issues using ISP DNS instead of public.

Not a bad idea although I have not had this issue with using 8.8.8.8 or 4.2.2.2 in the past. The worst lookup issues I have found typically relate to an incorrect forwarder address. The incorrect forwarder would take forever to fail and then the lookups would succeed.

Scott

Hmm, I still have the issue.  In addition to these DNS issues I also get slow bandwidth for even LAN connectivity.  I'm thinking that may be related.  I didn't think it was since DNS is so lean, but then again if there is latency it could be affecting the DNS packets as well (causing timeouts).

Check for IP address conflicts, does the server have a single nic with a single IP address? What happens if you run an internet speed test?

Scott

The server is a VM and the host has several NICs setup so I don't think that's it.  I haven't seen any IP conflicts.  

However, I am suspect of two older switches installed here.  One is an old Cisco 1900 switch.  Quite possibly dying.  The other is a Zyxel switch... I just don't like Zyxel so that makes me a little biased there.

Oh yeah, and speed test came back fine on the DNS server.

Older switches might be the culprit. Only concerned if the Domain controller has multiple nics or IP's.

Scott

Not sure if this helps, but running this test gives intermittent fails.  Haven't tried replacing the switches, but I'm doubtful that is really the issue.
DNS-Test-Fails.PNG

Found this in the DNS log.  Dunno if it means anything pertinent.
DNS-Event-Error.PNG

Hmmm, I found old DNS server IPs on the network adapter on our DNS server.  What IPs should I have in there?  I have the loopback (127.0.0.1) and another server that we have an AD trust with.  Should I have others in there like 8.8.8.8?  What order should they be in?

The recursive test may fail if you are currently not using forwarders. Have you tried rebooting your router? Also is the firewall turned on on the DNS server?

Scott

YES OLD DNS server entries could absolutely cause this, NO do NOT put 8.8.8.8 on the DNS server IPV4 configuration. JUST PUT the DNS server itself there. I don't typically use 127.0.0.1 (loopback)

I think you found the issue

Scott

How many DNS servers do you have on the network

Scott

Ok, I put in only two entries on the adapter: actual IP of the DNS server itself and below that the IP of the other AD server with which we have a trust (it made building the trust possible, don't know if it is still needed though).

This server is the only DNS server we have on the network.  (the AD server we have a trust with is on another network we connect to over site-to-site VPN)

This server does have its Windows Firewall enabled.

Anything else?

Okay that sounds fine, you can put the address of each domain controller as primary and the other domain controller as secondary, also make sure that both servers are replicating properly and maybe confirm that your records are updating properly on both Domain controllers.

This should resolve your issues

Scott

Hmmm.... I removed all but the local IP of the domain controller in the DNS list on the only network adapter on the DC.  We are STILL getting DNS errors...

However, a pattern has emerged, it happens a lot more frequently at lunch and in the late afternoon...  Now, we have a 300Mbps symetrical pipe (Uverse) with QoS giving streaming data (ie YouTube) only 20Mbps.  So... what gives?  Why is DNS failing when we have a lot of users surfing?  Should I QoS DNS with a high priority (port 53)?  Do you think it is getting choked out?

We have about 50 users here.

Can you post the DNS errors you are seeing?

Scott

I have posted the errors we are getting earlier in this thread, but I've attached a new one.
DNS-Event-Error2.PNG

I found this in our logs when I do a debug log with our DNS server.  That IP is for a DNS server I removed (I thought gracefully).  When I look in the SOA on our actual DNS server the other one is NOT listed... Where is this getting referenced?

Forgot to upload it
DNS-SOA-Error.PNG

You may have to look through your DNS server entries and manually clean up the stale records.

Scott

I'm not sure what I'm looking for.  Any recommendations?  I've gone through DNS a million times looking for settings referencing old DNS servers.  What gets me is that the new DNS server should failover to other DNS servers if it gets a response, but it isn't doing that.

Here are my "advanced" settings in DNS.  Is something miss-configured?
DNS-Advanced-Properties.PNG

If the problem is that DNS fails to resolve and then a "few" seconds later resolves, could reducing the timeout time help?  See attached.
DNS-Forwarders.PNG

I don't see any problems here

I didn't see this mentioned yet (but I may have missed it), but is there anything on your network blocking UDP packets or UDP 53 in particular?

The DNS service attempts to resolve via UDP 53.  When that times out, it resolves using TCP 53.  That would account for the delay.

I had a very similar situation happen to one of my customers yesterday.  In this case, though, the network was being flooded with BOOTPC packets from a Windows 2012 R2 DHCP Server whose rogue detection was out of control.  Stopping that service and letting the network settle down solved the problem.

Go into task manager, expand "More details", go to the "Performance" tab and then click on "Open Resource Meter" at the bottom.  Go to the "Network" tab and look to see if any services are acting wonky.

I worked with MS support and found out about an issue with MS DNS that makes the packets rather large.  By slimming down the DNS packet size we significantly decreased the errors we were seeing.  This leads me to think it has something to do with network traffic.  I also implemented QoS to prioritize DNS requests to help the issue.

Article: https://support.microsoft.com/en-us/kb/832223

I did notice a lot of LSASS.exe traffic on our DC from a dev box.  I'll try to replace that dev box.  Also, I know its bad practice, but just how "bad" is it that our DC is also our print server?

Yeah... I think MS uses EDNS which is extended DNS.  I know that when we've had Cisco firewalls, we've had to disable the inspection on DNS because they don't like the larger packet size.

What firewall are you using?

Palo Alto 500

Maybe disable:
- Passive DNS monitoring.
- DNS sinkholing

If, they're even enabled.

I found out that it was the modem.  No one suggested that the modem was the issue.  Not really their fault, very strange issue.

Wow... a session limit on an ISP's modem?  I mean, I've seen that many times on firewalls that require per-user licenses... but an ISP?

Glad you figured it out!

Yeah, watch out for AT&T Uverse (Business Account).  300mbps/300mbps but with a session limit...

Did they modify that?  I mean, that's not reasonable.

They said it is a limitation of the modem and that it is the only modem they offer.  I'm not pleased with this answer and working with the sales rep to find a viable solution.