DNS resolution really slow on Win 2012 R2 with forwarders

This is strange to me, but my DNS server on my network is periodically resolving public DNS really slow and sometimes timing out.  However, if I put Google's DNS servers as my primary DNS on my laptop, my laptop it is lightening fast.  What gives?  I tried clearing DNS cache on the server and rebooting it over the weekend, but it has not helped.

Extra info: my DNS server is on my Active Directory server and is Active Directory integrated.
Capture.PNG
LVL 6
Tim PhillipsWindows Systems AdministratorAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Cliff GaliherCommented:
To ensure an apples to apples comparison, you are doing your tests on the same network, using the same gateway device, and your DNS server(s) are also using the same Google servers as their only forwarders as your test laptop?
Scott GorcesterCTOCommented:
Have you tried removing your forwarder entries and just use the root hint servers. I have seen problems with forwarders and slow response in the past.

Scott Gorcester
Tim PhillipsWindows Systems AdministratorAuthor Commented:
@Cliff I am on the same network using the same router with 8.8.8.8 as my DNS (which is being used as the forwarder on the DNS server)

@Scott I didn't realize that would work.  I thought you had to specify DNS servers to forward to.  So, if I check the "use root hints if forwarders not available" box and remove the forwarders then DNS for public sites will still work?  It may even fix the speed issue?
Expert Spotlight: Joe Anderson (DatabaseMX)

We’ve posted a new Expert Spotlight!  Joe Anderson (DatabaseMX) has been on Experts Exchange since 2006. Learn more about this database architect, guitar aficionado, and Microsoft MVP.

Scott GorcesterCTOCommented:
You do not require a forwarder as root hints can provide all of the necessary information. I would also look for bad forwarder addresses as this can slow things down considerably. I also look to see if my root hints are properly listed and resolving.

Scott
Tim PhillipsWindows Systems AdministratorAuthor Commented:
I removed the forwarders and checked that I can still resolve names.  So far so good.  It was pretty fast too.  However, the problem is intermittent so I'll keep this open and see how tomorrow goes.  If we don't have DNS resolution problems tomorrow then I think we've found the culprit.
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Bummer, that didn't fix it.  I'm still getting slow DNS responses.  A page will timeout and then about 5 seconds later load just fine.  I was able to get a screen cap of the error before the page successfully loads.  See attached.
DNS-Slow.PNG
Scott GorcesterCTOCommented:
Lets try putting a forwarder back in, use 4.2.2.1 for now, also have a look at this post.

https://support.opendns.com/entries/21684834-getting-slow-response-to-my-dns-lookups-

Scott
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Since we are using AT&T Uverse, I updated our forwarders to be AT&T's DNS servers.  Maybe that'll help since they are on their network?

DNS1: 68.94.156.1
DNS2: 68.94.157.1

I noticed from the link you sent that some people fixed issues using ISP DNS instead of public.
Scott GorcesterCTOCommented:
Not a bad idea although I have not had this issue with using 8.8.8.8 or 4.2.2.2 in the past. The worst lookup issues I have found typically relate to an incorrect forwarder address. The incorrect forwarder would take forever to fail and then the lookups would succeed.

Scott
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Hmm, I still have the issue.  In addition to these DNS issues I also get slow bandwidth for even LAN connectivity.  I'm thinking that may be related.  I didn't think it was since DNS is so lean, but then again if there is latency it could be affecting the DNS packets as well (causing timeouts).
Scott GorcesterCTOCommented:
Check for IP address conflicts, does the server have a single nic with a single IP address? What happens if you run an internet speed test?

Scott
Tim PhillipsWindows Systems AdministratorAuthor Commented:
The server is a VM and the host has several NICs setup so I don't think that's it.  I haven't seen any IP conflicts.  

However, I am suspect of two older switches installed here.  One is an old Cisco 1900 switch.  Quite possibly dying.  The other is a Zyxel switch... I just don't like Zyxel so that makes me a little biased there.
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Oh yeah, and speed test came back fine on the DNS server.
Scott GorcesterCTOCommented:
Older switches might be the culprit. Only concerned if the Domain controller has multiple nics or IP's.

Scott
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Not sure if this helps, but running this test gives intermittent fails.  Haven't tried replacing the switches, but I'm doubtful that is really the issue.
DNS-Test-Fails.PNG
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Found this in the DNS log.  Dunno if it means anything pertinent.
DNS-Event-Error.PNG
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Hmmm, I found old DNS server IPs on the network adapter on our DNS server.  What IPs should I have in there?  I have the loopback (127.0.0.1) and another server that we have an AD trust with.  Should I have others in there like 8.8.8.8?  What order should they be in?
Scott GorcesterCTOCommented:
The recursive test may fail if you are currently not using forwarders. Have you tried rebooting your router? Also is the firewall turned on on the DNS server?

Scott
Scott GorcesterCTOCommented:
YES OLD DNS server entries could absolutely cause this, NO do NOT put 8.8.8.8 on the DNS server IPV4 configuration. JUST PUT the DNS server itself there. I don't typically use 127.0.0.1 (loopback)

I think you found the issue

Scott
Scott GorcesterCTOCommented:
How many DNS servers do you have on the network

Scott
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Ok, I put in only two entries on the adapter: actual IP of the DNS server itself and below that the IP of the other AD server with which we have a trust (it made building the trust possible, don't know if it is still needed though).

This server is the only DNS server we have on the network.  (the AD server we have a trust with is on another network we connect to over site-to-site VPN)

This server does have its Windows Firewall enabled.

Anything else?
Scott GorcesterCTOCommented:
Okay that sounds fine, you can put the address of each domain controller as primary and the other domain controller as secondary, also make sure that both servers are replicating properly and maybe confirm that your records are updating properly on both Domain controllers.

This should resolve your issues

Scott
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Hmmm.... I removed all but the local IP of the domain controller in the DNS list on the only network adapter on the DC.  We are STILL getting DNS errors...

However, a pattern has emerged, it happens a lot more frequently at lunch and in the late afternoon...  Now, we have a 300Mbps symetrical pipe (Uverse) with QoS giving streaming data (ie YouTube) only 20Mbps.  So... what gives?  Why is DNS failing when we have a lot of users surfing?  Should I QoS DNS with a high priority (port 53)?  Do you think it is getting choked out?

We have about 50 users here.
Scott GorcesterCTOCommented:
Can you post the DNS errors you are seeing?

Scott
Tim PhillipsWindows Systems AdministratorAuthor Commented:
I have posted the errors we are getting earlier in this thread, but I've attached a new one.
DNS-Event-Error2.PNG
Tim PhillipsWindows Systems AdministratorAuthor Commented:
I found this in our logs when I do a debug log with our DNS server.  That IP is for a DNS server I removed (I thought gracefully).  When I look in the SOA on our actual DNS server the other one is NOT listed... Where is this getting referenced?
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Forgot to upload it
DNS-SOA-Error.PNG
Scott GorcesterCTOCommented:
You may have to look through your DNS server entries and manually clean up the stale records.

Scott
Tim PhillipsWindows Systems AdministratorAuthor Commented:
I'm not sure what I'm looking for.  Any recommendations?  I've gone through DNS a million times looking for settings referencing old DNS servers.  What gets me is that the new DNS server should failover to other DNS servers if it gets a response, but it isn't doing that.
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Here are my "advanced" settings in DNS.  Is something miss-configured?
DNS-Advanced-Properties.PNG
Tim PhillipsWindows Systems AdministratorAuthor Commented:
If the problem is that DNS fails to resolve and then a "few" seconds later resolves, could reducing the timeout time help?  See attached.
DNS-Forwarders.PNG
Scott GorcesterCTOCommented:
I don't see any problems here
Joseph HornseyPresident and JanitorCommented:
I didn't see this mentioned yet (but I may have missed it), but is there anything on your network blocking UDP packets or UDP 53 in particular?

The DNS service attempts to resolve via UDP 53.  When that times out, it resolves using TCP 53.  That would account for the delay.

I had a very similar situation happen to one of my customers yesterday.  In this case, though, the network was being flooded with BOOTPC packets from a Windows 2012 R2 DHCP Server whose rogue detection was out of control.  Stopping that service and letting the network settle down solved the problem.

Go into task manager, expand "More details", go to the "Performance" tab and then click on "Open Resource Meter" at the bottom.  Go to the "Network" tab and look to see if any services are acting wonky.
Tim PhillipsWindows Systems AdministratorAuthor Commented:
I worked with MS support and found out about an issue with MS DNS that makes the packets rather large.  By slimming down the DNS packet size we significantly decreased the errors we were seeing.  This leads me to think it has something to do with network traffic.  I also implemented QoS to prioritize DNS requests to help the issue.

Article: https://support.microsoft.com/en-us/kb/832223
Tim PhillipsWindows Systems AdministratorAuthor Commented:
I did notice a lot of LSASS.exe traffic on our DC from a dev box.  I'll try to replace that dev box.  Also, I know its bad practice, but just how "bad" is it that our DC is also our print server?
Joseph HornseyPresident and JanitorCommented:
Yeah... I think MS uses EDNS which is extended DNS.  I know that when we've had Cisco firewalls, we've had to disable the inspection on DNS because they don't like the larger packet size.

What firewall are you using?
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Palo Alto 500
Joseph HornseyPresident and JanitorCommented:
Maybe disable:
- Passive DNS monitoring.
- DNS sinkholing

If, they're even enabled.
Tim PhillipsWindows Systems AdministratorAuthor Commented:
After a lot of troubleshooting and working with several vendors to diagnose the issue... it turns out our modem has a session limit on it (Max TCPIP sessions).  That's the issue.  We were maxing out on sessions and not able to make new connections to DNS servers for DNS requests.  ...I hate ISP's.  ...Uverse...

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Tim PhillipsWindows Systems AdministratorAuthor Commented:
I found out that it was the modem.  No one suggested that the modem was the issue.  Not really their fault, very strange issue.
Joseph HornseyPresident and JanitorCommented:
Wow... a session limit on an ISP's modem?  I mean, I've seen that many times on firewalls that require per-user licenses... but an ISP?

Glad you figured it out!
Tim PhillipsWindows Systems AdministratorAuthor Commented:
Yeah, watch out for AT&T Uverse (Business Account).  300mbps/300mbps but with a session limit...
Joseph HornseyPresident and JanitorCommented:
Did they modify that?  I mean, that's not reasonable.
Tim PhillipsWindows Systems AdministratorAuthor Commented:
They said it is a limitation of the modem and that it is the only modem they offer.  I'm not pleased with this answer and working with the sales rep to find a viable solution.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
DNS

From novice to tech pro — start learning today.