DNS issue with resolving request

Without wanting to disagree with @David Johnson, I would like to throw another thought in.  When dealing with multiple Windows based name servers, I have seen the problems that you describe on several occasions.  The problem in each case was being caused by Forwarding and the solution was to find a way to get DNS working as required, without using Forwarding.
I'm not sure if this would be possible with your requirements, but if it is, I would recommend not using Forwarding at all.

I would use nslookup on the problematic domain name and see which DNS server can and cannot resolve it.

For example, let's assume you are having problems resolving difficult.com, your internal DNS server is 10.0.0.2, and your external DNS server is 10.1.0.2.  I'd run the following on a Windows machine:

start
run
cmd
nslookup
difficult.com
server 10.0.0.2
difficult.com
server 10.1.0.2
difficult.com
server <put in the IP address of your first forwarder here>
difficult.com
server 4.2.2.2
difficult.com
exit
exit

The responses you get will give you a good clue of what can and cannot resolve the name.  If you can post the results, that will make it much easier to pinpoint where you are running into trouble.

@compprobsolv The major site we have seen issues with is ed.gov and fasfa.gov. outside our network or if we set our DNS to 8.8.8.8 it works fine.

@compprobsolv My company wants to keep recursion disabled on our Public DNS. If Recursion is disabled I can't turn on forwarders to google etc.

I did test turning recursion ON on the public DNS and then we can resolve ed.gov and fasfa.gov. But as soon as we turn recursion back off and ed.gov and fasfa.gov get cleared out of DNS than we can't ping/nslookup those addresses again.



Thanks

Still having issues - here is a picture of our set up https://ibb.co/er2OgF

How "public" are those DMZ DNS servers? Can they be queried from the Internet, or is there a firewall preventing that? If the latter, I'm not sure I see the point of disabling recursion on them.

Also, I'm curious: are your root DNS servers (the ones that forward to the DMZ) able to resolve Internet hostnames using only their root hints? (You can test this by removing all forwarders from one of them and clearing its cache.)

I'm sure that there's a good reason why you haven't considered the suggestion that I've already made, but just in case you've missed it: I would strongly recommend not using Forwarding and allow your internal DNS servers to act independently.  If there are zones on the servers in the DMZ that need reflecting on your internal servers then setup stub zones on the internal servers to pull a copy of those domains/zones over.

Are you scavenging dns record if not, you may be suffering from old and obsolete info, dns gets finicky when it can't find what it is looking for.
root hints are nice to have but they are slow and causes problem. Forwarders you can trust are comodo, norton, opendns and google. However I'd setup my forwarders like this 127.0.0.1 followed by 208.67.222.222 reason is I want anything local that I'm looking for is to check locally first b4 going to the internet.

So now we are leaving our DMZ for just public records of our websites we control. our internal DNS servers still point to our Root servers. Those servers we are using forwarding on (they are behind our firewall) and that seems to be working great. I still don't understand why the root hints were not resolving...

Thanks

There's some confusion about what a Windows DNS server does when you disable recursion, so I set up a test environment similar to yours: I've got DNS1 configured to forward to DNS2, which has recursion disabled. Both servers have a valid list of root hints. I then configured a client to use DNS1 as its only DNS server, cleared caches everywhere, started packet captures on both DNS servers, and had the client ping www.ed.gov. I'll let you decide whether the results are what you would expect to see.

DNS1 received the query from the client and immediately forwarded it to DNS2, as expected. DNS2 responded with a referral, which consisted of the FQDNs and IP addresses of all 13 root servers. That's it. DNS1 then picked one of those root servers and sent the query to it. The rest is as you would expect: after walking the tree, DNS1 eventually received an authoritative response and sent the response back to the client. There's a little more in the packet trace, because www.ed.gov is a CNAME and because the client got impatient and re-sent the query to DNS1, but that's the gist of it.

So, disabling recursion doesn't just tell a server not to use forwarders; it won't use its own root hints either, except to include them in a referral response. This makes sense, I suppose, but it doesn't seem to be well documented on the Windows side.

In your particular case, a packet capture on one of your root DC servers will likely show why it can't resolve those FQDNs. If I had to guess, I'd suspect that your firewall rules only allow the DMZ servers to query the public root DNS servers, so the root DCs can't do anything with a referral response.

BTW, it's fafsa.gov, not fasfa.gov. :)

@DrDave:

Thank you for an excellent explanation.  I've been reading about recursion since seeing this thread but your explanation finally clarified it.

To summarize: if recursion is on, the DNS server will try to get an answer itself and then pass the answer back to who requested it.  If recursion is off, the DNS server will send back a list of who to ask instead and considers itself done with the request.  I'm assuming in both cases that the DNS server doesn't have the answer already.

Do I have this right?

Exactly. If the server receives a query for something it's got in its own database (something it's authoritative for, in other words), it'll provide an authoritative response regardless of whether recursion is enabled or not. I'm assuming it'll also provide a cached response if it's got one, but if recursion has been disabled for a while, there's probably not a lot in its cache.