All DNS Records Disappeared

We've adopted a network that includes one Windows Server 2003 domain controller and about 70 workstaions of varying types. Mostly XP, a few W2Ks and a few Window 7s.  There are also a couple of member servers. The domain controller is running DNS as well as AD. I immediately noticed a lot of strange behaviour on the network.  Things like workstations and member servers being able to join the domin properly but not being able to log on after joining, citing reasons like "a domain controller for this domain cannot be found".  I then discovered about a dozen regularly occuring error (red) and warning (yellow) events in the system, applicaion, directory and dns logs that suggest communicaion issues between AD and DNS.  Then when I looked at the DNS applicaiton I discoverd that there are no forward lookup zones.  Obviously there are no DNS records in those forward lookup zones because the forward lookkup zones dont exist.  There are three reverse lookup zones each with two records in them. The strange thing is that the network is still reasonably functional.  Alot of staff members can log on wth their domain accounts and those that cant just log on with local workstation acoounts.  The bottom line is that I need to get rid of this strange behaviour, fix the errors, and get the DNS server working as it should.  Any help would be appricated.
Who is Participating?
TomislavjConnect With a Mentor System AdminCommented:
Why don't you try to delete dns completely and then create new one?
TomislavjConnect With a Mentor System AdminCommented:
how many subnets are there? if only one i would recreate complete dns with forward and reverse lookup zones from start (deleting old one) and through dhcp asssign correct network settings to all clients

also, you can run diagnostic against dns server with script like this (if you don't have the support tools installed, install them from your server install disk):

DFSUTIL /PurgeMupCache
DCDIAG /V /C /D /E /s:servername > c:\dcdiag.log
netdiag.exe /v > c:\netdiag.log
repadmin.exe /showrepl servername /verbose /all /intersite > c:\repl.txt
Bruno PACIConnect With a Mentor IT ConsultantCommented:

Can you confirm that in your IP settings on your servers, DCs and computers you NEVER have mentionned any external DNS server ??
Internal machines (DC, servers and computers) must ONLY use internal DNS servers.

Have a good day.
Has Powershell sent you back into the Stone Age?

If managing Active Directory using Windows Powershell® is making you feel like you stepped back in time, you are not alone.  For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why.

sswmooreAuthor Commented:
Thanks Pac1B, Actually, I can confirm that the DC and most other computers DO refer to an external DNS server as well as the internal DNS server. Should this be changed so that only the internal DNS server, the DC, is refered to? More specifically, on the DC, when I check the TCPIP settings on the NIC, the IP address of the primary DNS server is the same as the IP address of the DC, and the secondary is set to an external DNS server, not on our network.  And this is the case for almost all of the workstations and the few member servers we have as the DHCP server is configured to supply the DNS settings to each DHCP client in that manner.  By the way, the DC does not perform DHCP services. DHCP is done by a different device.
Bruno PACIConnect With a Mentor IT ConsultantCommented:

As I told you, internal computers member of an AD domain MUST ONLY use internal DNS server ("computers" means all machine, even DCs !!).

The fact that the internal DNS server appears as the primary DNS does not help at all.

The Microsoft DNS client (I'm not sure with Linux but I suppose it's the same because this behavior is probably in a RFC) takes care of what I may call "DNS preferred server".
When your DNS client will have to resolve a DNS name it will AT FIRST interrogate its current DNS preferred server. And the preferred DNS server IS NOT the primary DNS server.
So what is the preferred DNS server ? Well... very simple. The preferred DNS server is the DNS server that had answered to the last DNS query.
That means that a DNS client will always interrogate the same DNS server (primary or not) until this DNS server stop to answer. In this case the DNS client will then try the next DNS server in its IP settings list and will use this new one until this one stops to answer.
This is important: a Microsoft DNS client do not always interrogate the primary DNS server at first for a DNS request !

Also, what you have to know is the difference between "authoritative answer" and "non-authoritative answer".
Positive DNS answers are always authoritative (do not confuse authoritative answer and authoritative server that is another thing that don't have link with your problem).
Negative answers may be authoritative or not.
A negative non-authoritative answer is in fact something like "I don't know this name".
A negative AUTHORITATIVE answer is in fact something like "IT DOES NOT EXIST" , meaning no need to search more for it...

And finally, what you have to know is that external DNS servers on the web ALWAYS give AUTHORITATIVE answers ! They can not give a negative non-authoritative answer.
This is because on Internet all DNS servers can reach the DNS root zone that is supposed to know ALL the existing DNS domain names in the world (public domain names of course !).

So what happens if your ask an external DNS server for an internal DNS name ?
The external DNS server will give you a negative authoritative DNS answer that will say "This domain does not exist !" and as it is authotitative you DNS client will not try to resolve it through another DNS server because the answer is clear : the DNS server said the domain does not exist and the answer is authoritative so it can't exist anywhere else.

So what happens randomly in your domain is that if for any reason (service stop, temporary DNS server hanging, low speed DNS answer...) the DNS client on a computer/server does not receive an answer from the internal DNS server it will consider it as defunct and start to interrogate the next one (the external DNS server) and consider now this one as its preferred DNS server. But this new DNS server is alive but always answers that internal DNS domain does not exist !
Dah dah !! Your internal DNS client is now unable to reach the internal AD domain because it is unable to resolve DNS domain name ! The computer/server is unable to authenticate clients, it is unable to authenticate itself, etc, etc..

As the DNS client maintains a DNS cache the problem may appear some long minutes after the failover to the external DNS.

So, that's why you must NEVER use external DNS server on internal machines in and AD domain.

What must be done is that:

1) All machines must be configured to only interrogate internal DNS servers that host the same copy of internal DNS zones (all interrogated DNS server must be able to answer the same thing to the same request).
2) On your DNS server, in the DNS console you must add a DNS forwarder that points to external DNS servers. Forwarders are used by a DNS server to transmit request that are not for a DNS zone that is locally hosted.
With a forwarder, your DNS server will transmit the DNS request to external DNS servers and wiat for the answer, then the DNS server will transmit the answer to the requesting client.

To resolve your case, first of all add the forwarder on your internal DNS server and if needed make things on firewall so that this internal DNS server is able to dialog with external DNS server.
Then on each internal machine, modify the IP settings on all NICs so that only internal DNS servers are used. And don't forget to make an IPCONFIG /FLUSHDNS to empty the DNS cache or else it will take very long minutes before the problem solves.
Or course, you should correct the DC first, then the servers and finally the clients.

As you use DHCP for computers, you must do the change in the DHCP range, but this change will only appliy on computer at the end of the DHCP lease, or at the next reboot of the client.

Have a good day.
sswmooreAuthor Commented:
Thanks very much Pac1B for the highly detailed comment. You obviously spent alot of time on that and there's some great information there that I didn't know.  I will change the TCPIP settings on the DCs NIC so that there is no refrence to external DNS server and I will configure several workstations in the same manner and see what results I get.  I'll run the IPCONFIG /FLUSH as well.  I will point out that whoever set this up did add forwarders and they are present in the DNS manager.

In my original question I mentioned that there are about a dozen errors in the event logs that seem to indicate AD and/or DNS issues.  I have them listed in an excel spread sheet which i will attach.

Thanks Tomislavj for the script.  I will run it. I'm hoping that it will at least shed light on these errors.
Bruno PACIConnect With a Mentor IT ConsultantCommented:

Ok so your internal DNS servers already have DNS forwarders configured and these forwarders are correct (it must point to IP addresses of external DNS servers). That's good.

Now you must remove external DNS servers on any NIC settings on all internal machines.
Of course, you should start fixing Domain Controllers because if they are not able to locate each other your domain can't work and there'll still be issues on clients.
So fix the DC at first, one by one, reboot each one and check AD replication.
Then fix other servers, and force them to register in DNS (IPCONFIG /REGISTERDNS).
Finally fix your clients.

I took a look at your Excel file. Except the fourth event (that is about TREND) all I've seen may be a sequel of your bad DNS configuration, because all these errors all linked to DNS resolution.

After I answered to you I decided to write an article about the behavior of DNS client:
You may find here more precise details on your issue.

Have a good day.
sswmooreAuthor Commented:
We have some new information.  There is currently only one DC in this network and now I'm told that there used to be two.  The second domain controller had a bad motherboard and was taken off line and not reconnected.  Isn't there a procedure used to extract an abandoned DC from AD?  I think I've found instructions for that and I'll proceed with that but before I do are there any comments regarding that issue?

Also, in order for AD and a to function properly does the DNS server that AD uses have to be on the DC.?  I don't think it does.  Does it even have to be on the same network.?   is it possible that the DNS servers that our DC uses and that our AD uses is not the DNS server on our DC but some other DNS server, perhaps the  external DNS servers specified as forwarders? If so, how do I tell for sure?  Those DNS servers are operated by anotther agency related to ours and they may have set this up originally.  We've asked them but nobody seems to know anything.  The reason I ask is just the lack of any foreward lookup zones in the DNS manager on the DC. No forward lookup zones, no A records, no records of any kind.

Bruno PACIConnect With a Mentor IT ConsultantCommented:

Having DNS service on the DC is not mandatory but it's easier and more secure because you can use AD integrated DNS zones that use secure dynamic registration.

You can use a DNS service on another Windows server that is not a DC but in this case you have 2 choices :
1) you don't use DNS dynamic registration and you create all DNS records manually (it's a lot of work for necessary AD records for DCs)
2) you accept DNS dynamic registration but it can not be "secured" registration. That means that any machine can register even if it is not a domain member, and any machine can overwrite the DNS record of another machine, which is very very dangerous.

You can even use a non Windows DNS server if it is compliant with some required points (it must support SRV type records). But you'll have the same problems with the two choices explained above.

And if I can give my own strong opinion: KEEP it on your side ! Make things so that you are able to fix things by yourself. The DNS is the absolutely required service for AD. If DNS does not work, AD does not work !
So if something is wrong you should not be in a situation where you must rely on another company. Keep DNS under your managing !

And as you told that persons managing the "forwarders" DNS are not able to tell you if they can host your DNS zone or not, that means they know nothing about AD DNS zones... And that the KEY point ! It's the best reason to not let them manage your AD DNS zone !

You forward lookup DNS zone on your DC has not disappeared by any magic. Probably some bad action had been made to delete it.
If the zone does not exist it will not reappear by itself. But if the zone exist, DNS records should be recreated by themselves.
So, try to recreate an AD integrated forward lookup DNS zone on your DNS server on your DC. The name of the zone must match the DNS name of the domain.
After that, make things so that your DC server interrogate itself as a DNS server in the IP settings.
Reboot the DC and wait for 15 minutes. Take a look at the DNS zone, you should see DNS records for the DC reappear in the DNS zone. You should see a subzone named "_msdcs", etc...

Have a good day.
sswmooreAuthor Commented:
As I think I mentioned before, in the DNS manager, the folder titled "forward lookup zones" is empty.  There is nothing in it.

So if I just right click the forward lookup zones folder and create a new zone and follow the promts to create a new zone with the connrect name and correct settings such as primary zone, store in active directory, replicate to all domain controllers, allow only secure dynamic updates, etc, I should be OK.

I should reboot the DC.  In the NIC's TCPIP settings the IP address of the primary DNS server is the same as the IP address of the DC and when it finishes rebooting I should see the correct DNS records for the DC under Forward Lookup zones. Then as the workstations register, some may register on their own or I can use ipconfig /registerdns,  as they register, I should see their records under forward lookup zones.

As this happens I should then be able to log on to the domain using domain accounts.

Is this an accurate description of how it should play out.  Will I make anything worse by doing this.

By the way I read your article.  It was great. I really enjoyed it.

Bruno PACIConnect With a Mentor IT ConsultantCommented:

Yes this quiet accurate.
But something is missing: Just after you create you forward lookup DNS zone, in the properties of the DNS server you should add the DNS forwarders to point to external DNS server. Doing like this your DNS server is ready to resolve any name.

You configure IP settings on the DC as you said, and then you may try the command NETDIAG /FIX in a CMD prompt to make the DC create a part of its DNS records. This will help it to restart faster.

And then yes you should reboot the DC to let it fully auto-register in DNS and create all the requested records. It may take some long minutes to be back online.

After the DC is fixed, you should fix servers and workstations.

Have a good day.
sswmooreAuthor Commented:
When attempting to create the forwared look up zone, the system pauses for a 30 seconds and then returns the following message....

The zone cannot be created - the data is invalid.

I also have some new information.  As I mentioned earlier,  there used to be two domain controllers on this network.  It was a very old machine and its motherboard died.  Its warranty had expired long ago and a replacement motherboard was assumed to be too hard to find and too expensive.  It was rarely used, there wasn't much of anything on it, so it was decidded that it would be abandoned.

Also the other DC was never removed from the domain.  Isn't there something called a "metadata cleanup" that properly removes an abandoned DC from a domaim?  That was never done.

So with the forward lookup zone gone, I decided to take a look at the domain.dns file in the c:\windows\system32\dns folder.  It's full of DNS infomaion and I've attached it here. One thing that got my attention was that there is one SOA record and it has the name of the other, now abandoned, DC.  Doesn't the SOA record mean that not only was it a DNS server in addition to a DC, but also the primary DNS server for the zone.

Perhaps it was the first DC created in this domain. Perhaps is was the PDC FSMO.

By the way, does the DNS server that Active Directory uses have to be a member of the domain?
sswmooreAuthor Commented:
If I go to Active Directory Users and Computes, right click on the domain and choose operations masters, I get a dialog box with a tab for RID, PDC, and Infastructure.  Each tab specifies the name of the DC.  So from that I presume that the current DC has the PDC FMSO role. It is not the case that the the old DC, the one that died, had the PDC FMSO role.
sswmooreAuthor Commented:
The DNS at first could not be recreated because we discovered that the Kerberos service was disabled.  Once enabled, we did as Tomislavj suggested and forward lookup zone returned. We seized the PDC emulator role and the other four FSMO roles and ran a metadata cleanup. We then did as Pac1B suggested and ran NETDIAG /FIX and added forwarder.  Next we fixed the remaing workstations and servers.  Thanks for your help.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.