DC failure Windows Server 2008 R2

Both machines mentioned are VMs (xenserver)

I have a dc server that has failed. (Raid failure and corrupted VM)  It was also a DNS server and had DFRS namespace function.

There is a second server that has all the roles and is a global catalog.

I do have a vm image (3-4 days old) of the failed server.  
1 - Would there be any benefit to boot the image and run dcpromo to remove it from the domain?  Or would there be AD contamination?
2 - I can live without the DFRS for a few days
3 - If I do not do dcpromo on the image, how do I make sure that the server is removed from the domain?
4 - What about DNS?

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Larry Struckmeyer MVPCommented:
Consider what you would do if these machines were physical.  Adsiedit would be required to remove the failed server.  So you could go that route.  But if the VM's images are only 3/4 days old could you not replace the failed drives and restore the images?

Consider what would happen if the failed DC were simply accidently turned off for the weekend.  On the next working day, when restarted it would replicate with the live DC's in the domain, updating and exchanging information until they were in sync.  I don't believe restoring a 3/4 day old image will cause any harm.

In the case of DNS, it seems to me you will want to have a DNS server on your network.  You say this one was "a dns server".  If there are no others, you can add the role to any other server in the domain either permanently or temp.
dustypenguinAuthor Commented:
Can I indeed, just use the Active Directory Users and Computers snap in and remove the failed server from the Domain Controllers list?
dustypenguinAuthor Commented:
Thanks for the reply fl_flyfishing, yes there is another DNS server on the network.

Having lived through a domain crash, I am hesitant to just restore and go if there is a chance that it would inject bad data into the AD.  Especially in light of wanting the next couple of days off!
Get Blueprints for Increased Customer Retention

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

Larry Struckmeyer MVPCommented:
No ADUC will not clean up the other DC's.  ADSIedit is required.  You should be able to find the steps in the MS KB.  As far as the other DC's are concerned, this DC has simply been turned off.  ADUC is for removing AFTER the DC has been DCPromo'd out of the AD.
dustypenguinAuthor Commented:
I am restoring the image ... we'll give that a go and monitor the event viewer.  I would have done that earlier (last night), except for my bad experience about 5 years ago.  Ugly ugly ugly.
Will SzymkowskiSenior Solution ArchitectCommented:
The appropriate way to cleanup AD is below...
- Use ntdsutil (perform a metadata cleanup) to remove the broken server
- DO NOT bring up the old imaged server that was from 3-4 days ago
*as soon as you bring this server back up in the environment it is going to start replicating and authenticating users. The USN will be out of sync due to this failure.
- Once you have remove cleaned up the server using ntdsutil, you will need to go into ADUC delete the computer account
- Open Sites and Services and remove the computer object from there
- Open DNS Manager and under _msdcs folder (SRV records) you will need to remove all of the records that were associated with your DC that you are removing (kereros/ldap/gc/etc)
- You will also want to change your DNS server settings on your DHCP client scopes so that they no longer point to this DNS server

Once you have completely removed all of the objects from your environment you can bring up a new server and promote this server as a domain controller and it will replicate it's data from healthy DC.

Below is a link on how to use NTDSUTIL to remove cleanup metadata and also removing the object from AD. http://technet.microsoft.com/en-us/library/cc816907(v=ws.10).aspx

dustypenguinAuthor Commented:
Thanks for the reply, Will.  I think I agree with you.

Here is the part I do not understand.  I did not have to seize any roles, since the roles already resided on the other DC.

What is my next step directly after that?

Your link says "Metadata cleanup is a required procedure after a forced removal of Active Directory Domain Services"  I am not sure I have done that part ("forced removal") yet ....
dustypenguinAuthor Commented:

In the text of the link it says "Expand the domain of the domain controller that was forcibly removed ... "

That's the part I am unsure of.  Do I have to do an action to forcibly remove the DC or is the failure itself the removal?

Will SzymkowskiSenior Solution ArchitectCommented:
If the FSMO role holder is dead you need to seize the role and perform metadata cleanup. Not your issue.

If you have a BDC (backup DC) that has failed you still need to perform the metadata cleanup and remove this DC from Active Directory and Sites and Services. As stated as well you need to update your DHCP scopes DNS server settings. SRV records are also required.

Basically all you need to do from NTDSUTIL is do metadata cleanup and then proceed with the removal of computer objects for that account.

dustypenguinAuthor Commented:
Ok, good.

From what I see from the link provided, I can do both things from Active Directory Users and Computers, and I do not need to use NTDSUTIL, as metadata will be cleaned up as well through the GUI.  

("When you use Remote Server Administration Tools (RSAT) or the Active Directory Users and Computers console (Dsa.msc) that is included with Windows Server 2008 or Windows Server 2008 R2 to delete a domain controller computer account from the Domain Controllers organizational unit (OU), the cleanup of server metadata is performed automatically. Previously, you had to perform a separate metadata cleanup procedure.")

Would you agree with that assessment?
Will SzymkowskiSenior Solution ArchitectCommented:
This is correct. I prefer to do it from NTDSUTIL (personal preference). However if you choose to do it from the GUI it would be a good idea to ensure that it is removed by using ntdsutil which can verify this.

You still need to cleanup sites and services and most importantly is SRV records in DNS.


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
dustypenguinAuthor Commented:
Ok, thanks ...
Leon FesterSenior Solutions ArchitectCommented:
Restoring a VM image of a failed domain controller is not a support recovery option by Microsoft or VM. Both companies recommend not doing that.

You are on the right track about removing the old DC. You don't need to have it running to delete it.

It is a failed DC so you can safely just delete the computer account for the DC from ADUC.
Using the GUI is the preferred method on AD2008.

Once you've deleted the computer account you should then format the drive and re-install Windows and then add the Directory Services.
dustypenguinAuthor Commented:
This article ( https://blogs.technet.com/b/askds/archive/2009/06/05/dc-s-and-vm-s-avoiding-the-do-over.aspx?Redirected=true ) was also very helpful in understanding why this was the right answer.

While fl_flyfishing's answer was correct for a physical machine that had died, the issue with a VM is that between the time of the VM backup, and the time the machine had died, there may have been (and likely were) USN updates that would have made the VM backup out of sync with the rest of the DCs on the network.  Since the failed machine had accepted some new USNs since the backup, it can not just get back to the level of the other machines since they would see a conflict there in trying to resubmit USNs that had already been accepted.

Thanks for each of your participation.
Larry Struckmeyer MVPCommented:
Lesson Learned.  Thanks everyone.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2008

From novice to tech pro — start learning today.