Disaster Recovery Scenario - Rebuild Active Directory / Exchange

Hello --

I'm working on implementing a DR plan in the case of total company loss of hardware. In this case, we would have off site backups that would have company data. I the event this should happen, we would order hardware and build an offsite location to restore network services.

I have built a test lab and begun the restore of an Active Directory server. In normal cases, it would be easy to just bring another server up and replicate to it, however in this case we have nothing but backup tapes.

I have installed Server 2003 Enterprise (The same version on the old DC) and fully patched it. We use backup exec for our updates. I renamed the server to be the same as the original, same IP, everything. I installed the Backup Exec agent and then booted into Directory Services Restore Mode and did a restore of the System State, the C:\WINNT\NTDS files, and sysvol files. I marked in the backup for it to mark these as the primary, so it is essentially setting the burflag to D4. Once I boot the machine back up, I get the following error --

Directory Services could not start because of the following error: The specified network password is not correct. Error Status: 0xc000006a. Please click OK to shutdown this system and reboot into Directory Services Restore Mode, check the event log for more detailed information.

All accounts and passwords are the same as the original DC. I am really at a loss at to what is going on here. I haven't been able to find any good documentation for this scenario, only restoring in an environment where 1 dc is still present.

Please let me know if you need any further clarification. Thanks!

Once I get AD working, I will be also restoring exchange and restoring various files on the file server just to test that as well. One thing at a time though...
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Bruno PACIIT ConsultantCommented:

No no no... In case of total loss you can not only restore the AD database on a new server that have the same name and expect it to work.

You have to restore the whole server with its System State.
The server is not the most important. The most important is the server SID and as you can not decide to use a specific SID during server install you must restore it via the system state...

What you have to do for the first restored server is:

1) Install Server 2003 entreprise on a new server. Don't waste your time patching it except if this is mandatory to make your backup exec software to work. No need to give the same name because the name will be restored with system state.
2) Using your backup exec software, restore the whole server, files AND system state... That means that you must have backed up your server with all files and System State.
3) Restart your server in Active Directory Restore Mode.
4) Use NTDSUTIL to force the AD database as "Authoritative".
5) Restart your server normally.

That's all... The most complicated thing is to deal with different hardware on the new server. That can be a challenge if the new server use an IDE or S-ATA disk and the dead server was using SCSI disk...

By the way. You must never restore a backup older than 60 days. That means that you must backup up your whole server at least every 60 days.

Now, to restore other DCs you have 2 choices :
a) Use the same steps by restoring whole servers but don't play the NTDSUTIL step because you only need it on the first restored server.
b) Use NTDSUTIL on you first server to force removal of other DCs and resinstall all other DCs by a standard way (from CD, patch, join domain, DC promo).

As I said, the biggest problem is to restore a whole server on a different hardware. Virtualization technologies help a lot for that. What about having an ESXi server on your LAN and create at least one DC in a VM ?
This is really simpler cause ESXi is easy to install on almost all constructors servers and all VMware VMs see the same standard virtual hardware.

Another thing. I've not enumerated all the steps you need restoring AD servers... I did not talked about seizing FSMO roles and transfer them. You might have to include these steps depending of the restore scenario you finally choose.

Have a good day.

RyrenAuthor Commented:

I will give your method a try first thing tomorrow and comment back on how it goes. We are currently doing full backups every week of the DCs and rotating tapes, so oldest I have is 30 days, no worries about tombstone.

I like the virtualization idea, as I've been leaning that way for this, and the actual DR plan will be built around a virtual environment. For now, this test environment is a physical lab here and this is just what I have to use for the time being.

I will download the drivers for this server tomorrow as well to be ready to get everything working. I assume if drivers are too out of whack, I can just run a repair on server and then install drivers?

Thanks for your quick response!! I'm glad I have a good direction to head in the morning.
Bruno PACIIT ConsultantCommented:
Hi again,

Thinking a lot more about that I realized that disk drivers probably won't be a problem because as you must start installing a minimal Windows OS before restoring you'll install the disk drivers. And as the restore won't delete these driver files all should be ok.

We're waiting feedback from you about you success restoring AD to start talking Exchange restore.

Have a good day
Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

Look into DoubleTake Cloud, it replicates your data to the Amazon computer services cloud and then it gives you the option to bring virtual machines online with your data in the Amazon cloud until your hardware is rebuilt, then you can recovery with your new hardware.
RyrenAuthor Commented:
DeweyS, I will definitely give this a good look as we're going to explore multiple options once this physical DR plan has been developed.

PaciB, Awesome -- I will let you know tomorrow morning how the restore goes. I'm switching from a differential to full backup tonight so I have fresh data to work with tomorrow. Talk to you then.
Also look at Acronis for "bare metal" recovery.  It is an expensive, but good product that will allow you to restore a server image to a different hardware platform.  We've been using it for over a year and have actually recovered a server two different times and it went pretty well.  There may be other cheaper or better products - not trying to sell it, it just works well for us.
RyrenAuthor Commented:
Well, just got done with a new restore and same error as above.

Directory Services could not start because of the following error: The specified network password is not correct. Error Status: 0xc000006a. Please click OK to shutdown this system and reboot into Directory Services Restore Mode, check the event log for more detailed information.

Here's what I did. I rebuilt the system, in order to do the restore of the old system, I renamed it to the same as the DC which I Have a backup of, and gave it the same IP. I was having issues restoring because this system wasn't on the domain, and the previous was. So I went ahead and dcpromo'd the system and gave it the exact same settings as the current one. Once it was on the domain, I installed the Backup Exec client and pushed the full restore back to it. This included ALL system files, System State, and ShadowCopy Components. This was successful. Oh, and to restore the System State, it said the system had to be in Directory Services Restore Mode already. In your instructions you hadn't hit that step yet, but I went ahead, booted into that mode and then did the restore. Once finished, rebooted and went right back into Directory Services Restore Mode.

At this point I went ahead and did the ntdsutil commands as follows.

authoritative restore
restore database
confirmed, and let it do its thing.

Once this was done, I rebooted into normal mode, and bam, there was the ugly error all over again....

Any ideas what is snagging this process? From your description, seems like you've gotten this to work before.
Bruno PACIIT ConsultantCommented:

Let me remind you what I said:

1) Install Server 2003 entreprise on a new server. Don't waste your time patching it except if this is mandatory to make your backup exec software to work. No need to give the same name because the name will be restored with system state.
2) Using your backup exec software, restore the whole server, files AND system state... That means that you must have backed up your server with all files and System State.
3) Restart your server in Active Directory Restore Mode.
4) Use NTDSUTIL to force the AD database as "Authoritative".
5) Restart your server normally.

I've never told you to put the new server into the domain before restoring !! You only have to install a basic Windows OS just to be able to restore the whole system. Install Windows, don't rename it, don't insert it into domain, DON'T make DCPROMO ! JUST RESTORE THE WHOLE THING including system state.

No problem about AD restore mode because there's no AD database on the reinstalled server when your start the restore.

What you have done is that renaming the server and addind it to the domain you have erased the old computer account and changed it by another with a new SID. After that you restore the system that get back its old SID and BAM ! OF course...

Now, you can not retry again because you have polluted your AD with bad SID.

You have to start from scratch your tests... and do only the steps that are in the list.

Good luck
RyrenAuthor Commented:
I'll try to figure out a different way to restore then.. it wasn't wanting to do a complete restore to original directories because it was a different machine. That's why I went ahead and just mirrored the old settings again.

I'll give this another whirl. I'll get back to you.
RyrenAuthor Commented:
OK. This time this was a restore to the machine with a different name and different IP than the original. I did a restore of the original machine using file redirection which I tested was still putting the documents in the same place by restoring a file a couple of levels deep. Everything went where it was supposed to. So I moved on to the full restore.

Rebooted, went into Directory Services Restore Mode and ran ntdsutil.. here are the results

authoritative restore
restore database
confirm 'Yes' to restore
Opening DIT database...
Could not initialize the Jet engine: Jet Warning 1
Authoritative Restore Failed
Error 8000ffff parsing input - illegal syntax?

I then tried to reboot the machine normally to see if anything would happen, it didn't even seem like anything changed. No domain was available to log into, logged into the original admin account on that box and nothing different. Looking around the C:\ drive the files are all there, but just seems it didn't boot into them.

Any ideas?
Bruno PACIIT ConsultantCommented:

Can you verify if NTDS.DIT file is present on the server after restoring ?
Did you restore the system state also ?
Dependign of backup/restore software you might have to restore in two steps : One for System State alone, and another for disk files...
What version of BackupExec do you use ?

Have a good day.
Bruno PACIIT ConsultantCommented:
By the way...

Which failover scenario are you acutelly testing ?

Total crash of all DCs of your domain, or only crash of one DC of your domain ?

If the last case (loss of one DC only), you should skip the NTDSUTIL steps because you don't need an authoritative restore. The authoritative restore is only usefull if your want to restore a corrupted/deleted AD content or when you restore the first DC in a total failure scenario (loss of all DCs).

Anyway, the problem your have with your restored DC has no link with authoritative restore or not.
RyrenAuthor Commented:
Thanks for getting back to me.

NTDS.DIT is present after the restore.

I am using the newest version of backup exec, BE 2010.

So if I do it in two steps, what would be the process for restoring System State, then Data?

I am testing a scenario where we lost all hardware (significant event) and I have to rebuild the network from scratch with only backup tapes available.
Bruno PACIIT ConsultantCommented:

I found this article: http://seer.entsupport.symantec.com/docs/236286.htm
It's a bit old but things shouldn't have changed a lot...

The article says that because of BackupExec design the target server must have the same name than the crashed server.
So you have to install a basic Windows, give it the same name as the original server BUT not join the domain and NOT do DCPROMO: only a standalone server having the same netbios name than the crashed server to allow BE to restore on.

This is specific to BE because I already have restored DCs without having to rename the target server, but I usually use NTBackup or some other backup/restore softwares...

So you have to retry, doing like you did except you have to rename the target server to match the failed server.

Good Luck
RyrenAuthor Commented:
I think I just found the problem. Other than renaming the server so it's the same as the backup, the Windows directory is different. The original DC is C:\WINNT while the new build is C:\Windows. Problematic.
RyrenAuthor Commented:
I found this which may help get me around this issue... http://support.microsoft.com/kb/811944

Looks like I'll need to reinstall Server 2003 again and point it to the C:\Winnt folder, then delete the old C:\Windows folder.. I assume I'll have to do something to the boot.ini as well.

I'll give this a shot unless you have an alternate suggestion.
Bruno PACIIT ConsultantCommented:

Yes the target server must use the same system folder. You need to choose C:\winnt when installing the basic OS before restoring on it.

But if your original servers don't all use the same system folder (I mean if some of them are installed in C:\WINNT and other in C:\Windows) it will be very hard to have an efficent restore procedure because after the server has crashed it's too late to see if it was in C:\WINNT or C:\Windows...
So you have to write these parameters somewhere to use at restore time...

When you'll have to restore it will probably be a urgent task in a stressing environment. It's not the good time to look for documentation about how the original servers has configured...
So you should make things so that all your servers are installed in the standard Windows 2003 folder C:\Windows, instead of C:\WINNT which is probably a sequel of an upgrade from Windows 2000 or Windows NT.

Have a good day.
RyrenAuthor Commented:
Well, not making too much headway. I followed the instructions exactly, and no luck.

Here were the steps taken --

Install Windows Server 2003
Reinstall Windows Server 2003 so you can rename install directory to C:\WINNT
Delete C:\Windows
Update server to current patching levels as domain controller (BE wouldn't function without certain updates)
Installed agent
Pushed System Files, System State, Shadow Copy, rebooted
Came back to normal, non-domain screen.

Logged in, all files are there, nothing. Still functioning as a normal workgroup machine.
RyrenAuthor Commented:
I want to clarify, the system was named the same as the current domain controller, IP was the same as well.. did not run dcpromo.
Bruno PACIIT ConsultantCommented:

If the system state has really been restored you should find some typical services on your server like "NTDS", "Key distribution Center", ...
Can you check if these services appears in the registry after the restore ?
For that, look in HKLM\System\CurrentControlSet\Services and look for subkey NTDS, NTFRS, KDC, ...
Are these subkeys present in registry ?
RyrenAuthor Commented:
I do not see "NTDS" or "Key distribution Center" running in the services snap-in.

As for Registry, I do see kdc and Ntfrs folders under the path given above.

So it seems that a full restore wasn't done. But all files, System State, and Shadow Copy Components were set to be restored and the job was successful without any errors. Also looking in C:\WINNT\SYSVOL the folder is empty.

Backup Exec actually says when restoring system state that if the restore contains Active Directory and the target is a domain controller, it must be in Directory Restore mode. I know you said this wasn't necessary and you were able to complete a restore without this option, I just want to be sure.

Is there something I should try differently?
RyrenAuthor Commented:
I also meant to state I did not see the NTDS folder in the registry. Just the kdc and Ntfrs.
Bruno PACIIT ConsultantCommented:

I don't really know specific procedures for BackupExec... I found this article: http://seer.entsupport.symantec.com/docs/283648.htm
Does this article means something to you and have you done what is written in it ?

RyrenAuthor Commented:
Not really - that is for Continuous Protection Server, which I'm not running. I instead just do Full Backups every day.

You said you aren't familiar with Backup Exec -- Which platform did you use prior to do the backup jobs / restore data that worked for you? Are you still currently using this system?

Depending on what method you use, I may look into additionally backing up Exchange and AD with the method that has worked for you in case we do need to do a restore, being this is becoming an issue.

I look forward to hearing your response.
Bruno PACIIT ConsultantCommented:
By the way,

I just verified on some servers and NTFRS and KDC services registry subkeys are always present in registry even if the server is not a domain controller. These services are present bu disabled.

The NTDS service is not present until the server is promoted as a domain controller.
So, the fact that NTDS registry subkey is missing on your server after restoring prooves that the system state has not been restored by BackupExec, or the backup used was not taken from a domain controller...

Is there any way in BE to confirm that NTDS.DIT file is present in the backup set ?

Another thing: usually when using a backup software the system state of a computer can be deployed and show sub-components: registry, SAM, etc...
On a domain controller you should see a sub-component named like "Active Directory database" or something like that...

Ensure that all the subcomponents are included at the time you backup your source domain controller.
RyrenAuthor Commented:
I'll check into that stuff now. I just noticed that the SYSVOL folder is actually empty on the backup, proving problematic. I'm going to run a single backup in a minute to just backup that folder and make sure it works. If that works, then I'm not sure why the full backup is missing the contents of this folder.

When you get a chance, please look at my question about what you're using and what has worked in the past for this process? I may just go that route.
Bruno PACIIT ConsultantCommented:
When I write a backup/restore scenario for my cutomers I usually do it for basic tools like NTBackup.
This is because the steps are always the same and I just have to "instanciate" (not sure of english term) these procedures with my customer to be usable with his backup software.

Frequently here my customers use TiNa (Time Navigator) which is a french developed backup software and might be unknown anywhere else than Europe...

Also, the more and more customers now uses virtual machines as the "kernel" of disaster recovery scenarios. This eliminate the problem of hardware and permit them to backup the whole machine as if you were doing a disk image of a server...

With NTBackup here are the requirements for backup :

Complete Backup the whole C drive (I suppose you made a standard installation so the C drive is where the boot files and %system% folder are installed)
Backup the System State (there is no incremental or differential backup of System state, it's alwaays a complete backup).

For my part, I prefer to make two separate backup jobs: one for C drive and one for System State because NTBackup features are limited and it's easier to have separate backup sets when you want to restore only files or only System State.

Here are the steps for restoring with NTBackup:

1) Install a basic Windows 2003 OS standalone server (No need to install update, no need to use the same name, no join domain).
3) Configure IP settings on the NIC to match the settings of the crashed server. Verify network connectivity by making some pings.
2) Install the tape drive drivers, or any drivers that is needed to reach the backup sets.
3) Restore the whole C drive with "overwrite files" option, do not restart yet.
4) Restore the System State, do not restart yet.
5) unplug the network wire to isolate the server from other domain controllers and from any client computer.
6) Restart the server. Let it start normally to check if Active Directory services start well. You can do it as you are isolated from network. I suppose here that the restored DC is also its own DNS server, else without a reachable DNS server AD services won't start correctly but even in this case you can go on because this error is expected.
7) Restart the server in Active Directory Restore Mode.
8) Launch NTDSUTIL and choose authoritative restore for the whole domain (you should see a message that says the version number of each AD object has been incremented by 100000).
9) Plug the network wire.
10) Restart in normal mode...

Of course I supposed here that the %system% folder path is the same on the original crashed server and on the reinstalled basis target server... I never had to try with different pathes...

NTBackup is a basic tools but these steps always worked well until now... It takes around 1 hour to get a DC working, depending of the time it takes to install the basic target OS (usually the windows basic installation takes 40 minutes).
You can divide this time by 3 if the recovery server is already presintalled with basic Windows OS.

About BackupExec procedure, you should open a new question on EE specifically for BackupExec... something like "How to restore the first DC on new harware with BE after a total crash", because as this question already contains lot of comments other EExperts probably won't take a look at it...

Good luck

Bruno PACIIT ConsultantCommented:
Oh... another thing....
Step 11) If there's no chance to restore other lost DCs after restoring the first one, or if you prefer reinstall completly the other DCs, you then need to seize all the FSMO roles on the restored DC before installing and promoting any new DC in your restored forest. This is made with NTDSUTIL but your don't need to be in AD Restore Mode this time.
RyrenAuthor Commented:
PaciB --

Thanks again for all of your help! By tweaking Backup exec and the environment I was restoring in, I was able to successfully restore a domain controller to a test lab environment.

Previously you had mentioned once we got the DC restored you'd explain how to restore Exchange in this same kind of scenario. Now that AD is working, I need to introduce exchange and move on the to the second step of this question. I assume just rebuild a system, install exchange 2003 and all of the latest updates and go from there?
Bruno PACIIT ConsultantCommented:

Restoring a definitly lost Exchange server is a specific procedure.
Briefly, the steps are the following:
1) reinstall a new Windows server. Try to respect disk partitions as they were on the lost server.
2) give this server the same name as the lost Exchange server and make it a member of the domain.
3) patch it as it was before the failure: if it was a Windows 2003 SP1 then install SP1, if it was a Windows 2003 SP2 then install SP2, etc..
4) install the prerequisites for Exchange: IIS with ASP .NET, NNTP, SMTP and WWW services.
5) to reinstall Exchange, you have to use the /DisasterRecovery switch in the SETUP.EXE command. You can not just reinstall a brand new Exchange Organization because there's already one in your AD forest so you MUST use the /DisasterRecovery switch.
6) When Exchange has been reinstalled and Exchange services are running, you can see that your Exchange Oragnization configuration has been restored (administrative groups, routing groups, conectors, etc...) but the datastores are missing.
7) you then have to restore a full backup of your Exchange datastores.

This article will help you in this procedure : http://www.petri.co.il/exchange_disasterecovery_switch.htm

HAve a good day

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
What abot disk space and partitioning for the ad restore. Looking to the exact same thing.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Active Directory

From novice to tech pro — start learning today.