Disaster Recovery test - Question about domain controllers

Posted on 2013-01-16
Last Modified: 2013-04-01
Hi all,
In a couple of weeks we'll be testing our disaster recovery plan, and I have a question about domain controllers

First some info on how we're setup:
-3 sites, each as 2 DC's in it, one domain across all 3 sites.
-The 3 sites linked with VPN tunnels
-IP's in the office are setup so one office uses 10.1.x.x, another 10.2.x.x, another 10.3.x.x
-Each office has Equallogic SAN being used in iSCSI mode, connecting to vsphere
-The SAN's replicate nightly. Site 1 and 3 replicate to site#2, site#2 replicates to site#3.
-Each office has a 3 server vSphere essentials plus farm
-All three offices have virtual servers doing file/print, WSUS, antivirus, and a few others
-The main site, lets call it site#1, also hosts a Citrix farm and all our main shared apps like microsoft dynamics and others.

So our plain is that we'll basically just unplug the firewall in site#1 on a saturday morning, simulating that office burning down (to the outside world). We'll update public DNS entries of things we want to test to point to site#2. In site#2, we'll then promote the replicated volumes on the Equallogics to full volumes (in test mode so we can revert back without changing those when we're done). On the firewall in Site#2 we'll remove reference to site#1 in the VPN settings so it's not trying to route to the VPN for site#1's internal IP's anymore.

Then we'll connect those Equallogic volumes to the vSphere environment in site#2 and connect the VM's. I've already created vSwitches for the subnets in use (and the vLAN's on the procurve switches), which we'll connect the VM's to.

update firewall in site#3 so vpn looking for site#1 now points to site#2. The firewall in site#2 already has all the NAT rules and the like setup, just disabled until needed.

At that point, I think we'll be ready to fire up the servers. The main things we want to test is to see if our core apps come up, and if staff can get at them from Citrix and basically have no idea we are running from a different site. All their shared drives should be present, and other than physical things like printing or phones, be able to do just about any of their normal work.

I have a couple questions though.
1)Anything glaring that I am missing?
2)For domain controllers, should we start the virtual DC's from site#1 when we do this, or just let the machines from site#1 connect to site#2's DC's. I am concerned if I do start up the DC's from site#1, then when we shut down the test, the "real" DC's back in site#1 will be out of sync.
Question by:GreenEnvy

Accepted Solution

Harsem earned 500 total points
ID: 38798158

sounds like a pretty good plan.

In regards to 2) I would not worry, as your test is over within 1 -2 days. What you are doing (in essence) is to restore a domain controller from backup that is 1 or 2 days out of synch. Microsoft has a default value of 60 days for which a Domain Controller can be disconnected from the network. So 1 or 2 days would not be an issue.

To check that value above please go to:
to check what this value is for your AD Forest.


Author Comment

ID: 39037604
As a followup, this test went well.

We shutdown the firewall in site #1, leaving all the servers running there but inaccessible to the outside world.

In site#2, we then promoted the Equallogic replicated volumes to full volumes. We imported the machines we wanted to test into our vSphere.
I had to manually go put some IP's into our Procurve switches in Site#2 so they would know how to route the traffic for the servers from Site#1. Also modified our watchguards to not try to route traffic for site#1's ip ranges over the VPN.

Turned on the servers from site#1, other than the DC's from Site#1. We just let them connect to the site#2 DC's. Had to go manually change the DNS server IP's for the servers from site#1 (though we could have also just added a second virtual NIC to the site#2 DC's with the corresponding IP, but didn't want to mess with the "production" equipment).

Those servers got connectivity right away. We had changed the DNS for our citrix servers the night before, so we tested that from external computers and it was up and running right away. Tested our file servers, SQL, finance apps, all OK.

All in all, it only took a couple hours to failover. We had 2 staff go out to site#2 for this, but next time we test, or in a real situation, we can do it remotely and it should take less than an hour. Most of the time was going through the 30 or so servers and updating DNS.

Since then I've actually done this on a smaller scale one time when our primary backups failed one night and a user desperately needed a folder back that had been created the day before. Did this type of failover for that file server only (from the replication of the volumes) and got the folder back.

Featured Post

Gigs: Get Your Project Delivered by an Expert

Select from freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Backup VM'S 5 91
fm200 2 81
Server Battery Backup -- redundant power supply ? 7 79
Backup and restore to dissimilar hardware 7 86
Backing up data is essential for any office small or large. Most think that a simple USB drive will suffice. Even the USB drives themselves display words like backup.   Most novices will ask themselves the question “Will this work for my business…
First I will try to share a design of a Veeam Backup Infrastructure without Direct NFS Access backup. Note: Direct NFS Access backup transport mechanism is only available in Veeam v9 In above I try to design the Veeam Backup flow between i…
This tutorial will walk an individual through locating and launching the BEUtility application to properly change the service account username and\or password in situation where it may be necessary or where the password has been inadvertently change…
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…

815 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now