Using VMware to Manage a potentially messy break-up

The Background

For the purposes of this article let's call the customer the "ABC Group". The root forest domain was abc.com with two separate trees in the forest, def.com and ghi.com. Abc.com and def.com had approximately 300 users each, with approx 100 in ghi.com. The abc.com domain operated as a separate business to def.com and ghi.com and each of the domains was separate Active Directory Site. The ABC group as a whole had originally commissioned our Company to upgrade the entire infrastructure from Windows Server 2003. When we first started supporting the ABC Group each site had approx 16 physical tower servers with no virtualisation, running Windows Server 2003 throughout. The Windows Infrastructure, from Active Directory to Exchange was all set-up by the book and was in good health. Each site had two Windows Server 2003 Domain Controllers and a front end/back end Exchange server 2003 infrastructure. The remaining servers were member servers running various applications which were servicing only the local domains.

We introduced a server rack with three HP Proliant DL360 G8 hosts and a single HP P2000 SAS SAN at abc.com and def.com, the two larger sites. It had become an urgent requirement to upgrade the Exchange at both locations, which we did, taking it up to Exchange 2010 SP3. The Domain Controllers were updated at abc.com and def.com up to Windows Server 2008 R2, with the Forest and Domain functional levels being upgraded also - all of these new systems were virtualised (with a physical DC and a virtual secondary DC at each upgraded location.
It was in the midst of this upgrade work that a large multi-national swept in and bought the def.com and ghi.com businesses. After some brief meetings we were told that the IT infrastructure had to be split, within the next two weeks.
Having done Cross Forest Domain migrations in the past I was not overjoyed at the prospect of doing another one. Once I began planning it out the more unlikely it was that I was going to hit the deadline. And to be honest, I was quite burned out from all of the upgrade work I had just completed. There must be another way?......

Figure 1.0:- The below logical diagram provides an overview of the "ABC Group" infrastructure. The forest root Domain is located at ABC.com.

The Solution

On talking with a colleague he suggested that I should cut off the abc.com domain from the def.com/ghi.com domains and seize the requisite FSMO roles. How obvious I thought, however the def.com and ghi.com domains would be without the forest FSMO roles (Schema Master and the Domain naming master) which would lead to issues for the def/ghi.com domains later on down the track.

I posted my thoughts on Experts-Exchange, and the great Sembee replied with the seeds of a solution. In short, he suggested taking a copy of the forest root DC (abc.com) at a VM level and restoring it in the def.com domain.....that might just work I thought to myself, so the planning began.

Figure 2.0:- The proposed solution was to copy the existing root DC from the abc.com domain and transfer it to a VLAN on the def.com domain, in order to restore forest root functionality to the remaining domains after the abc,com is split off on its own.

The Solution Detail

After several whiteboards and napkin sketches later I had a rough plan in place. Unlike my current role as a Technical Architect where everything is planned in detail, most of the plan I had for this project resided in my head - I don't recommend this for obvious reasons, but the summary of what I wanted to do was as follows:-

Bring down the VPN between abc.com and def/ghi.com.
Take a V2V to disk of the Primary DC at abc.com, for the purposes of this article lets call it DC1.abc.com.
Drive the DC1.abc.com VM copy over to the def.com offices (approx 100 miles away)
On the def.com domain create new VLAN on the core switch called "TempROOT", add the subnet to this VLAN which is identical to the previously attached abc.com subnet.
Add the "TempROOT" subnet to the site to site VPN so that ghi.com LAN can also route through to it.
Restore DC1.abc.com to "TempROOT" using the VMware vCentre Standalone Converter.
Re-establish DC communications between all sites (TempROOT's abc.com, def.com and ghi.com)
Clear all references to def.com and ghi.com from the abc.com domain
Fix Exchange - Correct Exchange email flow to allow it to work correctly between abc.com and the def/ghi.com domains as before.
Troubleshoot and re-mediate any outstanding issues.
Final task list

I ran through the steps above and everything went as planned. Re-establishing communications between the Domain Controllers was not as straightforward as I had first hoped. However using the following commands, as well as confirming general network connectivity was healthy, saw it restored to full health:

Run Command line as an Administrator:-

Repadmin /Syncnow

Repadmin /Showrepl

Figure 3.0:- With the Domain Controller replication restored on the def.com and ghi.com domains the separated infrastructure was back at full health. An additional Domain Controller was then created on the mock abc.com domain for redundancy purposes.

I should mention at this point that one of the pre-requisites of the project was that abc.com must be able to email def.com and ghi.com and vice versa once the split was completed.

- Checkpoint -

The ABC Group was now split. The abc.com domain believed that the def.com and ghi.com domains had gone offline. Aside from emailing users in those domains everything was working as expected. The def.com and ghi.com domains believed everything was working as expected, however emailing abc.com was failing.

The remaining tasks:-

Fix Exchange
Troubleshoot and re-mediate any outstanding issues.
Final task list

Fixing the Exchange

In order to repair the Exchange email for abc.com to the now seperate domains was straightforward, in that as long as any remaining evidence of def.com and ghi.com were removed from the abc.com Domain Controllers then outbound email should work correctly. This would also resolve all of the replication errors being caused by the absent DC's.

In summary the places to clean up items were:-

NTDSUTIL (To clean up the Site, Domain and DC items
Check Active Directory Sites and Services to confirm all sites, IP ranges and replication partnerships are remove.
Review DNS, using the management console and remove any reference to the domains.

There were maybe some additional areas searched and tools run in order to iron out the last few Event log errors - but unfortunately I did not note these.

Now email was working out of ABC.com to def/ghi.com. To get mail flow back the other way I removed all of the user accounts and contacts from the migrated dc1.abc.com. I also removed all of the address books that contained an abc.com user reference - this was the key to restoring mail flow to abc.com, as without any reference to the users on the abc.com domain then it forced mail flow out.

Final tasks

Everything was working as expected and both Companies were happy that they were fully split. I ran through some additional tasks which came to mind as the project unfolded.

I created a secondary DC for the abc.com domain on the "TempROOT" VLAN just to ensure that this mock forest root had availability.
I added the two TempROOT DC's to the backup set as a matter of best practice.
The external DNS hosting had to be split according to the external Domains that would remain with each Company. This was obviously not a quick fix but I started the ball rolling and this completed within the month.

Summary

So thanks to virtualisation I was able to split the forest in approx three days, with little or no disruption to the end users. This could have been done with physical servers, but there would have been a lot more lifting of heavy objects and likely a lot more waiting on restoring of hardware images.

To be clear this is not supposed to replace a cross forest migration, however it might serve as a useful solution to someone landed with the same time constrained project - who needs a temporary fix.

Using VMware to Manage a potentially messy break-up

Comments (2)