We help IT Professionals succeed at work.

Protecting an AD Environment

clebo99 asked
Within the past week, our environment suffered a MAJOR Active Directory SNAFU.  Someone within our Domain (which holds probably 20,000 users/systems) deleted a somewhat "high-up" OU, which in turn wiped out several hundred important user and workstation accounts.  Luckily, our organization (which as a "root OU" within the tree structure) was pretty insulated by the deletion.  The department that lost the OU had to have all of their accounts re-created manually which took over 2 days.  People were obviously fired because of this and we are still trying to estimate the outage costs.  

My boss is now asking me how I can protect our environment from suffering from the same fate.  There are several skilled engineers in my group that have very elevated rights within our OU structure and one slip of a mouse could potentially delete 2,500 users.  

A little about our environment.  We have over 2,500 users spread across the country in 20 or so sites.  Each site has (2) AD controllers and are in the same OU structure.  We do NOT manage our DC's.  This is done by another group.  There are probably 200+ production servers throughout our enterprise.  

So, my question is, are there any white papers or system guides that we could use as a start for creating our data protection guide?  Is there a backup solution that we could implement that would protect our environment OU and if so, what are the steps/best practices for restoration?  Should we develop a script that can "scrape" the AD user/system/group information into some sort batch file that can be run to restore the environment?  

Has anyone been involved in something similiar and can give me some tips on how they recovered?  
Watch Question

I am hugely supprised for such scale company not having any DR plan. Accidently removing OU can be completely restored if backup exists. Manually recreating users is obviously the most costly "solution" I've ever seen.

To protect an environment is more a polictical issue than a technical issue, even both are equally important.
 - keep minimum number of domain admins
 - use proper delegation for OUs. If you don't trust someone, ethic-wise or technical wise, than don't give him permission
 - enable certain auditing

DR is a broad topic. For AD, I recommend the Operation Guide as start point.

Specifically to recover from accidentaly delete,
 1. A backup is ALWAYS A MUST! I can't emphasize this too much.
 2. Always test if your backup can actually be restored.
 3. Create a dedicate AD site that is intentionally configured replicate behind the rest of domain, e.g. 2 days, or one week, that way you always have a ample time to perform authoritative restore from DCs in recovery site.
Below is certainly not the best of article of this idea on the internet, but it gives you a basic concept. There are other issues to consider, such as preventing the recovery site from regstering certain DNS records, from providing authentication, etc.
Top Expert 2006
that is very shocking to see such a large org with no backup! AD backup and restore is one of the simplest things to complete!

For some AD guides

And Lee has a great guide here on backup options


Thanks Guys.........I'm not at liberty to say where I work (unless I wanted to start collecting unemployment).  Our section has our OWN DR plan.  The problem is that we do not manage the main infrastructure (DC's, certain routers, etc) so we are at the mercy of the "controlling department".  

We also have probably 60-65 DC's within the US infrastructure.  We argued for months to have our own forest but were outvoted by the higher ups.  We quoted this EXACT issue as why we wanted our own.  

Finally, apparently a backup was restored but the information was not being replicated back out to all the DC's (or if it were, it was causing an "AD Update Storm" that completely knocked out some of the sites.  That may be why they had to create the accounts manually....I'm investigating.

Thanks again guys......
Top Expert 2006

that sounds like a good fun environment to be working in :) Good luck mate, i am envious at the moment!