Domain Administrator vs Local Admin Issues on Windows Server 2003

Here is the scenario that occured last week at a company I used to work at and now do some freelance work for -- I wasn't there but I am trying to understand what happened and I need some help filling in the gaps.  

The company has two servers, both running Windows Server 2003 with the latest SP and updates.  One server is the domain controller, a DNS server, backup server, and hosts some office files and the other is used exclusively for SQL Server 2005.  They are connected via a simple LAN.  There are about 10 workstations running XP Pro on the network, too.

One of the IT guys -- let's call him Bob -- was making some changes to permissions on folders and files on the DC.  We think what happened is he somehow set some permissions such that various services could no longer start on the DC.  Once that happened, workstations around the office started losing access to the file server and the SQL Server -- it didn't happen all at once.  I think, from what I have heard, that this was due to failures when each workstation was trying to authenticate across the domain.  It also was no longer possible to use remote desktop to connect to the server.  Most Internet access for the office went down (I'm assuming because DNS wasn't working on the server now) but Bob could still 'chat' with me via MSM.

Bob then connected directly to the server and attempted to logon.  He could logon without any issue using his domain user and password.  Bob is a domain admin.  On the DC, domain admins are members of the local admin group.  Once logged on to the server, Bob noticed many issues.  The 'Manage Your Server' window that he normally sees was open, but blank.  He tried to look at events.  He could look at the event list but not open the detail screen.  He could look at the Services panel but he couldn't open the services properties screens.  And so on.

Bob is relatively inexperienced and couldn't get things working again.  Bob called me and without remote desktop, I couldn't do much.  I suggested a call to paid Microsoft support was in order.  They had already been down for 5 hours.  They didn't call yet.

One of the manager's called in a friend, Fred, to help out.  Fred needed the local administrator password and no one had it.  So here is my first question:  There were multiple users in the office that had domain admin rights -- Fred claimed that I had somehow deleted the Admin account.  Fred spent hours trying to figure out the local admin password.

It ended up that Fred had to make the call to MS Support and they remoted in with Bob logged on and were able to fix the problem via, I think, some registry changes.  They had Fred attempt to reset the local admin password via the Active Directory GUI and he wasn't able to ( I don't know why).  They then were able to force a reset of the local admin password on reboot -- everything was fine after that.

So, my questions -- and I'm sorry I don't have more details!  

Why might have caused the office workstations to lose access to the server gradually?  Cached credentials?  

What could have caused what Bob saw we he logged on to the server (blank screens, etc)?  

Is there anything that would have stopped Fred from using those domain admin credentials to simply reset the local admin password?  Further, even without the local administrator password, is there anything that the domain admins couldn't do that the local admin account could?  Could there have been some failed service that caused the domain admins to behave differently than the local admins?

Any help?

Who is Participating?
I don't know of any specifically.  However, I'm sure that someone could come up with one.  The beauty of being a Domain Administrator (or a local administrator on a given machine) is that you can deny yourself access to resources, such as a file, but then take ownership of the file and give access back to yourself.

With regard to the local administrator password on a  domain controller: this is used only for the recovery console and directory services restore mode.  You can not log on locally to a DC except using these methods.  You can not change these passwords through Active Directory Users and Computers because they are not stored there, they are stored on the local domain controller.  You can not change these passwords through Local Users and Groups because you can not access Local Users and Groups on a DC.  The two methods for changing the Directory Services Restore Mode password (for 2000 and then for 2003) are linked below.

My guess is this.  Bob really screwed things up with permissions.  Fred wanted to log into the recovery console, but couldn't because no one knew the password.  Either that or Fred didn't realize that they only ways to log into a DC locally is through the recovery console or DSRM.  Bob and Fred couldn't change the recovery console password because they didn't know the methods below.  Microsoft came in, got them logged into the console, walked them through fixing the permissions and/or replacing files that Bob mucked with, and then everything was hunky-dorey.

Anyway, it's all speculation.  The only way to know what happened for sure is to ask Fred and Bob and trust that they are truthful.  
My guess is that he changed some permissions on the WINDOWS folder.  That may explain why some of these utilities opened but didn't display everything.

The local administrator on a DC is only good for logging into the recovery console or for directory services restore mode.  There is a method for resetting this password however it isn't as easy as going into computer management and doing so as you cannot access Local Users and Groups on a DC.

If the netlogon service was stopped or had a problem, then users who logged off and then tried to log on would not be able to do so.
jedwar26Author Commented:
Is there a scenario where any of the various methods for resetting the local admin password on a DC would fail because of the permissions issues discussed above?  
jedwar26Author Commented:
Granted, this answer involves speculation but the Export's comments fit the scenario and further research I have done on the actual events that transpired.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.