Here is the scenario that occured last week at a company I used to work at and now do some freelance work for -- I wasn't there but I am trying to understand what happened and I need some help filling in the gaps.
The company has two servers, both running Windows Server 2003 with the latest SP and updates. One server is the domain controller, a DNS server, backup server, and hosts some office files and the other is used exclusively for SQL Server 2005. They are connected via a simple LAN. There are about 10 workstations running XP Pro on the network, too.
One of the IT guys -- let's call him Bob -- was making some changes to permissions on folders and files on the DC. We think what happened is he somehow set some permissions such that various services could no longer start on the DC. Once that happened, workstations around the office started losing access to the file server and the SQL Server -- it didn't happen all at once. I think, from what I have heard, that this was due to failures when each workstation was trying to authenticate across the domain. It also was no longer possible to use remote desktop to connect to the server. Most Internet access for the office went down (I'm assuming because DNS wasn't working on the server now) but Bob could still 'chat' with me via MSM.
Bob then connected directly to the server and attempted to logon. He could logon without any issue using his domain user and password. Bob is a domain admin. On the DC, domain admins are members of the local admin group. Once logged on to the server, Bob noticed many issues. The 'Manage Your Server' window that he normally sees was open, but blank. He tried to look at events. He could look at the event list but not open the detail screen. He could look at the Services panel but he couldn't open the services properties screens. And so on.
Bob is relatively inexperienced and couldn't get things working again. Bob called me and without remote desktop, I couldn't do much. I suggested a call to paid Microsoft support was in order. They had already been down for 5 hours. They didn't call yet.
One of the manager's called in a friend, Fred, to help out. Fred needed the local administrator password and no one had it. So here is my first question: There were multiple users in the office that had domain admin rights -- Fred claimed that I had somehow deleted the Admin account. Fred spent hours trying to figure out the local admin password.
It ended up that Fred had to make the call to MS Support and they remoted in with Bob logged on and were able to fix the problem via, I think, some registry changes. They had Fred attempt to reset the local admin password via the Active Directory GUI and he wasn't able to ( I don't know why). They then were able to force a reset of the local admin password on reboot -- everything was fine after that.
So, my questions -- and I'm sorry I don't have more details!
Why might have caused the office workstations to lose access to the server gradually? Cached credentials?
What could have caused what Bob saw we he logged on to the server (blank screens, etc)?
Is there anything that would have stopped Fred from using those domain admin credentials to simply reset the local admin password? Further, even without the local administrator password, is there anything that the domain admins couldn't do that the local admin account could? Could there have been some failed service that caused the domain admins to behave differently than the local admins?