Server (Domain Controller) Unresponsive - Have to HARD REBOOT Server to Fix Lockups

Problem:  COMPLETE NETWORK LOCKUP:  
    All workstations freeze;
    Cannot authenticate into any network device until reset;  
    Server (DC) has to be hard powered down to restart.  
    Hard reset on server fixes problem

Notes:

1. Domain Controller is 2003 R2 X64 SP2 with all critical and security patches applied

2. No events are written to log files when lockups occur

3. Server IP (and other devices) can be pinged during lockup

4. Login screen can be accessed on all other servers but if user name and password is entered, the system "locks" up.  (authentication failing)

5. Disk arrays are healthy

6. Network traffic normal during lockup

HELP!
sfjcpuAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mike KlineCommented:
Is this your only DC?

Thanks

Mike
0
sfjcpuAuthor Commented:
No.  There are 2 others.
0
Mike KlineCommented:
Odd, do the clients have the IP of the other DCs in their DNS.  Does this box that freezes hold all the FSMO roles.

Are the other DCs also global catalog servers?
0
Creating Active Directory Users from a Text File

If your organization has a need to mass-create AD user accounts, watch this video to see how its done without the need for scripting or other unnecessary complexities.

sfjcpuAuthor Commented:
Yes the clients use one of the other DCs as their secondary DNS. The server holds all FSMO roles except for infrastructure. The server that holds the infrastructure roll is not a global catalog server.
0
StuWhitbyCommented:
Something on the DC is still responding to a heartbeat, so workstations keep trying to access that one.  When you say you have to hard reboot.... is that because a soft shutdown hangs or because you can't start it due to entire UI hang?  The reason I ask is that running up Process Monitor and leaving it on the DC may show you what's going on at the time of the hang.

How many CPU cores in the DC?  If there's only one, it may simply be that one thread has gone crazy, resulting in an unresponsive system.  If so, run up Process Explorer and increase the window manager and Process Explorer's priority to Realtime.  This will now take precedence over a single thread, but may have a slight performance impact on the system.  Once the system hangs, you should now be able to look and see which process is eating CPU, examine the threads in that process and see what they're doing.  

Both tools available from http://technet.microsoft.com/sysinternals.  Set up Process Explorer with the symbol path "srv*c:\symbols*http://msdl.microsoft.com/download/symbols" under Options/Configure Symbols.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Leon FesterSenior Solutions ArchitectCommented:
If you're not seeing anything written to the windows logs, then it generally points to a hardware issue.
Now I've never come across a scenario, where 1 DC out 3 fails and the other two can't still authenticate users.

Have you run a AD health check, are all DC's healthy?
Run a DCDIAG to test.

To me "lockup" means that the keyboard/mouse/screen is non-responsive.
Is that what you mean when you say "lockup", or what exactly does it mean to you?
0
StuWhitbyCommented:
I disagres that "nothing written to the windows logs [generally] equals hardware issue".  This is regularly that it's so busy doing something that requilres no logging that nothing gets written.  It's rare with a hang that anything gets logged as all it's actually doing is waiting for something.  It is entirely possible that this is an intermittent issue with the disk though, where networking is sitll working fine but the disk can't be accessed to write event logs or read to authenticate users.

The other systems don't authenticate users because the primary is still responding on the network, so things are working for this system.  

To check the disk, try setting up ftp on the system disk and see if you can access data that way.  rpcinfo -a should also give you further information as regards what services this system is seeing as available behind the networking (not exhaustive, but proves that the OS itself is answering basic commands).
0
sfjcpuAuthor Commented:
DC's are healthy.  There is only one DNS entry on workstations.  We will add another DNS entry.  Of course that will not fix the problem but it needs to be done.  Thanks for that advice.  I downloaded Sysinternals (nice set of tools).   Process monitor and process explorer are running on the DC this morning.  I'm logged on server and am keeping the logon open so I can see screen.   Client will contact immediately if the DC becomes unresponsive.  I'll post the results.
0
sfjcpuAuthor Commented:
Still having lockups.  We suspect a profile issue coming from a Windows2000 Terminal Server login.  The DC usually hangs about the same time some users are logging into the old TermServer.  The log files grow too quickly and too large to be practical in this diagnostic.  Any ideas on whether or not a profile could hang up the DC?
0
StuWhitbyCommented:
If Process Monitor logging is too large to be usable and you notice this issue as soon as it occurs, set up a scheduled job to stop and start process monitor and "interact with desktop" on an hourly basis.  When it hangs, get to the DC, wait until it's responsive again, stop the capture, save the log (pml format) and stop the scheduled job.  Then go back through the log to a point during which the system was unresponsive and try to figure out what led up to that hang.
0
sfjcpuAuthor Commented:
Stu, Thanks for the post.  The hangups are so random, days in between, it would be difficult to do that.  When it hangs, my client restarts the server right away since they have all their profiles and shares set up on this server.  

If I could write a script to save the log files for 10 minutes, then stop ProcMon, save the log file to another file name, then start it again.  Maybe keep 10 of these files and have the script cycle through the names, such as ProcMonLog1, ProcMonLog2, ProcMonLog3...ProcMonLog10.

Would this work, and if so, do I need to post a separate question on how to write the script?

Thanks!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2003

From novice to tech pro — start learning today.