We help IT Professionals succeed at work.

Hoist the colors! We’ve added location flags to usernames sitewide, so it's easier to connect with the global community on EE. View My Profile

x

Server hangs / locks up and won't boot with network cable plugged in.

3,615 Views
Last Modified: 2012-05-07
Our client is running a Dell PowerEdge 2900 with a PERC 6i RAID controller and Windows Server 2003 Small Business.  It's running Active Directory, Exchange, DNS, and DHCP.  Software installed is Syspro, and Viper Antivirus.  No changes to the server have been made within the past couple of weeks other than installing the Antivirus and changing the Administrator password (which has since been changed back).  In case it may be relevant, my first step I took was to reserve the UDP ports for IPsec and Viper Antivirus in the registery due to an error in the event log, and because the server would not get online.

The problem started with the server rebooting itself a couple of times and would hang at "Applying Computer Settings" until it would eventually lock up (meaning the mouse and keyboard are non-responsive and a hard reboot is required).  Safe mode works all the time, with networking works sometimes, and normal boot works fine if I unplug the network cable.  It takes a VERY long time to boot when booting in normal mode.  I also found that if I turn off the DNS Server service that the system boots into normal mode fine.  If at any point I turn DNS Server back on, after about half an hour, the system will go unresponsive with no HDD activity and a hard reboot is required.  At this point there aren't any key events in the Event Log but there are a few misc events regarding a general problem with Active Directory.  

What do you recommend as a next step in troubleshooting?  This problem is urgent as our client is totally down at this point and we've been troubleshooting for 2 days now.  I'm offering 500 points to whoever resolves the issue!
Comment
Watch Question

Ady FootSharePoint Consultant

Commented:
When you turn the DNS server off and the system seems to boot fine, are you able to logon to the system at all?  Are you able to logon to any network workstations using a domain account?  

Regards,

Ady

Commented:
Is the server the only domain controller and DNS server on the network?  Are there any log entries regarding the DNS server in the event log?  

Can you post the results of the following commands:

ipconfig /all

and

nslookup server.yourdomain.com


Ady FootSharePoint Consultant

Commented:
My thoughts exactly -DJL- hence my question to the author.  Seems to me like there might be another DNS server somewhere or that the DNS server on the SBS machine is forwarding all queries elsewhere.

Author

Commented:
This server is the only DNS server and the only DC on the network.   The system seems fully functional after it boots into normal mode and I start DNS manually.  I'm then able to log on locally and all users are able to log on via the network and access all of their shares and mail.  I haven't seen any entries in the DNS event log.

I'm not physically at the machine now so I'll get you the ipconfig and nslookup shortly.

Thanks for the quick replies!

Commented:
It sounds like either;

The network adapter settings are incorrect - the primary DNS entry should be the servers IP or 127.0.0.1, secondary should be blank

The antivirus is blocking/causing problems with traffic on port 53 TCP/UDP

Some of the entries in the DNS zone could be corrupt, or the zone isn't loading correctly.  

You could try running NETDIAG and DCDIAG available in the 2003 support tools: http://support.microsoft.com/kb/892777 

Author

Commented:
I've now taken the server to our office and it's doing the same thing so I don't believe it's due to any rogue DNS server somewhere.  The Antivirus has been disabled completely for troubleshooting purposes and the problem persists.  This problem happened out of nowhere and the network adapter settings have been quadruple checked.  I have reinstalled DNS and no luck so I'll try to just delete the zone and create a new one to see what happens.

I've neverrun NETDIAG or DCDIAG before and I see there are a lot of options.  What should I run for these circumstances?
Ady FootSharePoint Consultant

Commented:
Please post the result of netdiag and dcdiag here so that I can have a look.

Author

Commented:
Results for IPCONFIG and NSLOOKUP:

C:\Documents and Settings\Administrator>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : fs3
   Primary Dns Suffix  . . . . . . . : ellison.local
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : ellison.local

Ethernet adapter Local Area Connection 2:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Broadcom BCM5708C NetXtreme II GigE (NDIS
 VBD Client) #2
   Physical Address. . . . . . . . . : 00-1E-C9-DB-EE-37
   DHCP Enabled. . . . . . . . . . . : No
   IP Address. . . . . . . . . . . . : 192.168.99.118
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.99.1
   DNS Servers . . . . . . . . . . . : 127.0.0.1

C:\Documents and Settings\Administrator>nslookup fs3.ellison.local
Server:  localhost
Address:  127.0.0.1

Name:    fs3.ellison.local
Address:  192.168.99.118

Author

Commented:
Results for NETDIAG and DCDIAG:

C:\Documents and Settings\Administrator>dcdiag

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\FS3
      Starting test: Connectivity
         ......................... FS3 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\FS3
      Starting test: Replications
         ......................... FS3 passed test Replications
      Starting test: NCSecDesc
         ......................... FS3 passed test NCSecDesc
      Starting test: NetLogons
         ......................... FS3 passed test NetLogons
      Starting test: Advertising
         Warning: FS3 is not advertising as a time server.
         ......................... FS3 failed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... FS3 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... FS3 passed test RidManager
      Starting test: MachineAccount
         ......................... FS3 passed test MachineAccount
      Starting test: Services
            IsmServ Service is stopped on [FS3]
            w32time Service is stopped on [FS3]
         ......................... FS3 failed test Services
      Starting test: ObjectsReplicated
         ......................... FS3 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... FS3 passed test frssysvol
      Starting test: frsevent
         ......................... FS3 passed test frsevent
      Starting test: kccevent
         ......................... FS3 passed test kccevent
      Starting test: systemlog
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168F
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic deletion of the DNS record
         An Error Event occured.  EventID: 0x0000168F
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic deletion of the DNS record
         An Error Event occured.  EventID: 0x0000168F
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic deletion of the DNS record
         An Error Event occured.  EventID: 0x0000168F
            Time Generated: 10/10/2009   14:41:14
            Event String: The dynamic deletion of the DNS record
         An Error Event occured.  EventID: 0x0000168E
            Time Generated: 10/10/2009   14:43:49
            Event String: The dynamic registration of the DNS record
         An Error Event occured.  EventID: 0x0000168F
            Time Generated: 10/10/2009   14:43:49
            Event String: The dynamic deletion of the DNS record
         An Error Event occured.  EventID: 0x0000168F
            Time Generated: 10/10/2009   14:43:49
            Event String: The dynamic deletion of the DNS record
         An Error Event occured.  EventID: 0x0000168F
            Time Generated: 10/10/2009   14:43:49
            Event String: The dynamic deletion of the DNS record
         An Error Event occured.  EventID: 0x0000168F
            Time Generated: 10/10/2009   14:43:49
            Event String: The dynamic deletion of the DNS record
         An Error Event occured.  EventID: 0xC0001B70
            Time Generated: 10/10/2009   14:44:14
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x80001778
            Time Generated: 10/10/2009   14:47:13
            Event String: The previous system shutdown at 2:44:04 PM on
         An Error Event occured.  EventID: 0x80001778
            Time Generated: 10/10/2009   14:52:44
            Event String: The previous system shutdown at 2:47:13 PM on
         An Error Event occured.  EventID: 0xC0002715
            Time Generated: 10/10/2009   14:53:32
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002715
            Time Generated: 10/10/2009   14:53:45
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002715
            Time Generated: 10/10/2009   14:54:31
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC00007DD
            Time Generated: 10/10/2009   14:57:24
            Event String: SMTP could not connect to any DNS server. Either
         An Error Event occured.  EventID: 0xC1010022
            Time Generated: 10/10/2009   14:58:31
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC1010022
            Time Generated: 10/10/2009   14:58:31
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC101003A
            Time Generated: 10/10/2009   14:58:31
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC101003B
            Time Generated: 10/10/2009   14:58:31
            Event String: Generate Activation Context failed for
         An Error Event occured.  EventID: 0xC1010022
            Time Generated: 10/10/2009   14:58:31
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC1010022
            Time Generated: 10/10/2009   14:58:31
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC101003A
            Time Generated: 10/10/2009   14:58:31
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC101003B
            Time Generated: 10/10/2009   14:58:31
            Event String: Generate Activation Context failed for
         An Error Event occured.  EventID: 0xC1010022
            Time Generated: 10/10/2009   14:58:33
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC1010022
            Time Generated: 10/10/2009   14:58:33
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC101003A
            Time Generated: 10/10/2009   14:58:33
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC101003B
            Time Generated: 10/10/2009   14:58:33
            Event String: Generate Activation Context failed for
         An Error Event occured.  EventID: 0xC0001B7A
            Time Generated: 10/10/2009   15:00:37
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC000271A
            Time Generated: 10/10/2009   15:00:40
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0001B7A
            Time Generated: 10/10/2009   15:00:47
            (Event String could not be retrieved)
         ......................... FS3 failed test systemlog
      Starting test: VerifyReferences
         ......................... FS3 passed test VerifyReferences

   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom

   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom

   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom

   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom

   Running partition tests on : ellison
      Starting test: CrossRefValidation
         ......................... ellison passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ellison passed test CheckSDRefDom

   Running enterprise tests on : ellison.local
      Starting test: Intersite
         ......................... ellison.local passed test Intersite
      Starting test: FsmoCheck
         Warning: DcGetDcName(TIME_SERVER) call failed, error 1355
         A Time Server could not be located.
         The server holding the PDC role is down.
         Warning: DcGetDcName(GOOD_TIME_SERVER_PREFERRED) call failed, error 135
5
         A Good Time Server could not be located.
         ......................... ellison.local failed test FsmoCheck



C:\Documents and Settings\Administrator>netdiag

....................................

    Computer Name: FS3
    DNS Host Name: fs3.ellison.local
    System info : Microsoft Windows Server 2003 (Build 3790)
    Processor : x86 Family 6 Model 23 Stepping 6, GenuineIntel
    List of installed hotfixes :
        KB923561
        KB924667-v2
        KB925398_WMP64
        KB925902
        KB927891
        KB929123
        KB930178
        KB931784
        KB932168
        KB933729
        KB933854
        KB935839
        KB935840
        KB936021
        KB936357
        KB936782
        KB938127
        KB938127-IE7
        KB938464
        KB941569
        KB941693
        KB942830
        KB942831
        KB943055
        KB943460
        KB943485
        KB944338-v2
        KB944653
        KB945553
        KB946026
        KB948496
        KB948590
        KB948745
        KB949014
        KB950762
        KB950974
        KB951066
        KB951072-v2
        KB951698
        KB951746
        KB951748
        KB952004
        KB952069
        KB952954
        KB953838
        KB953838-IE7
        KB953839
        KB954211
        KB954550-v5
        KB954600
        KB955069
        KB955839
        KB956391
        KB956572
        KB956802
        KB956803
        KB956841
        KB956844
        KB957097
        KB958215-IE7
        KB958469
        KB958644
        KB958687
        KB959426
        KB960225
        KB960714-IE7
        KB960803
        KB960859
        KB961063
        KB961371-v2
        KB961501
        KB967715
        KB967723
        KB968537
        KB968816
        KB969805
        KB969883
        KB970238
        KB970483
        KB970653-v3
        KB971032
        KB971557
        KB971633
        KB971657
        KB971961
        KB972260-IE7
        KB973346
        KB973354
        KB973507
        KB973540
        KB973815
        KB973869
        Q147222


Netcard queries test . . . . . . . : Passed



Per interface results:

    Adapter : Local Area Connection 2

        Netcard queries test . . . : Passed

        Host Name. . . . . . . . . : fs3
        IP Address . . . . . . . . : 192.168.99.118
        Subnet Mask. . . . . . . . : 255.255.255.0
        Default Gateway. . . . . . : 192.168.99.1
        Dns Servers. . . . . . . . : 127.0.0.1


        AutoConfiguration results. . . . . . : Passed

        Default gateway test . . . : Failed
            No gateway reachable for this adapter.

        NetBT name test. . . . . . : Passed
            No remote names have been found.

        WINS service test. . . . . : Skipped
            There are no WINS servers configured for this interface.


Global results:


Domain membership test . . . . . . : Passed


NetBT transports test. . . . . . . : Passed
    List of NetBt transports currently configured:
        NetBT_Tcpip_{BDCF456C-EDC7-4208-ABB1-6A167A1D3A2F}
    1 NetBt transport currently configured.


Autonet address test . . . . . . . : Passed


IP loopback ping test. . . . . . . : Passed


Default gateway test . . . . . . . : Failed

    [FATAL] NO GATEWAYS ARE REACHABLE.
    You have no connectivity to other network segments.
    If you configured the IP protocol manually then
    you need to add at least one valid gateway.


NetBT name test. . . . . . . . . . : Passed


Winsock test . . . . . . . . . . . : Passed


DNS test . . . . . . . . . . . . . : Passed
    PASS - All the DNS entries for DC are registered on DNS server '127.0.0.1'.


Redir and Browser test . . . . . . : Passed
    List of NetBt transports currently bound to the Redir
        NetBT_Tcpip_{BDCF456C-EDC7-4208-ABB1-6A167A1D3A2F}
    The redir is bound to 1 NetBt transport.

    List of NetBt transports currently bound to the browser
        NetBT_Tcpip_{BDCF456C-EDC7-4208-ABB1-6A167A1D3A2F}
    The browser is bound to 1 NetBt transport.


DC discovery test. . . . . . . . . : Passed


DC list test . . . . . . . . . . . : Passed


Trust relationship test. . . . . . : Skipped


Kerberos test. . . . . . . . . . . : Passed


LDAP test. . . . . . . . . . . . . : Passed


Bindings test. . . . . . . . . . . : Passed


WAN configuration test . . . . . . : Skipped
    No active remote access connections.


Modem diagnostics test . . . . . . : Passed

IP Security test . . . . . . . . . : Skipped

    Note: run "netsh ipsec dynamic show /?" for more detailed information


The command completed successfully
SharePoint Consultant
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
Well I ended up recreating the forward lookup zone half an hour ago but the settings I selected match what you wanted me to verify.  Shouldn't the forward lookup zone have propagated all of the information that's in AD already?  Or would the zone fill only once it's connected back to the network?  The only record that's in DNS now is the server itself.   I also received a DNS Event ID: 4521 which says:

The DNS server encountered error 32 attempting to load zone ellison.local from Active Directory. The DNS server will attempt to load this zone again on the next timeout cycle. This can be caused by high Active Directory load and may be a transient condition.

Good news is that it hasn't locked up yet when it previously would have...
Ady FootSharePoint Consultant

Commented:
If you deleted the zone it will take some time to recreate properly once the server is connected back to the network and the workstations 'talk to the server' once again.  The workstations will re-register themsevles when you turn them on with the server back in place.

Regards,

Ady

Author

Commented:
The server ran for a couple of hours but it locked itself up again.
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
My problem in thinking that is that the server only locks up / reboots itself when the DNS server service is running.  Otherwise I would agree that it seems like a hardware problem.  I've run all Dell diagnostics and looked through Open Manage and don't see anything.  There is one error in Open manage about the Perc 6i firmware stating:

Controller event log: Fatal firmware error: Line 3622 in ../../raid/1078dma.c : Controller 0 (PERC 6/i Integrated)

This only came up once in there so I didn't think much of it.

I ran through Windows Updates and there are no available updates for any of the hardware.  All software updates available were optional.  

Unexpected Shutdown event:
The reason supplied by user ELLISON\Administrator for the last unexpected shutdown of this computer is: Other Failure: System Unresponsive
 Reason Code: 0x8000005
 Bug ID:
 Bugcheck String:
 Comment:

Author

Commented:
These answers led me in the right direction but didn't resolve the problem directly.

Commented:
lccjlt

Did you managed to sort the problem with the firmware? I too have the same issue and it looks like that hard drive firmware is problematic.

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.