Link to home
Start Free TrialLog in
Avatar of ravenrx7
ravenrx7

asked on

DC down after power outage

This past friday our company lost power and even though our 2003 Server was on an APC, after the power came on, of systems came up and this 1 of the 2 DCs can't connect to the network. I checked the NIC, the port it's plugged into, both are active.  


I did see some DNS errors which might be my issue. Any ideas?
Avatar of blakogre
blakogre
Flag of United States of America image

When you say "can't connect to the network" -- can it ping the other dc?  Can other clients ping both DCs?  Are the times on each system the same/correct?

Avatar of ravenrx7
ravenrx7

ASKER

I can not ping...but it has. Second nice and configured that..same results
Avatar of Miguel Angel Perez Muñoz
Network connections service is UP?
Check event logs/system logs for error messages and report anything unusual/that started after the power issue, here.

Right click and disable the nic, and re-enable.

well a day before this happened i was prepping my servers for exchange 2010 install and all the servers had  to moved to 2003 mode in AD, now i have a some AD errors
The dunamic dletion of the DNS record
forestDNSZones.XXXXX.org 600 in A 169.254.254.70 failed on the following DNS server
DNS IP < UnKnown>
Check if your server has not entered restricted ipsec mode and blocked everything

http://support.microsoft.com/kb/912023/en-us
How u do that?
this cant be good, i just ran a dcdiag on that DC and got an error like that exe isnt installed
can you post the DCDIAG error ?
can not find "dcdiag"
Go into Win 2003 server I386 directory, sort by size and execute adminpack.msi

This will install DCDIAG

But before,, you must be sure your server is not rejecting packets. Open lcoal area connection infos and check if sent/received packets are not stuck to 0

start event viewer (start : run : eventvwr) and look for ID 4292 source IpSec errors
the server is receiving packets and sending,loading adminpak now
Server Connected today or still refusing to talk with other servers ?
well Ive ran into a problem Ive told the admins about, which is low disk space, so now I had no space to install the adminpak. Ive ordered Acronis disk manager for server to allocate some space. I hate that its got to get to this point before doing anything, so right now im dead in the water till that software key is sent
Before going deeper into problems, make a few space by removing temp datas and using disk cleanup tools.

It also might sound obvious but I suggest you make a full backup before you expand your partition.

Regards,

Gérald
correct, it would be great if this is resolved just from disk space issues, but i dont think it is. Im reading up on the dcdiag tool, looks like that will at least give me a lead
I faced the same problem long time ago
it tuned out to be.
.
.
.
.
.
Check time on both DC
If the time is different, they wont communicate
time is good,
wow after using acronis to free up some place, i log in and install adminpak and it wont let me install because my documents folder is redirected and the server is offline so i cant reach it. I guess admin pak installs there/./goodness!
Admin pack installs itself in c:\program files\support tools
But you can remove the My Document redirection if it blocks your install.

By the way, support tools will help you diagnose AD problem, but if you still can not ping any other DC, computers or device, it won't help you much. There is a lower problem to fix first.

Can you ping your main gateway from this server ?
ok i got adminpak to install.. but ... still dcdiag cant be found error
Got the DcDiag Results ran on the DC (waps-dc3)that is down..waps-ex is the other DC
F:\>dcdiag.exe

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: WAPS\WAPS-DC3
      Starting test: Connectivity
         ......................... WAPS-DC3 passed test Connectivity

Doing primary tests

   Testing server: WAPS\WAPS-DC3
      Starting test: Replications
         [Replications Check,WAPS-DC3] A recent replication attempt failed:
            From WAPS-EX to WAPS-DC3
            Naming Context: DC=ForestDnsZones,DC=campus,DC=XXXXX,
DC=org
            The replication generated an error (1256):
            The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
            The failure occurred at 2011-10-05 11:49:07.
            The last success occurred at 2011-10-05 11:34:36.
            1 failures have occurred since the last success.
         [WAPS-EX] DsBindWithSpnEx() failed with error 1722,
         The RPC server is unavailable..
         [Replications Check,WAPS-DC3] A recent replication attempt failed:
            From WAPS-EX to WAPS-DC3
            Naming Context: DC=DomainDnsZones,DC=campus,DC=XXXXDC=org
            The replication generated an error (1256):
            The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
            The failure occurred at 2011-10-05 11:49:07.
            The last success occurred at 2011-10-05 11:34:36.
            1 failures have occurred since the last success.
         [Replications Check,WAPS-DC3] A recent replication attempt failed:
            From WAPS-EX to WAPS-DC3
            Naming Context: CN=Schema,CN=Configuration,DC=campus,DC=XXX,DC=org
            The replication generated an error (1722):
            The RPC server is unavailable.
            The failure occurred at 2011-10-05 11:49:49.
            The last success occurred at 2011-10-05 11:34:36.
            1 failures have occurred since the last success.
            The source remains down. Please check the machine.
         [Replications Check,WAPS-DC3] A recent replication attempt failed:
            From WAPS-EX to WAPS-DC3
            Naming Context: CN=Configuration,DC=campus,DC=XXXXX,D
C=org
            The replication generated an error (1722):
            The RPC server is unavailable.
            The failure occurred at 2011-10-05 11:49:28.
            The last success occurred at 2011-10-05 11:34:35.
            1 failures have occurred since the last success.
            The source remains down. Please check the machine.
         [Replications Check,WAPS-DC3] A recent replication attempt failed:
            From WAPS-EX to WAPS-DC3
            Naming Context: DC=campus,DC=XXXXX,DC=org
            The replication generated an error (1722):
            The RPC server is unavailable.
            The failure occurred at 2011-10-05 11:49:07.
            The last success occurred at 2011-10-05 11:34:35.
            1 failures have occurred since the last success.
            The source remains down. Please check the machine.
         ......................... WAPS-DC3 passed test Replications
      Starting test: NCSecDesc
         ......................... WAPS-DC3 passed test NCSecDesc
      Starting test: NetLogons
         ......................... WAPS-DC3 passed test NetLogons
      Starting test: Advertising
         Warning: WAPS-DC3 is not advertising as a time server.
         ......................... WAPS-DC3 failed test Advertising
      Starting test: KnowsOfRoleHolders
         Warning: WAPS-EX is the Schema Owner, but is not responding to DS RPC B
ind.
         [WAPS-EX] LDAP search failed with error 58,
         The specified server cannot perform the requested operation..
         Warning: WAPS-EX is the Schema Owner, but is not responding to LDAP Bin
d.
         Warning: WAPS-EX is the Domain Owner, but is not responding to DS RPC B
ind.
         Warning: WAPS-EX is the Domain Owner, but is not responding to LDAP Bin
d.
         Warning: WAPS-EX is the PDC Owner, but is not responding to DS RPC Bind
.
         Warning: WAPS-EX is the PDC Owner, but is not responding to LDAP Bind.
         Warning: WAPS-EX is the Rid Owner, but is not responding to DS RPC Bind
.
         Warning: WAPS-EX is the Rid Owner, but is not responding to LDAP Bind.
         Warning: WAPS-EX is the Infrastructure Update Owner, but is not respond
ing to DS RPC Bind.
         Warning: WAPS-EX is the Infrastructure Update Owner, but is not respond
ing to LDAP Bind.
         ......................... WAPS-DC3 failed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... WAPS-DC3 failed test RidManager
      Starting test: MachineAccount
         ......................... WAPS-DC3 passed test MachineAccount
      Starting test: Services
         ......................... WAPS-DC3 passed test Services
      Starting test: ObjectsReplicated
         ......................... WAPS-DC3 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... WAPS-DC3 passed test frssysvol
      Starting test: frsevent
         Message 0xc0001830 not found.
         ......................... WAPS-DC3 failed test frsevent
      Starting test: kccevent
         ......................... WAPS-DC3 passed test kccevent
      Starting test: systemlog
         An Error Event occured.  EventID: 0xC00010C4
            Time Generated: 10/05/2011   11:34:43
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC25A001D
            Time Generated: 10/05/2011   11:38:29
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC25A001D
            Time Generated: 10/05/2011   11:57:13
            (Event String could not be retrieved)
         ......................... WAPS-DC3 failed test systemlog
      Starting test: VerifyReferences
         ......................... WAPS-DC3 passed test VerifyReferences

   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom

   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom

   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom

   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom

   Running partition tests on : campus
      Starting test: CrossRefValidation
         ......................... campus passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... campus passed test CheckSDRefDom

   Running enterprise tests on : XXXXX.org
      Starting test: Intersite
         ......................... XXXXX passed test I
ntersite
      Starting test: FsmoCheck
         Warning: DcGetDcName(PDC_REQUIRED) call failed, error 1355
         A Primary Domain Controller could not be located.
         The server holding the PDC role is down.
         Warning: DcGetDcName(TIME_SERVER) call failed, error 1355
         A Time Server could not be located.
         The server holding the PDC role is down.
         Warning: DcGetDcName(GOOD_TIME_SERVER_PREFERRED) call failed, error 135
5
         A Good Time Server could not be located.
         ......................... XXXXX failed test F
smoCheck

F:\>




























Netdiag Log.. from WAPS-Dc3 ( Down DC)




    Computer Name: WAPS-DC3
    DNS Host Name: waps-dc3.XXXX.org
    System info : Microsoft Windows Server 2003 (Build 3790)
    Processor : x86 Family 6 Model 23 Stepping 10, GenuineIntel
    List of installed hotfixes :
        KB2079403
        KB2115168
        KB2121546
        KB2124261
        KB2141007
        KB2160329
        KB2183461-IE8
        KB2207559
        KB2229593
        KB2259922
        KB2286198
        KB2296011
        KB2296199
        KB2345886
        KB2347290
        KB2360937
        KB2378111
        KB2387149
        KB2393802
        KB2412687
        KB2416400-IE8
        KB2416451
        KB2419635
        KB2423089
        KB2436673
        KB2440591
        KB2443105
        KB2443685
        KB2467659
        KB2476490
        KB2476687
        KB2478953
        KB2478960
        KB2478971
        KB2479628
        KB2482017-IE8
        KB2483185
        KB2485376
        KB2485663
        KB2497640-IE8
        KB2503658
        KB2503665
        KB2506212
        KB2506223
        KB2507618
        KB2507938
        KB2508272
        KB2508429
        KB2509553
        KB2510531-IE8
        KB2511455
        KB2524375
        KB2530548-IE8
        KB2535512
        KB2536276
        KB2536276-v2
        KB2544521-IE8
        KB2544893
        KB2555917
        KB2559049-IE8
        KB2562485
        KB2562937
        KB2566454
        KB2567680
        KB2570222
        KB2570791
        KB923561
        KB924667-v2
        KB925398_WMP64
        KB925902-v2
        KB926122
        KB927891
        KB929123
        KB930178
        KB932168
        KB933729
        KB933854
        KB935839
        KB935840
        KB936357
        KB936782
        KB938127
        KB938464
        KB941569
        KB942830
        KB942831
        KB943055
        KB943460
        KB943485
        KB944338-v2
        KB944653
        KB945553
        KB946026
        KB948496
        KB949014
        KB950760
        KB950762
        KB950974
        KB951066
        KB951698
        KB951746
        KB951748
        KB952004
        KB952069
        KB952954
        KB953298
        KB954155
        KB954211
        KB954550-v5
        KB954600
        KB955069
        KB955759
        KB955839
        KB956391
        KB956572
        KB956802
        KB956803
        KB956841
        KB956844
        KB957097
        KB958215
        KB958469
        KB958644
        KB958687
        KB958690
        KB958869
        KB959426
        KB960225
        KB960714
        KB960715
        KB960803
        KB960859
        KB961063
        KB961118
        KB961371
        KB961371-v2
        KB961373
        KB961501
        KB963027
        KB967715
        KB967723
        KB968389
        KB968537
        KB968816
        KB969059
        KB969805
        KB969897
        KB969897-IE8
        KB969898
        KB969947
        KB970238
        KB970483
        KB970653-v3
        KB971029
        KB971032
        KB971468
        KB971486
        KB971557
        KB971633
        KB971657
        KB971737
        KB971961-IE8
        KB972260-IE8
        KB972270
        KB972636-IE8
        KB973037
        KB973346
        KB973354
        KB973507
        KB973525
        KB973540
        KB973687
        KB973815
        KB973825
        KB973869
        KB973904
        KB973917
        KB973917-v2
        KB974112
        KB974318
        KB974392
        KB974455-IE8
        KB974571
        KB975025
        KB975467
        KB975558_WM8
        KB975560
        KB975562
        KB975713
        KB976098-v2
        KB976325-IE8
        KB976662-IE8
        KB976749-IE8
        KB977290
        KB977816
        KB977914
        KB978037
        KB978207-IE8
        KB978338
        KB978542
        KB978601
        KB978695
        KB978706
        KB979309
        KB979482
        KB979559
        KB979683
        KB979687
        KB979907
        KB980195
        KB980218
        KB980232
        KB980436
        KB981322
        KB981332-IE8
        KB981550
        KB981793
        KB982132
        KB982214
        KB982381-IE8
        KB982666
        Q147222


Netcard queries test . . . . . . . : Passed
    [WARNING] The net card 'Broadcom BCM5708C NetXtreme II GigE (NDIS VBD Client) #2' may not be

working.



Per interface results:

    Adapter : Local Area Connection

        Netcard queries test . . . : Passed

        Host Name. . . . . . . . . : waps-dc3
        IP Address . . . . . . . . : 10.102.1.3
        Subnet Mask. . . . . . . . : 255.255.0.0
        Default Gateway. . . . . . : 10.102.1.1
        Dns Servers. . . . . . . . : 10.102.4.18
                                     10.102.1.3


        AutoConfiguration results. . . . . . : Passed

        Default gateway test . . . : Failed
            No gateway reachable for this adapter.

        NetBT name test. . . . . . : Passed
        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messenger Service', <20> 'WINS'

names is missing.
            No remote names have been found.

        WINS service test. . . . . : Skipped
            There are no WINS servers configured for this interface.

    Adapter : Local Area Connection 2

        Netcard queries test . . . : Failed
        NetCard Status:          DISCONNECTED
            Some tests will be skipped on this interface.

        Host Name. . . . . . . . . : waps-dc3
        Autoconfiguration IP Address : 169.254.156.70
        Subnet Mask. . . . . . . . : 255.255.0.0
        Default Gateway. . . . . . :
        Dns Servers. . . . . . . . : 10.102.4.18




Global results:


Domain membership test . . . . . . : Passed


NetBT transports test. . . . . . . : Passed
    List of NetBt transports currently configured:
        NetBT_Tcpip_{4F3F18F4-9B9A-4153-80A4-926148A68975}
        NetBT_Tcpip_{58A89523-8E5F-46F9-A0B2-33C11B489DDC}
    2 NetBt transports currently configured.


Autonet address test . . . . . . . : Passed


IP loopback ping test. . . . . . . : Passed


Default gateway test . . . . . . . : Failed

    [FATAL] NO GATEWAYS ARE REACHABLE.
    You have no connectivity to other network segments.
    If you configured the IP protocol manually then
    you need to add at least one valid gateway.


NetBT name test. . . . . . . . . . : Passed
    [WARNING] You don't have a single interface with the <00> 'WorkStation Service', <03> 'Messenger

Service', <20> 'WINS' names defined.


Winsock test . . . . . . . . . . . : Passed


DNS test . . . . . . . . . . . . . : Passed
          [WARNING] Cannot find a primary authoritative DNS server for the name
            'waps-dc3.XXXXXX.org.'. [ERROR_TIMEOUT]
            The name 'XXXXX.org.' may not be registered in DNS.
       [WARNING] The DNS entries for this DC cannot be verified right now on DNS server 10.102.4.18,

ERROR_TIMEOUT.
    PASS - All the DNS entries for DC are registered on DNS server '10.102.1.3' and other DCs also have

some of the names registered.


Redir and Browser test . . . . . . : Passed
    List of NetBt transports currently bound to the Redir
        NetBT_Tcpip_{4F3F18F4-9B9A-4153-80A4-926148A68975}
        NetBT_Tcpip_{58A89523-8E5F-46F9-A0B2-33C11B489DDC}
    The redir is bound to 2 NetBt transports.

    List of NetBt transports currently bound to the browser
        NetBT_Tcpip_{4F3F18F4-9B9A-4153-80A4-926148A68975}
        NetBT_Tcpip_{58A89523-8E5F-46F9-A0B2-33C11B489DDC}
    The browser is bound to 2 NetBt transports.


DC discovery test. . . . . . . . . : Passed


DC list test . . . . . . . . . . . : Failed
        Failed to enumerate DCs by using the browser. [ERROR_NO_BROWSER_SERVERS_FOUND]


Trust relationship test. . . . . . : Failed
    [FATAL] Secure channel to domain 'CAMPUS' is broken. [ERROR_NO_LOGON_SERVERS]


Kerberos test. . . . . . . . . . . : Passed


LDAP test. . . . . . . . . . . . . : Passed
    [WARNING] Failed to query SPN registration on DC 'WAPS-EX.XXXXX.org'.


Bindings test. . . . . . . . . . . : Passed


WAN configuration test . . . . . . : Skipped
    No active remote access connections.


Modem diagnostics test . . . . . . : Passed

IP Security test . . . . . . . . . : Skipped

    Note: run "netsh ipsec dynamic show /?" for more detailed information


The command completed successfully

repadmin running command /bind against server localhos

Bind to localhost succeeded.
Extensions supported:
    BASE                             : Yes
    ASYNCREPL                        : Yes
    REMOVEAPI                        : Yes
    MOVEREQ_V2                       : Yes
    GETCHG_COMPRESS                  : Yes
    DCINFO_V1                        : Yes
    RESTORE_USN_OPTIMIZATION         : Yes
    KCC_EXECUTE                      : Yes
    ADDENTRY_V2                      : Yes
    LINKED_VALUE_REPLICATION         : No    
DCINFO_V2                        : Yes
    INSTANCE_TYPE_NOT_REQ_ON_MOD     : Yes
    CRYPTO_BIND                      : Yes
    GET_REPL_INFO                    : Yes
    STRONG_ENCRYPTION                : Yes
    DCINFO_VFFFFFFFF                 : Yes
    TRANSITIVE_MEMBERSHIP            : Yes
    ADD_SID_HISTORY                  : Yes
    POST_BETA3                       : Yes
    GET_MEMBERSHIPS2                 : Yes
    GETCHGREQ_V6 (WHISTLER PREVIEW)  : Yes
    NONDOMAIN_NCS                    : Yes
    GETCHGREQ_V8 (WHISTLER BETA 1)   : Yes
    GETCHGREPLY_V5 (WHISTLER BETA 2) : Yes
    GETCHGREPLY_V6 (WHISTLER BETA 2) : Yes
    ADDENTRYREPLY_V3 (WHISTLER BETA 3): Yes
    GETCHGREPLY_V7 (WHISTLER BETA 3) : Yes
    VERIFY_OBJECT (WHISTLER BETA 3)  : Yes
    XPRESS_COMPRESSION               : Yes

Site GUID: b6ff6393-1f66-4d52-a814-72ecdea35d57
Repl epoch: 0


Microsoft Windows [Version 5.2.3790]
(C) Copyright 1985-2003 Microsoft Corp.

C:\Documents and Settings\Administrator.CAMPUS>f:

F:\>ntdsutil
ntdsutil: metatag cleanup
Error 80070057 parsing input - illegal syntax?
ntdsutil: NTDSUtil ?
Error 80070057 parsing input - illegal syntax?
ntdsutil: Metadata cleanup
metadata cleanup: connections
server connections: connect to localhost
Error 80070057 parsing input - illegal syntax?
server connections: connect to server localhost
Binding to localhost ...
Connected to localhost using credentials of locally logged on user.
server connections:

on the DCDIAG

Default gateway test . . . : Failed
            No gateway reachable for this adapter.

the gateway configured on this NIC is correct, I've even tried enabling the second NIC and see if that worked
All these errors are normal if your network card is not working or if there is a TPCIP problem.
Like i said before : "support tools will help you diagnose AD problem, but if you still can not ping any other DC, computers or device, it won't help you much"

I see you have 2 NICS on the server, one is disconnected. The first one can have a problem, lets try the second one, it's a quick test.

Remove IP configuration from 1st NIC, set it  with 'obtained by DHCP server' values and Deactivate it.
Set 2nd NIC with these parameters :
        IP Address . . . . . . . . : 10.102.1.3
        Subnet Mask. . . . . . . . : 255.255.0.0
        Default Gateway. . . . . . : 10.102.1.1
        Dns Servers. . . . . . . . : 10.102.4.18
                                     10.102.1.3

Plug in the network wire and try to ping your gateway 10.102.1.1

Post the result.
Results..
destination unrearchable
Also when I set the othe NIC to DHCP and disabled, it never found the DHCP server which is up and running fine on WAPS-EX
Optional question : Is it a Proliant HP Server ?

Let's get back in time and analyze what happened.
Open event viewer once more and check both application and system errors/warning that are registered when you turn the server on.

We are looking for SERVICE CONTROL MANAGER, IPSEC, TCPIP, NETSVC errors
Also anything else that can give clues about why the server is acting this way.

Dont look at errors about DHCP, DNS, AD, File replication, and other network dependent services.

Post the source and event ID of these errors.
its a Dell 2950

What I did right before the power outage, which I feel is causing this issue ...was running an AD prep on the two DCs preparing so i could install Exchange 2010 on our new 08 server, one of the prep steps is to changed the AD mode from 2000 native to 2003. Thats the only thing i did with AD before this whole thing.

Ill go up again and post the errors
As far as I know, that wouldnt stop the network from working.
ADprep may has not finished setting up the domain and schema extension, but in most situations you should still be able to ping your gateway... So we will focus on establishing network link again.
We will fix AD later
You were right IPsec blocking, following MS technet to fix this, will update
ASKER CERTIFIED SOLUTION
Avatar of Gerald26
Gerald26
Flag of France image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Wow , like he said start with the basics, was the IPsec block mode
thanks man for all your help!