Active Directory cannot connect to domain and no Global Catalog server in dcdiag

This is the dcdiag on PDC, GC,
 

C:\Users...>dcdiag

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = SMCDC01
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site\SMCDC01
      Starting test: Connectivity
         ......................... SMCDC01 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site\SMCDC01
      Starting test: Advertising
         Fatal Error:DsGetDcName (SMCDC01) call failed, error 1355
         The Locator could not find the server.
         ......................... SMCDC01 failed test Advertising
      Starting test: FrsEvent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... SMCDC01 failed test FrsEvent
      Starting test: DFSREvent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... SMCDC01 failed test DFSREvent
      Starting test: SysVolCheck
         ......................... SMCDC01 passed test SysVolCheck
      Starting test: KccEvent
         ......................... SMCDC01 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         ......................... SMCDC01 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         ......................... SMCDC01 passed test MachineAccount
      Starting test: NCSecDesc
         ......................... SMCDC01 passed test NCSecDesc
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\SMCDC01\netlogon)
         [SMCDC01] An net use or LsaPolicy operation failed with error 67,
         Win32 Error 67.
         ......................... SMCDC01 failed test NetLogons
      Starting test: ObjectsReplicated
         ......................... SMCDC01 passed test ObjectsReplicated
      Starting test: Replications
         [Replications Check,SMCDC01] A recent replication attempt failed:
            From SMCDC03 to SMCDC01
            Naming Context: DC=ForestDnsZones,DC=StMarys,DC=local
            The replication generated an error (1908):
            Win32 Error 1908
            The failure occurred at 2009-09-10 09:50:26.
            The last success occurred at 2009-09-10 09:46:55.
            1 failures have occurred since the last success.
            Kerberos Error.
            A KDC was not found to authenticate the call.
            Check that sufficient domain controllers are available.
         [Replications Check,SMCDC01] A recent replication attempt failed:
            From SMCDC03 to SMCDC01
            Naming Context: CN=Schema,CN=Configuration,DC=StMarys,DC=local
            The replication generated an error (1908):
            Win32 Error 1908
            The failure occurred at 2009-09-10 09:49:29.
            The last success occurred at 2009-09-10 09:46:55.
            1 failures have occurred since the last success.
            Kerberos Error.
            A KDC was not found to authenticate the call.
            Check that sufficient domain controllers are available.
         ......................... SMCDC01 failed test Replications
      Starting test: RidManager
         ......................... SMCDC01 passed test RidManager
      Starting test: Services
         ......................... SMCDC01 passed test Services
      Starting test: SystemLog
         An Warning Event occurred.  EventID: 0x800007DC
            Time Generated: 09/10/2009   10:06:21
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x800007DC
            Time Generated: 09/10/2009   10:06:21
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x800007DC
            Time Generated: 09/10/2009   10:06:21
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x800007DC
            Time Generated: 09/10/2009   10:06:21
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x800007DC
            Time Generated: 09/10/2009   10:06:21
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x800007DC
            Time Generated: 09/10/2009   10:06:21
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x800007DC
            Time Generated: 09/10/2009   10:06:21
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x800007DC
            Time Generated: 09/10/2009   10:06:21
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x800007DC
            Time Generated: 09/10/2009   10:06:21
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x800007DC
            Time Generated: 09/10/2009   10:06:23
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x8004001A
            Time Generated: 09/10/2009   10:06:58
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x8000001D
            Time Generated: 09/10/2009   10:07:43
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x8000A000
            Time Generated: 09/10/2009   10:08:13
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x8000A000
            Time Generated: 09/10/2009   09:46:22
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x8000A000
            Time Generated: 09/10/2009   09:46:25
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x8000A000
            Time Generated: 09/10/2009   09:46:25
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000469
            Time Generated: 09/10/2009   09:46:27
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x825A000C
            Time Generated: 09/10/2009   09:46:32
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000469
            Time Generated: 09/10/2009   09:47:41
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x00001695
            Time Generated: 09/10/2009   09:51:27
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x00001695
            Time Generated: 09/10/2009   09:51:27
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Warning Event occurred.  EventID: 0x00001695
            Time Generated: 09/10/2009   09:51:27
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   09:56:30
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   10:01:35
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   10:06:38
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000469
            Time Generated: 09/10/2009   10:11:38
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000457
            Time Generated: 09/10/2009   10:21:46
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000457
            Time Generated: 09/10/2009   10:21:46
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000457
            Time Generated: 09/10/2009   10:21:50
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000457
            Time Generated: 09/10/2009   10:21:51
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000457
            Time Generated: 09/10/2009   10:21:51
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000457
            Time Generated: 09/10/2009   10:21:52
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         ......................... SMCDC01 failed test SystemLog
      Starting test: VerifyReferences
         ......................... SMCDC01 passed test VerifyReferences


   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test
         CrossRefValidation

   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test
         CrossRefValidation

   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

   Running partition tests on : StMarys
      Starting test: CheckSDRefDom
         ......................... StMarys passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... StMarys passed test CrossRefValidation

   Running enterprise tests on : StMarys.local
      Starting test: LocatorCheck
         Warning: DcGetDcName(GC_SERVER_REQUIRED) call failed, error 1355
         A Global Catalog Server could not be located - All GC's are down.

Urgent help required  - no one can logon.  Any help would be useful.
birchy66Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Chris DentPowerShell DeveloperCommented:

Please would you show us:

ipconfig /all
netdiag

Or can you confirm the DNS settings for your servers / clients.

Can you confirm that you do have a Global Catalog available (AD Sites and Services)?

And would you provide details of the error messages from the Event Logs.

Cheers,

Chris
birchy66Author Commented:
DNS server settings are the two DC's and these are displayed in ipconfig results
The server that I ran the dcdiag on is listed as a global catalog server in AD sites and services but I cannot get into AD Domains and Trusts to verify the OM role.

Event error info to follow
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

birchy66Author Commented:
Here are the most pertinent event logs as requested:

DNS Event log warning

Log Name:      DNS Server
Source:        Microsoft-Windows-DNS-Server-Service
Date:          10/09/2009 09:50:22
Event ID:      4013
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      SMCDC01.StMarys.local
Description:
The DNS server is waiting for Active Directory Domain Services (AD DS) to signal that the initial synchronization of the directory has been completed. The DNS server service cannot start until the initial synchronization is complete because critical DNS data might not yet be replicated onto this domain controller. If events in the AD DS event log indicate that there is a problem with DNS name resolution, consider adding the IP address of another DNS server for this domain to the DNS server list in the Internet Protocol properties of this computer. This event will be logged every two minutes until AD DS has signaled that the initial synchronization has successfully completed.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-DNS-Server-Service" Guid="{71A551F5-C893-4849-886B-B5EC8502641E}" EventSourceName="DNS" />
    <EventID Qualifiers="32768">4013</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2009-09-10T08:50:22.000Z" />
    <EventRecordID>65</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>DNS Server</Channel>
    <Computer>SMCDC01.StMarys.local</Computer>
    <Security />
  </System>
  <EventData Name="DNS_EVENT_DS_OPEN_WAIT">
  </EventData>
</Event>

Directory Services Error

Log Name:      Directory Service
Source:        Microsoft-Windows-ActiveDirectory_DomainService
Date:          10/09/2009 11:00:49
Event ID:      1126
Task Category: Global Catalog
Level:         Error
Keywords:      Classic
User:          ANONYMOUS LOGON
Computer:      SMCDC01.StMarys.local
Description:
Active Directory Domain Services was unable to establish a connection with the global catalog.
 
Additional Data
Error value:
1355 The specified domain either does not exist or could not be contacted.
Internal ID:
3200d50
 
User Action:
Make sure a global catalog is available in the forest, and is reachable from this domain controller. You may use the nltest utility to diagnose this problem.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-ActiveDirectory_DomainService" Guid="{0e8478c5-3605-4e8c-8497-1e730c959516}" EventSourceName="NTDS General" />
    <EventID Qualifiers="49152">1126</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>18</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2009-09-10T10:00:49.785Z" />
    <EventRecordID>6687</EventRecordID>
    <Correlation />
    <Execution ProcessID="588" ThreadID="784" />
    <Channel>Directory Service</Channel>
    <Computer>SMCDC01.StMarys.local</Computer>
    <Security UserID="S-1-5-7" />
  </System>
  <EventData>
    <Data>3200d50</Data>
    <Data>1355</Data>
    <Data>The specified domain either does not exist or could not be contacted.</Data>
  </EventData>
</Event>

File Replication Service Warning

Log Name:      File Replication Service
Source:        NtFrs
Date:          10/09/2009 09:56:35
Event ID:      13508
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      SMCDC01.StMarys.local
Description:
The File Replication Service is having trouble enabling replication from SMCDC03 to SMCDC01 for c:\windows\sysvol\domain using the DNS name smcdc03.StMarys.local. FRS will keep retrying.
 Following are some of the reasons you would see this warning.
 
 [1] FRS can not correctly resolve the DNS name smcdc03.StMarys.local from this computer.
 [2] FRS is not running on smcdc03.StMarys.local.
 [3] The topology information in the Active Directory Domain Services for this replica has not yet replicated to all the Domain Controllers.
 
 This event log message will appear once per connection, After the problem is fixed you will see another event log message indicating that the connection has been established.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="NtFrs" />
    <EventID Qualifiers="32768">13508</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2009-09-10T08:56:35.000Z" />
    <EventRecordID>684</EventRecordID>
    <Channel>File Replication Service</Channel>
    <Computer>SMCDC01.StMarys.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>SMCDC03</Data>
    <Data>SMCDC01</Data>
    <Data>c:\windows\sysvol\domain</Data>
    <Data>smcdc03.StMarys.local</Data>
    <Binary>00000000</Binary>
  </EventData>
</Event>

Chris DentPowerShell DeveloperCommented:

Can you run:

nslookup -q=srv _gc._tcp.StMarys.local

And confirm that it returns some of your domain controllers?

Is Event ID 4013 repeating? Or just a single instance? If it's only one, does that coincide with a reboot of the server? If so, we won't worry about it.

Chris
birchy66Author Commented:

Thanks for the reply.

Here is the nslookup result:

C:\Users\administrator.STMARYS>nslookup -q=srv _gc._tcp.StMarys.local
Server:  smcdc01.stmarys.local
Address:  10.11.55.253

_gc._tcp.StMarys.local  SRV service location:
          priority       = 0
          weight         = 100
          port           = 3268
          svr hostname   = smcdc01.stmarys.local
_gc._tcp.StMarys.local  SRV service location:
          priority       = 0
          weight         = 100
          port           = 3268
          svr hostname   = smcdc03.stmarys.local
smcdc01.stmarys.local   internet address = 10.11.55.253
smcdc03.stmarys.local   internet address = 10.11.55.239

Event 4013 does coincide with a reboot yes.
Chris DentPowerShell DeveloperCommented:

Would you run this please?

nltest /DsGetDc:stmarys.local

From the above I would expect it to return one of your domain controllers and for that to have the GC flag. Note that this calls the same function DcDiag where it returned a fatal error previously.

There are a couple more records in DNS we may look at if that fails. However, you might also consider running:

netdiag /fix

Which should force it to re-register the records we'd be looking for.

Can you confirm that SysVol and NetLogon are shared on both of your domain controllers?

Chris
Chris DentPowerShell DeveloperCommented:

Sorry, one more.

Did you also run DCDiag on the second DC? The two will share the load, so it's not really important which has the PDC Emulator role. And I assume there's no smcdc02?

Chris
birchy66Author Commented:
DCdiag from the other DC is as follows and no there isn't an smcdc02:

C:\Users\administrator.STMARYS>dcdiag

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = smcdc03
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site\SMCDC03
      Starting test: Connectivity
         ......................... SMCDC03 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site\SMCDC03
      Starting test: Advertising
         Fatal Error:DsGetDcName (SMCDC03) call failed, error 1355
         The Locator could not find the server.
         ......................... SMCDC03 failed test Advertising
      Starting test: FrsEvent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... SMCDC03 passed test FrsEvent
      Starting test: DFSREvent
         ......................... SMCDC03 passed test DFSREvent
      Starting test: SysVolCheck
         ......................... SMCDC03 passed test SysVolCheck
      Starting test: KccEvent
         ......................... SMCDC03 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         ......................... SMCDC03 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         ......................... SMCDC03 passed test MachineAccount
      Starting test: NCSecDesc
         ......................... SMCDC03 passed test NCSecDesc
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\SMCDC03\netlogon)
         [SMCDC03] An net use or LsaPolicy operation failed with error 67,
         Win32 Error 67.
         ......................... SMCDC03 failed test NetLogons
      Starting test: ObjectsReplicated
         ......................... SMCDC03 passed test ObjectsReplicated
      Starting test: Replications
         ......................... SMCDC03 passed test Replications
      Starting test: RidManager
         ......................... SMCDC03 passed test RidManager
      Starting test: Services
         ......................... SMCDC03 passed test Services
      Starting test: SystemLog
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   10:44:31
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   10:49:34
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   10:54:36
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   10:59:39
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   11:04:42
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   11:09:45
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   11:14:47
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   11:19:50
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   11:24:53
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   11:29:56
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   11:34:58
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         An Error Event occurred.  EventID: 0x00000406
            Time Generated: 09/10/2009   11:39:59
            EvtFormatMessage failed, error 15100 Win32 Error 15100.
            (Event String (event log = System) could not be retrieved, error
            0x3afc)
         ......................... SMCDC03 failed test SystemLog
      Starting test: VerifyReferences
         ......................... SMCDC03 passed test VerifyReferences


   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test
         CrossRefValidation

   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test
         CrossRefValidation

   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

Thanks
Chris DentPowerShell DeveloperCommented:

If the nltest command above fails, would you be able to give us this from both DCs:

netdiag /v /l

This will create a netdiag.log file, the file will be pretty long so adding it as an attachment would be preferable if you don't mind.

Chris
birchy66Author Commented:


The nltest result is as follows:

C:\Users\administrator.STMARYS>nltest /DsGetDc:stmarys.local
Getting DC name failed: Status = 1355 0x54b ERROR_NO_SUCH_DOMAIN

Also the sysvol shares are up but no netlogon share.  I checked the default directory (%SystemRoot%\sysvol\sysvol\<DomainName>\scripts) and it is empty.

Servers are 2008 so unable to do a netdiag. Please find a dcdiag attached. Hopefully this provides the same information.

birchy66Author Commented:
Sorry Chris,
Here is the attachment
dcdiag.txt
Chris DentPowerShell DeveloperCommented:

Ack, always forget about that with 2008. We'll have to run some more manual checks in DNS to see what's missing.

Can you run...

nslookup -q=srv _gc._tcp.Default-First-Site._sites.stmarys.local
nslookup -q=srv _ldap._tcp.gc._msdcs.stmarys.local
nslookup -q=srv _ldap._tcp.Default-First-Site._sites.gc._msdcs.stmarys.local

That'll do for a start, if they all work I'll throw together a quick script to check the lot.

If the last two fail, can you verify that the _msdcs.stmarys.local zone exists (or that you have a folder called _msdcs in stmarys.local)?

Chris
birchy66Author Commented:
The last one failed. Results attached. However in DNS there is a folder called _msdcs in stmarys.local.

Andy
nslookup.txt
Chris DentPowerShell DeveloperCommented:

See this is where you really need NetDiag.

Ah well... try restarting the NetLogon service please? Then run the nslookup queries again?

And lets have DCDiag tell us a bit more with these:

DCDiag /c /v /f:dcdiag.log

That runs all tests with verbose output and logs it to the dcdiag.log file.

Do you have PowerShell installed on either of the servers? If not, would you add it? You'll find it under Features. We can run a few quick tests from there to emulate some of stuff netdiag used to tell us.

Chris
birchy66Author Commented:
Thanks Chris

Installing powershell now. The nslookups all worked after restarting netlogon.  Please find the dcdiag log file and nslookup results attached.
nslookup2.txt
birchy66Author Commented:
and the dcdiag.log
dcdiag.log
Chris DentPowerShell DeveloperCommented:

We're still failing on this one?

nltest /DsGetDc:stmarys.local

Could you open up Services and verify that the "Kerberos Key Distribution Center" service is running? That's the KDC it mentions. If it is, but is still failing connections (as mentioned in DCDiag) is there any chance we can go for a reboot?

It's inability to find the netlogon share is a bit troubling. Do you have any errors relating to that in either the NTFRS or DFS logs?

Chris
birchy66Author Commented:
The KDC service is running and I have done a reboot. I cannot access AD users and Computers or site and services. DCdiag looks the same. Also I notice that sysvol is no longer shared.

I can't see any particular errors relating to the netlogon share.

Andy
Chris DentPowerShell DeveloperCommented:

We can force the server to rebuild SysVol (and NetLogon) but you will potentially lose any group policies you have configured which is far from ideal.

Do you have Exchange as well? Or just AD on its own?

Do any of the errors in the FRS log mention Journal Wrap?

Chris
birchy66Author Commented:
Is there a way of exporting group policies and importing them after Sysvol and Netlogon are rebuilt?

We do have just AD and there are no errors in the FRS log relating to journal wrap.  If the worst comes to the worst I would be happy to get the server up and running again and rebuild the GPO's afterwards.

Andy
Chris DentPowerShell DeveloperCommented:

Yes if it will let you into the Group Policy Management Console, which isn't looking too promising if it won't let you into the rest.

Do both DCs exhibit the same symptoms?

Chris
birchy66Author Commented:
Yes both exhibit the same symptoms. One thing I hadn't mentioned as it didn't seem relevant was that this is a virtualised server and I have a snapshot of it from a couple of weeks back. This is not much good for restoring but I could revert to this snapshot in isolation and export the GPO's as not much has changed with them.

Andy
Chris DentPowerShell DeveloperCommented:

Lets rebuild the DNS part since that's pretty straight-forward. It would be nice to rule that out as being broken.

Are you able to open the DNS Console at the moment?

If so:

1. Nominate a server that's going to be the DNS server while we do this. Doesn't matter which you choose.
2. On both servers (and potentially all clients) change the Preferred DNS server to the nominated server (TCP/IP configuration). Do not configure an Alternate DNS server at this stage.
3. On the nominated server:
  a. Open the DNS Console
  b. Expand Forward Lookup Zones
  c. Select stmarys.local
  d. Right click and open the zone properties
  e. Next to Type, select Change. Untick store in Active Directory
  f. Under Dynamic Updates, change it to Secure and Non-Secure
  g. Repeat for _msdcs.stmarys.local
4. On both Domain Controllers run:

  ipconfig /registerdns
  net stop netlogon && net start netlogon

That takes DNS out of AD, we want to ensure that is available throughout.

Next, if you can reboot DC01 again please do so. See how well it started up, we'll hope that DNS is fine at this stage. If it's still failing to load SysVol and NetLogon we can move onto having it rebuild that.

Chris
birchy66Author Commented:

Can't find this:

  g. Repeat for _msdcs.stmarys.local

There is an _msdcs folder under stmarys.local but there is not a properties option when I right click on it. Is this what you meant?

I have completed the other steps and restarting now.

Andy
Chris DentPowerShell DeveloperCommented:

Okay, don't worry about _msdcs as a zone. That appears if the domain is built as 2008, but if it was built a long time ago and upgraded it wouldn't be present.

Either way it doesn't matter, as long as you have the folder somewhere :)

Chris
birchy66Author Commented:
Okay, server has rebooted. still no sysvol or netlogon on net share

Andy
Chris DentPowerShell DeveloperCommented:

Okay, rebuilding NetLogon / SysVol is covered by this KB article:

http://support.microsoft.com/kb/290762

If neither have the shares we're looking at an Authoritative Rebuild, one server will use the D4 setting, the second will get D2 (non-authoritative) so it copies from the first server.

I advise you take a snapshot of both servers before doing this if you can. It's never fun to have nowhere to turn if it only makes things worse.

Chris

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
birchy66Author Commented:
Chris

I took a snapshot of both and completed the steps as directed.  What should I see now / what is next?

Thanks for your help

Andy
Chris DentPowerShell DeveloperCommented:

Any sign of NetLogon / SysVol?

Chris
birchy66Author Commented:
Yes!  And I can access AD on both servers. What else do I need to do?

Does DNS need re associating with AD?
Chris DentPowerShell DeveloperCommented:

DCDiag again? See how happy it is before switching DNS back to AD Integrated.

Chris
birchy66Author Commented:
It's better than it was but still with errors. failing on Dfsrevent and systemlog.  Please see attached

Andy
dcdiag2.log
Chris DentPowerShell DeveloperCommented:

Check the date stamps on the event log errors it's flagging. They may well be from earlier, if you can't see any since the last reboot then I'd be quite happy :)

Chris
birchy66Author Commented:
Hi Chris

Yes you are correct the time stamps are before the sysvol rebuild

Can you advise of the next step.

Thanks
Chris DentPowerShell DeveloperCommented:

Switch DNS back to AD Integrated (DNS Console, zone properties, Change Type, and tick the box), and set updates to Secure Only.

Then it's just a case of keep an eye on it, make sure it doesn't become upset again.

Chris
birchy66Author Commented:
Have switched DNS back and rebooting smcdc01 for good measure.  Will check that users / systems can logon once it is rebuilt. I haven't checked GPO's, am I likely to need to reinstate these?

Andy

BTW you will have more than earned your points if this works!
Chris DentPowerShell DeveloperCommented:

Potentially. We can recreate the default policies easily, but anything which isn't default is likely to have been killed off. Of course, having an operational domain is more important at this stage :)

Chris
birchy66Author Commented:
Well I have tested computer and user accounts for logging on and all seems well. Group policies also seem to be working fine for drive mappings, folder redirection  and general security also!  You have definitely saved my bacon. Now I just have to work out why it happened and ensure it does not recurr.

Thanks

Andy
Chris DentPowerShell DeveloperCommented:

Yeah, that one will be the harder one, a good long look at the event logs is in order :)

Do you have Aging configured in DNS? I did wonder if some of the records we were missing were removed by Scavenging.

Chris
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2008

From novice to tech pro — start learning today.