Process STORE.EXE (PID=3388). All Global Catalog Servers in forest are not responding:

Is possible your group policies may not be replicating properly and as a result may not be applying the correct permissions as per my guide here: https://www.experts-exchange.com/Software/Server_Software/Email_Servers/Exchange/A_2829-Exchange-2007-Topology-discovery-failed.html

Can you check the settings mentioned in my guide on the new Domain Controller.

When I try and edit the GPO mentioned in your article I get this error:

Failed to open the group policy object, you may not have appropriate rights

Details: the network name cannot be found.

My user is a member of the domain admins and pretty much every other elevated group in AD.

yep, sounds like you have either a replication or a DNS issue.

Can you post DCDIAG output from both DC's and IPCONFIG /ALL from both.

I believe it is a DNS issue, I'm going to correct it and try again. Thanks

I faced a similar issue. This might happen if Exchange permissions are missing on the Domain side.
You check the Information Store Service it will be in the Starting status.
Try running setup /Adprep again and restart the Exchange Server.

Running that again will not help if the Group Policies are not being replicated.  the /ADPREP switch places some permissions in the Default DOmain Controller policy.  If they are not replicating this will not be applied to the new DC's.

 
 Testing server: Default-First-Site-Name\DC01
    Starting test: Advertising
       Warning: DsGetDcName returned information for \\DC name,
       when we were trying to reach DC01.
       SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.
       ......................... MISTY-DC01 failed test Advertising
    Starting test: FrsEvent

IT is an issue with replication as noted by the dcdiag output.  DCdiag is filled with errors on both my new server 2008 DC's

Thoughts on how to fix this?

firstly, can you post IPCONFIG /ALL from both DC's please?

    Starting test: FrsEvent
        There are warning or error events within the last 24 hours after the
        SYSVOL has been shared.  Failing SYSVOL replication problems may cause
        Group Policy problems.
        ......................... test-DC01 passed test FrsEvent
     Starting test: DFSREvent
        ......................... test-DC01 passed test DFSREvent
     Starting test: SysVolCheck
        ......................... test-DC01 passed test SysVolCheck
     Starting test: KccEvent
        ......................... test-DC01 passed test KccEvent
     Starting test: KnowsOfRoleHolders
        ......................... test-DC01 passed test KnowsOfRoleHolders
     Starting test: MachineAccount
        ......................... test-DC01 passed test MachineAccount
     Starting test: NCSecDesc
        ......................... test-DC01 passed test NCSecDesc
     Starting test: NetLogons
        Unable to connect to the NETLOGON share! (\\test-DC01\netlogon)
        [test-DC01] An net use or LsaPolicy operation failed with error 67,
        The network name cannot be found..
        ......................... test-DC01 failed test NetLogons
     Starting test: ObjectsReplicated
        ......................... test-DC01 passed test ObjectsReplicated
     Starting test: Replications
        [Replications Check,test-DC01] DsReplicaGetInfo(PENDING_OPS, NULL)
        failed, error 0x2105 "Replication access was denied."
        ......................... test-DC01 failed test Replications
     Starting test: RidManager
        ......................... test-DC01 passed test RidManager
     Starting test: Services
           Could not open NTDS Service on test-DC01, error 0x5
           "Access is denied."
        ......................... test-DC01 failed test Services
     Starting test: SystemLog
        A warning event occurred.  EventID: 0x8000001D
           Time Generated: 04/30/2011   11:24:15
           Event String:
           The Key Distribution Center (KDC) cannot find a suitable certificate
to use for smart card logons, or the KDC certificate could not be verified. Sma
t card logon may not function correctly if this problem is not resolved. To cor
ect this problem, either verify the existing KDC certificate using certutil.exe
or enroll for a new KDC certificate.
        A warning event occurred.  EventID: 0x00000C18
           Time Generated: 04/30/2011   11:24:19
           Event String:
           The primary Domain Controller for this domain could not be located.
        A warning event occurred.  EventID: 0x000003F6
           Time Generated: 04/30/2011   11:24:44
           Event String:
           Name resolution for the name _kerberos._tcp.Default-First-Site-Name.
sites.dc._msdcs.test.HOME timed out after none of the configured DNS servers r
sponded.
        An error event occurred.  EventID: 0xC00038D6
           Time Generated: 04/30/2011   11:24:46
           Event String:
           The DFS Namespace service could not initialize cross forest trust in
ormation on this domain controller, but it will periodically retry the operatio
. The return code is in the record data.
        A warning event occurred.  EventID: 0x000003F6
           Time Generated: 04/30/2011   11:24:58
           Event String:
           Name resolution for the name test.home timed out after none of the
onfigured DNS servers responded.
        An error event occurred.  EventID: 0xC00038D6
           Time Generated: 04/30/2011   11:25:13
           Event String:
           The DFS Namespace service could not initialize cross forest trust in
ormation on this domain controller, but it will periodically retry the operatio
. The return code is in the record data.
        A warning event occurred.  EventID: 0x00000081
           Time Generated: 04/30/2011   11:25:38
           Event String:
           NtpClient was unable to set a domain peer to use as a time source be
ause of discovery error. NtpClient will try again in 3473457 minutes and double
the reattempt interval thereafter. The error was: The entry is not found. (0x80
706E1)
        A warning event occurred.  EventID: 0x00000081
           Time Generated: 04/30/2011   11:25:40
           Event String:
           NtpClient was unable to set a domain peer to use as a time source be
ause of discovery error. NtpClient will try again in 3473457 minutes and double
the reattempt interval thereafter. The error was: The entry is not found. (0x80
706E1)
        An error event occurred.  EventID: 0xC00038D6
           Time Generated: 04/30/2011   11:25:40
           Event String:
           The DFS Namespace service could not initialize cross forest trust in
ormation on this domain controller, but it will periodically retry the operatio
. The return code is in the record data.
        An error event occurred.  EventID: 0xC00038D6
           Time Generated: 04/30/2011   11:26:07
           Event String:
           The DFS Namespace service could not initialize cross forest trust in
ormation on this domain controller, but it will periodically retry the operatio
. The return code is in the record data.
        An error event occurred.  EventID: 0xC00038D6
           Time Generated: 04/30/2011   11:26:34
           Event String:
           The DFS Namespace service could not initialize cross forest trust in
ormation on this domain controller, but it will periodically retry the operatio
. The return code is in the record data.
        An error event occurred.  EventID: 0xC00038D6
           Time Generated: 04/30/2011   11:27:01
           Event String:
           The DFS Namespace service could not initialize cross forest trust in
ormation on this domain controller, but it will periodically retry the operatio
. The return code is in the record data.
        An error event occurred.  EventID: 0xC00038D6
           Time Generated: 04/30/2011   11:27:28
           Event String:
           The DFS Namespace service could not initialize cross forest trust in
ormation on this domain controller, but it will periodically retry the operatio
. The return code is in the record data.
        An error event occurred.  EventID: 0xC00038D6
           Time Generated: 04/30/2011   11:27:55
           Event String:
           The DFS Namespace service could not initialize cross forest trust in
ormation on this domain controller, but it will periodically retry the operatio
. The return code is in the record data.
        A warning event occurred.  EventID: 0x000727AA
           Time Generated: 04/30/2011   11:28:15
           Event String:
           The WinRM service failed to create the following SPNs: WSMAN/test-D
01.test.home; WSMAN/test-DC01.
        An error event occurred.  EventID: 0xC00038D6
           Time Generated: 04/30/2011   11:28:22
           Event String:
           The DFS Namespace service could not initialize cross forest trust in
ormation on this domain controller, but it will periodically retry the operatio
. The return code is in the record data.
        An error event occurred.  EventID: 0xC00038D6
           Time Generated: 04/30/2011   11:28:49
           Event String:
           The DFS Namespace service could not initialize cross forest trust in
ormation on this domain controller, but it will periodically retry the operatio
. The return code is in the record data.
        An error event occurred.  EventID: 0xC00038D6
           Time Generated: 04/30/2011   11:29:16
           Event String:
           The DFS Namespace service could not initialize cross forest trust in
ormation on this domain controller, but it will periodically retry the operatio
. The return code is in the record data.
        A warning event occurred.  EventID: 0x00001695
           Time Generated: 04/30/2011   11:30:12
           Event String:
           Dynamic registration or deletion of one or more DNS records associat
d with DNS domain 'test.home.' failed.  These records are used by other comput
rs to locate this server as a domain controller (if the specified domain is an
ctive Directory domain) or as an LDAP server (if the specified domain is an app
ication partition).
        An error event occurred.  EventID: 0x00000469
           Time Generated: 04/30/2011   11:39:24
           Event String:
           The processing of Group Policy failed because of lack of network con
ectivity to a domain controller. This may be a transient condition. A success m
ssage would be generated once the machine gets connected to the domain controll
r and Group Policy has succesfully processed. If you do not see a success messa
e for several hours, then contact your administrator.
        An error event occurred.  EventID: 0x00000469
           Time Generated: 04/30/2011   11:41:35
           Event String:
           The processing of Group Policy failed because of lack of network con
ectivity to a domain controller. This may be a transient condition. A success m
ssage would be generated once the machine gets connected to the domain controll
r and Group Policy has succesfully processed. If you do not see a success messa
e for several hours, then contact your administrator.
        A warning event occurred.  EventID: 0x00001695
           Time Generated: 04/30/2011   11:42:50
           Event String:
           Dynamic registration or deletion of one or more DNS records associat
d with DNS domain 'DomainDnsZones.test.home.' failed.  These records are used
y other computers to locate this server as a domain controller (if the specifie
 domain is an Active Directory domain) or as an LDAP server (if the specified d
main is an application partition).
        A warning event occurred.  EventID: 0x00001695
           Time Generated: 04/30/2011   11:43:26
           Event String:
           Dynamic registration or deletion of one or more DNS records associat
d with DNS domain 'ForestDnsZones.test.home.' failed.  These records are used
y other computers to locate this server as a domain controller (if the specifie
 domain is an Active Directory domain) or as an LDAP server (if the specified d
main is an application partition).
        An error event occurred.  EventID: 0xC000042B
           Time Generated: 04/30/2011   11:44:07
           Event String:
           The terminal server cannot register 'TERMSRV' Service Principal Name
to be used for server authentication. The following error occured: The system d
tected a possible attempt to compromise security. Please ensure that you can co
tact the server that authenticated you.
        An error event occurred.  EventID: 0x0000168E
           Time Generated: 04/30/2011   11:44:17
           Event String:
           The dynamic registration of the DNS record '_gc._tcp.test.home. 600
IN SRV 0 100 3268 test-DC01.test.home.' failed on the following DNS server:

        A warning event occurred.  EventID: 0x00001695
           Time Generated: 04/30/2011   11:44:17
           Event String:
           Dynamic registration or deletion of one or more DNS records associat
d with DNS domain 'test.home.' failed.  These records are used by other comput
rs to locate this server as a domain controller (if the specified domain is an
ctive Directory domain) or as an LDAP server (if the specified domain is an app
ication partition).
        A warning event occurred.  EventID: 0x00001695
           Time Generated: 04/30/2011   11:44:17
           Event String:
           Dynamic registration or deletion of one or more DNS records associat
d with DNS domain 'DomainDnsZones.test.home.' failed.  These records are used
y other computers to locate this server as a domain controller (if the specifie
 domain is an Active Directory domain) or as an LDAP server (if the specified d
main is an application partition).
        A warning event occurred.  EventID: 0x00001695
           Time Generated: 04/30/2011   11:44:17
           Event String:
           Dynamic registration or deletion of one or more DNS records associat
d with DNS domain 'ForestDnsZones.test.home.' failed.  These records are used
y other computers to locate this server as a domain controller (if the specifie
 domain is an Active Directory domain) or as an LDAP server (if the specified d
main is an application partition).
        A warning event occurred.  EventID: 0x00000081
           Time Generated: 04/30/2011   11:44:54
           Event String:
           NtpClient was unable to set a domain peer to use as a time source be
ause of discovery error. NtpClient will try again in 3473457 minutes and double
the reattempt interval thereafter. The error was: The entry is not found. (0x80
706E1)
        A warning event occurred.  EventID: 0x00000081
           Time Generated: 04/30/2011   11:44:59
           Event String:
           NtpClient was unable to set a domain peer to use as a time source be
ause of discovery error. NtpClient will try again in 3473457 minutes and double
the reattempt interval thereafter. The error was: The entry is not found. (0x80
706E1)
        An error event occurred.  EventID: 0x00000422
           Time Generated: 04/30/2011   11:49:27
           Event String:
           The processing of Group Policy failed. Windows attempted to read the
file \\test.home\sysvol\test.home\Policies\{31B2F340-016D-11D2-945F-00C04FB98
F9}\gpt.ini from a domain controller and was not successful. Group Policy setti
gs may not be applied until this event is resolved. This issue may be transient
and could be caused by one or more of the following:
        ......................... test-DC01 failed test SystemLog
     Starting test: VerifyReferences
        ......................... test-DC01 passed test VerifyReferences


  Running partition tests on : ForestDnsZones
     Starting test: CheckSDRefDom
        ......................... ForestDnsZones passed test CheckSDRefDom
     Starting test: CrossRefValidation
        ......................... ForestDnsZones passed test
        CrossRefValidation

  Running partition tests on : DomainDnsZones
     Starting test: CheckSDRefDom
        ......................... DomainDnsZones passed test CheckSDRefDom
     Starting test: CrossRefValidation
        ......................... DomainDnsZones passed test
        CrossRefValidation

  Running partition tests on : Schema
     Starting test: CheckSDRefDom
        ......................... Schema passed test CheckSDRefDom
     Starting test: CrossRefValidation
        ......................... Schema passed test CrossRefValidation

  Running partition tests on : Configuration
     Starting test: CheckSDRefDom
        ......................... Configuration passed test CheckSDRefDom
     Starting test: CrossRefValidation
        ......................... Configuration passed test CrossRefValidation

  Running partition tests on : test
     Starting test: CheckSDRefDom
        ......................... test passed test CheckSDRefDom
     Starting test: CrossRefValidation
        ......................... test passed test CrossRefValidation

  Running enterprise tests on : test.home
     Starting test: LocatorCheck
        ......................... test.home passed test LocatorCheck
     Starting test: Intersite
        ......................... test.home passed test Intersite

:\Users\philh>

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : 00-0C-29-D8-12-C9
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.1.189(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1
   DNS Servers . . . . . . . . . . . : 192.168.1.188
                                       192.168.1.189
   NetBIOS over Tcpip. . . . . . . . : Enabled



Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : 00-0C-29-A1-22-2B
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.1.199(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1
   DNS Servers . . . . . . . . . . . : 192.168.1.188
                                       192.168.1.189
                                       127.0.0.1
   NetBIOS over Tcpip. . . . . . . . : Enabled

are these the 2 DC's?
Are they both DNS servers as well?

Are you planning on removing the 2003 DC?

If the answer to all 3 of the above is yes then set the PREFERED DNS entry on both servers to be the IP address of the 2008 server.

Don't enter the ALTERNATE DNS entry on either.

Then do the following:

from a command prompt run IPCONFIG /FLUSHDNS
then restart the NETOLOGON service
then run IPCONFIG /REGISTERDNS

Once done, run DCDIAG /FIX on both servers and post the complete output please.

1. Yes they are the two new server 2008 DC's
2. Yes they are both DNS servers
3. Yes the 2003 DC will be removed.


I will post the output shortly

:)

completed the process, same error from dcdiag /fix

are you seeing that on both DC's?

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : 00-0C-29-A1-22-2B
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.1.199(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1
   DNS Servers . . . . . . . . . . . : 192.168.1.199
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : 00-0C-29-D8-12-C9
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.1.189(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1
   DNS Servers . . . . . . . . . . . : 192.168.1.189
   NetBIOS over Tcpip. . . . . . . . : Enabled

Only on one so far let me try the other.

the DNS is wrong.
Both servers should be using only the Windows 2008 Server for DNS.

Same issue on both DC's

That is the output from ipconfig on both new server 2008 DC's that are the DNS servers.

Can you check the DNS please, both DC's should be using the 2008 server for PREFERRED DNS.  The output from above does not show this.

Ah OK.

Let's concentrate on 1 of the new DC's at the moment and the 2003 DC.

So, pick 1 of the 2008 DC's (preferably the one that holds the FSMO roles) and we will work on that one, once everything is working we will work on the 2nd.

So on the 2008 DC configure it to use itself for DNS then configure the 2003 server to use only this 2008 server for DNS.

Then follow the flush/restart/register process above.

I'm not understanding what you're referring to. could you clarify? I thought in your previous message you said to make the preferred DNS the IP of the server 2008 DC themselves. These are the only two servers I've been working on so far.

Sorry, I didn't realise you had 2 2008 DC's, I was refering to a 2008 DC and the 2003 DC (which must be on because it's not replicated correctly)

Understood, working on that now.

I actually split the fSMO roles across both new DC's. Is that an issue?

No, in that case, make both the 2008 servers and the 2003 server point to only one of the 2008 DC's

Basically, I just want to make sure all 3 DC's are using the same DNS server for Names Resolution.

Once we are happy with that we can address the replication issue.

All 3 DC's are pointing to .189 for DNS.

ace, and you have run the flushdns/netlogon restart/register dns?

What's the output from DCDIAG /FIX on all 3?

2008 DC's same issue after running the three steps, here is the output from the 2003 DC.


  Done gathering initial info.

oing initial required tests

  Testing server: Default-First-Site-Name\testMAIL
     Starting test: Connectivity
        ......................... testMAIL passed test Connectivity

oing primary tests

  Testing server: Default-First-Site-Name\testMAIL
     Starting test: Replications
        [test-DC02] DsBindWithSpnEx() failed with error 5,
        Access is denied..
        [Replications Check,testMAIL] A recent replication attempt failed:
           From test-DC02 to testMAIL
           Naming Context: DC=test,DC=home
           The replication generated an error (1396):
           Logon Failure: The target account name is incorrect.
           The failure occurred at 2011-04-30 12:40:30.
           The last success occurred at 2011-04-30 12:29:10.
           12 failures have occurred since the last success.
           Kerberos Error.
            The KDC could not find the SPN for the server test-DC02.
            This can be for several reasons:

              (1) - The SPN is not registered on the KDC (usually testMAIL).
              Check that the SPN is registered  on at least one other server
              besides test-DC02, and that replication is progressing between
              this server and the KDC.  The tool repadmin/syncall can be used
              for this purpose.
               (2) - This server could be a deleted server (and deleted DSA
              object), and this deletion has not replicated across the
              enterprise yet.  This will rectify itself within the general
              replication latency plus the latency of the KCC.  Should be less
              than a day.
               (3) - It's possible that this server was reclaimed, but it's
              DSA object was not deleted and an old DNS record representing
              the server is present.  This can result in this error for the
              duration of a DNS record lease.  Often about 2 weeks.  To fix
              this, please clean up the DSA's metadata with ntdsutil.
               (4) - Finally, it's possible that this server has acquired a
              new IP address, the server's old IP address has been reused, and
              DNS hasn't been updated to reflect the new IP address. If this
              problem persists, stop and restart the "Net Logon" service on
              test-DC02, and delete the old DNS record.
        ......................... testMAIL passed test Replications
     Starting test: NCSecDesc
        ......................... testMAIL passed test NCSecDesc
     Starting test: NetLogons
        ......................... testMAIL passed test NetLogons
     Starting test: Advertising
        ......................... testMAIL passed test Advertising
     Starting test: KnowsOfRoleHolders
        Warning: test-DC02 is the Domain Owner, but is not responding to DS RP
 Bind.
        [test-DC02] LDAP bind failed with error 8341,
        A directory service error has occurred..
        Warning: test-DC02 is the Domain Owner, but is not responding to LDAP
ind.
        Warning: test-DC02 is the PDC Owner, but is not responding to DS RPC B
nd.
        Warning: test-DC02 is the PDC Owner, but is not responding to LDAP Bin
.
        Warning: test-DC02 is the Rid Owner, but is not responding to DS RPC B
nd.
        Warning: test-DC02 is the Rid Owner, but is not responding to LDAP Bin
.
        Warning: test-DC02 is the Infrastructure Update Owner, but is not resp
nding to DS RPC Bind.
        Warning: test-DC02 is the Infrastructure Update Owner, but is not resp
nding to LDAP Bind.
        ......................... testMAIL failed test KnowsOfRoleHolders
     Starting test: RidManager
        ......................... testMAIL failed test RidManager
     Starting test: MachineAccount
        ......................... testMAIL passed test MachineAccount
     Starting test: Services
        ......................... testMAIL passed test Services
     Starting test: ObjectsReplicated
        ......................... testMAIL passed test ObjectsReplicated
     Starting test: frssysvol
        ......................... testMAIL passed test frssysvol
     Starting test: frsevent
        There are warning or error events within the last 24 hours after the
        SYSVOL has been shared.  Failing SYSVOL replication problems may cause
        Group Policy problems.
        ......................... testMAIL failed test frsevent
     Starting test: kccevent
        An Error Event occured.  EventID: 0xC000066D
           Time Generated: 04/30/2011   12:30:58
           (Event String could not be retrieved)
        ......................... testMAIL failed test kccevent
     Starting test: systemlog
        An Error Event occured.  EventID: 0x40000005
           Time Generated: 04/30/2011   12:33:22
           Event String: The kerberos client received a KRB_AP_ERR_TKT_NYV
        ......................... testMAIL failed test systemlog
     Starting test: VerifyReferences
        ......................... testMAIL passed test VerifyReferences

  Running partition tests on : ForestDnsZones
     Starting test: CrossRefValidation
        ......................... ForestDnsZones passed test CrossRefValidation

     Starting test: CheckSDRefDom
        ......................... ForestDnsZones passed test CheckSDRefDom

  Running partition tests on : DomainDnsZones
     Starting test: CrossRefValidation
        ......................... DomainDnsZones passed test CrossRefValidation

     Starting test: CheckSDRefDom
        ......................... DomainDnsZones passed test CheckSDRefDom

  Running partition tests on : Schema
     Starting test: CrossRefValidation
        ......................... Schema passed test CrossRefValidation
     Starting test: CheckSDRefDom
        ......................... Schema passed test CheckSDRefDom

  Running partition tests on : Configuration
     Starting test: CrossRefValidation
        ......................... Configuration passed test CrossRefValidation
     Starting test: CheckSDRefDom
        ......................... Configuration passed test CheckSDRefDom

  Running partition tests on : test
     Starting test: CrossRefValidation
        ......................... test passed test CrossRefValidation
     Starting test: CheckSDRefDom
        ......................... test passed test CheckSDRefDom

  Running enterprise tests on : test.home
     Starting test: Intersite
        ......................... test.home passed test Intersite
     Starting test: FsmoCheck
        ......................... test.home passed test FsmoCheck

:\Documents and Settings\philh>






















  Done gathering initial info.

oing initial required tests

  Testing server: Default-First-Site-Name\testMAIL
     Starting test: Connectivity
        ......................... testMAIL passed test Connectivity

oing primary tests

  Testing server: Default-First-Site-Name\testMAIL
     Starting test: Replications
        [test-DC02] DsBindWithSpnEx() failed with error 5,
        Access is denied..
        [Replications Check,testMAIL] A recent replication attempt failed:
           From test-DC02 to testMAIL
           Naming Context: DC=test,DC=home
           The replication generated an error (1396):
           Logon Failure: The target account name is incorrect.
           The failure occurred at 2011-04-30 12:40:30.
           The last success occurred at 2011-04-30 12:29:10.
           12 failures have occurred since the last success.
           Kerberos Error.
            The KDC could not find the SPN for the server test-DC02.
            This can be for several reasons:

              (1) - The SPN is not registered on the KDC (usually testMAIL).
              Check that the SPN is registered  on at least one other server
              besides test-DC02, and that replication is progressing between
              this server and the KDC.  The tool repadmin/syncall can be used
              for this purpose.
               (2) - This server could be a deleted server (and deleted DSA
              object), and this deletion has not replicated across the
              enterprise yet.  This will rectify itself within the general
              replication latency plus the latency of the KCC.  Should be less
              than a day.
               (3) - It's possible that this server was reclaimed, but it's
              DSA object was not deleted and an old DNS record representing
              the server is present.  This can result in this error for the
              duration of a DNS record lease.  Often about 2 weeks.  To fix
              this, please clean up the DSA's metadata with ntdsutil.
               (4) - Finally, it's possible that this server has acquired a
              new IP address, the server's old IP address has been reused, and
              DNS hasn't been updated to reflect the new IP address. If this
              problem persists, stop and restart the "Net Logon" service on
              test-DC02, and delete the old DNS record.
        ......................... testMAIL passed test Replications
     Starting test: NCSecDesc
        ......................... testMAIL passed test NCSecDesc
     Starting test: NetLogons
        ......................... testMAIL passed test NetLogons
     Starting test: Advertising
        ......................... testMAIL passed test Advertising
     Starting test: KnowsOfRoleHolders
        Warning: test-DC02 is the Domain Owner, but is not responding to DS RP
 Bind.
        [test-DC02] LDAP bind failed with error 8341,
        A directory service error has occurred..
        Warning: test-DC02 is the Domain Owner, but is not responding to LDAP
ind.
        Warning: test-DC02 is the PDC Owner, but is not responding to DS RPC B
nd.
        Warning: test-DC02 is the PDC Owner, but is not responding to LDAP Bin
.
        Warning: test-DC02 is the Rid Owner, but is not responding to DS RPC B
nd.
        Warning: test-DC02 is the Rid Owner, but is not responding to LDAP Bin
.
        Warning: test-DC02 is the Infrastructure Update Owner, but is not resp
nding to DS RPC Bind.
        Warning: test-DC02 is the Infrastructure Update Owner, but is not resp
nding to LDAP Bind.
        ......................... testMAIL failed test KnowsOfRoleHolders
     Starting test: RidManager
        ......................... testMAIL failed test RidManager
     Starting test: MachineAccount
        ......................... testMAIL passed test MachineAccount
     Starting test: Services
        ......................... testMAIL passed test Services
     Starting test: ObjectsReplicated
        ......................... testMAIL passed test ObjectsReplicated
     Starting test: frssysvol
        ......................... testMAIL passed test frssysvol
     Starting test: frsevent
        There are warning or error events within the last 24 hours after the
        SYSVOL has been shared.  Failing SYSVOL replication problems may cause
        Group Policy problems.
        ......................... testMAIL failed test frsevent
     Starting test: kccevent
        An Error Event occured.  EventID: 0xC000066D
           Time Generated: 04/30/2011   12:30:58
           (Event String could not be retrieved)
        ......................... testMAIL failed test kccevent
     Starting test: systemlog
        An Error Event occured.  EventID: 0x40000005
           Time Generated: 04/30/2011   12:33:22
           Event String: The kerberos client received a KRB_AP_ERR_TKT_NYV
        ......................... testMAIL failed test systemlog
     Starting test: VerifyReferences
        ......................... testMAIL passed test VerifyReferences

  Running partition tests on : ForestDnsZones
     Starting test: CrossRefValidation
        ......................... ForestDnsZones passed test CrossRefValidation

     Starting test: CheckSDRefDom
        ......................... ForestDnsZones passed test CheckSDRefDom

  Running partition tests on : DomainDnsZones
     Starting test: CrossRefValidation
        ......................... DomainDnsZones passed test CrossRefValidation

     Starting test: CheckSDRefDom
        ......................... DomainDnsZones passed test CheckSDRefDom

  Running partition tests on : Schema
     Starting test: CrossRefValidation
        ......................... Schema passed test CrossRefValidation
     Starting test: CheckSDRefDom
        ......................... Schema passed test CheckSDRefDom

  Running partition tests on : Configuration
     Starting test: CrossRefValidation
        ......................... Configuration passed test CrossRefValidation
     Starting test: CheckSDRefDom
        ......................... Configuration passed test CheckSDRefDom

  Running partition tests on : test
     Starting test: CrossRefValidation
        ......................... test passed test CrossRefValidation
     Starting test: CheckSDRefDom
        ......................... test passed test CheckSDRefDom

  Running enterprise tests on : test.home
     Starting test: Intersite
        ......................... test.home passed test Intersite
     Starting test: FsmoCheck
        ......................... test.home passed test FsmoCheck

:\Documents and Settings\philh>









































  Done gathering initial info.

oing initial required tests

  Testing server: Default-First-Site-Name\testMAIL
     Starting test: Connectivity
        ......................... testMAIL passed test Connectivity

oing primary tests

  Testing server: Default-First-Site-Name\testMAIL
     Starting test: Replications
        [test-DC02] DsBindWithSpnEx() failed with error 5,
        Access is denied..
        [Replications Check,testMAIL] A recent replication attempt failed:
           From test-DC02 to testMAIL
           Naming Context: DC=test,DC=home
           The replication generated an error (1396):
           Logon Failure: The target account name is incorrect.
           The failure occurred at 2011-04-30 12:40:30.
           The last success occurred at 2011-04-30 12:29:10.
           12 failures have occurred since the last success.
           Kerberos Error.
            The KDC could not find the SPN for the server test-DC02.
            This can be for several reasons:

              (1) - The SPN is not registered on the KDC (usually testMAIL).
              Check that the SPN is registered  on at least one other server
              besides test-DC02, and that replication is progressing between
              this server and the KDC.  The tool repadmin/syncall can be used
              for this purpose.
               (2) - This server could be a deleted server (and deleted DSA
              object), and this deletion has not replicated across the
              enterprise yet.  This will rectify itself within the general
              replication latency plus the latency of the KCC.  Should be less
              than a day.
               (3) - It's possible that this server was reclaimed, but it's
              DSA object was not deleted and an old DNS record representing
              the server is present.  This can result in this error for the
              duration of a DNS record lease.  Often about 2 weeks.  To fix
              this, please clean up the DSA's metadata with ntdsutil.
               (4) - Finally, it's possible that this server has acquired a
              new IP address, the server's old IP address has been reused, and
              DNS hasn't been updated to reflect the new IP address. If this
              problem persists, stop and restart the "Net Logon" service on
              test-DC02, and delete the old DNS record.
        ......................... testMAIL passed test Replications
     Starting test: NCSecDesc
        ......................... testMAIL passed test NCSecDesc
     Starting test: NetLogons
        ......................... testMAIL passed test NetLogons
     Starting test: Advertising
        ......................... testMAIL passed test Advertising
     Starting test: KnowsOfRoleHolders
        Warning: test-DC02 is the Domain Owner, but is not responding to DS RP
 Bind.
        [test-DC02] LDAP bind failed with error 8341,
        A directory service error has occurred..
        Warning: test-DC02 is the Domain Owner, but is not responding to LDAP
ind.
        Warning: test-DC02 is the PDC Owner, but is not responding to DS RPC B
nd.
        Warning: test-DC02 is the PDC Owner, but is not responding to LDAP Bin
.
        Warning: test-DC02 is the Rid Owner, but is not responding to DS RPC B
nd.
        Warning: test-DC02 is the Rid Owner, but is not responding to LDAP Bin
.
        Warning: test-DC02 is the Infrastructure Update Owner, but is not resp
nding to DS RPC Bind.
        Warning: test-DC02 is the Infrastructure Update Owner, but is not resp
nding to LDAP Bind.
        ......................... testMAIL failed test KnowsOfRoleHolders
     Starting test: RidManager
        ......................... testMAIL failed test RidManager
     Starting test: MachineAccount
        ......................... testMAIL passed test MachineAccount
     Starting test: Services
        ......................... testMAIL passed test Services
     Starting test: ObjectsReplicated
        ......................... testMAIL passed test ObjectsReplicated
     Starting test: frssysvol
        ......................... testMAIL passed test frssysvol
     Starting test: frsevent
        There are warning or error events within the last 24 hours after the
        SYSVOL has been shared.  Failing SYSVOL replication problems may cause
        Group Policy problems.
        ......................... testMAIL failed test frsevent
     Starting test: kccevent
        An Error Event occured.  EventID: 0xC000066D
           Time Generated: 04/30/2011   12:30:58
           (Event String could not be retrieved)
        ......................... testMAIL failed test kccevent
     Starting test: systemlog
        An Error Event occured.  EventID: 0x40000005
           Time Generated: 04/30/2011   12:33:22
           Event String: The kerberos client received a KRB_AP_ERR_TKT_NYV
        ......................... testMAIL failed test systemlog
     Starting test: VerifyReferences
        ......................... testMAIL passed test VerifyReferences

  Running partition tests on : ForestDnsZones
     Starting test: CrossRefValidation
        ......................... ForestDnsZones passed test CrossRefValidation

     Starting test: CheckSDRefDom
        ......................... ForestDnsZones passed test CheckSDRefDom

  Running partition tests on : DomainDnsZones
     Starting test: CrossRefValidation
        ......................... DomainDnsZones passed test CrossRefValidation

     Starting test: CheckSDRefDom
        ......................... DomainDnsZones passed test CheckSDRefDom

  Running partition tests on : Schema
     Starting test: CrossRefValidation
        ......................... Schema passed test CrossRefValidation
     Starting test: CheckSDRefDom
        ......................... Schema passed test CheckSDRefDom

  Running partition tests on : Configuration
     Starting test: CrossRefValidation
        ......................... Configuration passed test CrossRefValidation
     Starting test: CheckSDRefDom
        ......................... Configuration passed test CheckSDRefDom

  Running partition tests on : test
     Starting test: CrossRefValidation
        ......................... test passed test CrossRefValidation
     Starting test: CheckSDRefDom
        ......................... test passed test CheckSDRefDom

  Running enterprise tests on : test.home
     Starting test: Intersite
        ......................... test.home passed test Intersite
     Starting test: FsmoCheck
        ......................... test.home passed test FsmoCheck

:\Documents and Settings\philh>








Doing initial required tests

   Testing server: Default-First-Site-Name\testMAIL
      Starting test: Connectivity
         ......................... testMAIL passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\testMAIL
      Starting test: Replications
         [test-DC02] DsBindWithSpnEx() failed with error 5,
         Access is denied..
         [Replications Check,testMAIL] A recent replication attempt failed:
            From test-DC02 to testMAIL
            Naming Context: DC=test,DC=home
            The replication generated an error (1396):
            Logon Failure: The target account name is incorrect.
            The failure occurred at 2011-04-30 12:40:30.
            The last success occurred at 2011-04-30 12:29:10.
            12 failures have occurred since the last success.
            Kerberos Error.
             The KDC could not find the SPN for the server test-DC02.
             This can be for several reasons:

               (1) - The SPN is not registered on the KDC (usually testMAIL).
               Check that the SPN is registered  on at least one other server
               besides test-DC02, and that replication is progressing between
               this server and the KDC.  The tool repadmin/syncall can be used
               for this purpose.
                (2) - This server could be a deleted server (and deleted DSA
               object), and this deletion has not replicated across the
               enterprise yet.  This will rectify itself within the general
               replication latency plus the latency of the KCC.  Should be less
               than a day.
                (3) - It's possible that this server was reclaimed, but it's
               DSA object was not deleted and an old DNS record representing
               the server is present.  This can result in this error for the
               duration of a DNS record lease.  Often about 2 weeks.  To fix
               this, please clean up the DSA's metadata with ntdsutil.
                (4) - Finally, it's possible that this server has acquired a
               new IP address, the server's old IP address has been reused, and
               DNS hasn't been updated to reflect the new IP address. If this
               problem persists, stop and restart the "Net Logon" service on
               test-DC02, and delete the old DNS record.
         ......................... testMAIL passed test Replications
      Starting test: NCSecDesc
         ......................... testMAIL passed test NCSecDesc
      Starting test: NetLogons
         ......................... testMAIL passed test NetLogons
      Starting test: Advertising
         ......................... testMAIL passed test Advertising
      Starting test: KnowsOfRoleHolders
         Warning: test-DC02 is the Domain Owner, but is not responding to DS RP
C Bind.
         [test-DC02] LDAP bind failed with error 8341,
         A directory service error has occurred..
         Warning: test-DC02 is the Domain Owner, but is not responding to LDAP
Bind.
         Warning: test-DC02 is the PDC Owner, but is not responding to DS RPC B
ind.
         Warning: test-DC02 is the PDC Owner, but is not responding to LDAP Bin
d.
         Warning: test-DC02 is the Rid Owner, but is not responding to DS RPC B
ind.
         Warning: test-DC02 is the Rid Owner, but is not responding to LDAP Bin
d.

what is test-DC02?

server 2008 r2 DC, the testmail is the 2003

can you ping the 2 Windows 2008 DC's from the 2003 server?

Try with IP, NETBIOS name and Fully Qualified Domain Name.

Yes. Connectivity looks good. Ill try a telnet test on the ldap port. Any other suggestions

I've pinged by netbios name. Ill try the fqdn

can you check the DNS server that you have configured them all to use, in the DNS console, right click on the forward lookup zone for your internal domain name and select properties.  Under Updates what does it say?

Is the zone stored in AD? And set to Primary?

Ill check and get back to you in a few minutes. It is an active directory integrated zone I know that. But I had to step away for a moment. If I do nslookups pointed towards server 2008 dc s they do resolve names.  

Also something interesting is ad let me transfer all fsmo roles with no issues. If the environment has having issues I would think that would fail.

run NETDOM QUERY FSMO on all 3 DC's, do they all report the same?

Will do. Ill have the results shortly.

Here are DNS settings you requested.
dns.png

And what is the updates option set to?

same results from all three dc's:
C:\Documents and Settings\philh>netdom query fsmo
Schema owner                Misty-DC01.misty.home

Domain role owner           Misty-dc02.misty.home

PDC role                    Misty-dc02.misty.home

RID pool manager            Misty-dc02.misty.home

Infrastructure owner        Misty-dc02.misty.home

The command completed successfully.

And presumably thats what it should say?

OK, so what is test-DC02? Is it live?

C:\Documents and Settings\philh>ping misty-dc02

Pinging MISTY-DC02.misty.home [192.168.1.199] with 32 bytes of data:

Reply from 192.168.1.199: bytes=32 time<1ms TTL=128
Reply from 192.168.1.199: bytes=32 time<1ms TTL=128
Reply from 192.168.1.199: bytes=32 time<1ms TTL=128
Reply from 192.168.1.199: bytes=32 time<1ms TTL=128

Ping statistics for 192.168.1.199:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Documents and Settings\philh>ping misty-dc01.misty.home

Pinging Misty-DC01.misty.home [192.168.1.189] with 32 bytes of data:

Reply from 192.168.1.189: bytes=32 time=1ms TTL=128
Reply from 192.168.1.189: bytes=32 time<1ms TTL=128
Reply from 192.168.1.189: bytes=32 time<1ms TTL=128
Reply from 192.168.1.189: bytes=32 time<1ms TTL=128

Ping statistics for 192.168.1.189:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 1ms, Average = 0ms

its actually misty-dc02 I was cleansing the output but not really an issue.

Three domain controllers misty-dc01, misty-dc02 (both server 2008)

2003 - mistymail

Yes all these DC's are live and there is no connectivity issues.

OK, so the update setting on the DNS zone, what is this set to?

Yes this all looks correct. But the replication issues persist.

What is the DNS zone update set to?

which update settings are you referring?

Replication?

on the properties of the forward lookup zone, there is an update option, what is it set to?

I dont see an "update" tab or option.

this one:
 

Secure Only on all the DC's

C:\Users\philh>dcdiag /test:replications

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = Misty-DC01
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\MISTY-DC01
      Starting test: Connectivity
         ......................... MISTY-DC01 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\MISTY-DC01
      Starting test: Replications
         [Replications Check,MISTY-DC01] DsReplicaGetInfo(PENDING_OPS, NULL)
         failed, error 0x2105 "Replication access was denied."
         ......................... MISTY-DC01 failed test Replications


   Running partition tests on : ForestDnsZones

   Running partition tests on : DomainDnsZones

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : misty

   Running enterprise tests on : misty.home

If that is an ouput from a Windows 2008 server can you run command prompt as administrator? (Right click Run As Administrator) and then run the command again.

Good point, I didnt run it with elevation. when I did it succeded.
C:\Windows\system32>dcdiag /test:replications

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = Misty-DC01
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\MISTY-DC01
      Starting test: Connectivity
         ......................... MISTY-DC01 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\MISTY-DC01
      Starting test: Replications
         ......................... MISTY-DC01 passed test Replications


   Running partition tests on : ForestDnsZones

   Running partition tests on : DomainDnsZones

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : misty

   Running enterprise tests on : misty.home

C:\Windows\system32>

ok, so can you again run DCDIAG /FIX with an elevated command prompt.  Just do one server at the moment and post the complete output please.

I just restarted both, I'll post the output in a moment.

DC's are back online when I run the /fix command I cant get all the output on the screen. I increased the buffer in the dos window but it still goes past it. It appears to be failing the same as before.

if you run DCDIAG /FIX > C:\DCDIAG.TXT

This will send the output to a text file.  Please upload the text file.

Here is part of it that I can grab

    Starting test: NetLogons
       Unable to connect to the NETLOGON share! (\\MISTY-DC01\netlogon)
       [MISTY-DC01] An net use or LsaPolicy operation failed with error 67,
       The network name cannot be found..
       ......................... MISTY-DC01 failed test NetLogons

OK, there could be an easy fix for all this.

Are the 2008 DC's doing anything else?  Are they just DC's and DNS servers?

Directory Server Diagnosis


Performing initial setup:

   Trying to find home server...

   Home Server = Misty-DC01

   * Identified AD Forest.
   Done gathering initial info.


Doing initial required tests

   
   Testing server: Default-First-Site-Name\MISTY-DC01

      Starting test: Connectivity

         ......................... MISTY-DC01 passed test Connectivity



Doing primary tests

   
   Testing server: Default-First-Site-Name\MISTY-DC01

      Starting test: Advertising

         Warning: DsGetDcName returned information for \\mistymail.misty.home,

         when we were trying to reach MISTY-DC01.

         SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.

         ......................... MISTY-DC01 failed test Advertising

      Starting test: FrsEvent

         There are warning or error events within the last 24 hours after the

         SYSVOL has been shared.  Failing SYSVOL replication problems may cause

         Group Policy problems.
         ......................... MISTY-DC01 passed test FrsEvent

      Starting test: DFSREvent

         ......................... MISTY-DC01 passed test DFSREvent

      Starting test: SysVolCheck

         ......................... MISTY-DC01 passed test SysVolCheck

      Starting test: KccEvent

         ......................... MISTY-DC01 passed test KccEvent

      Starting test: KnowsOfRoleHolders

         [MISTY-DC02] DsBindWithSpnEx() failed with error 5,

         Access is denied..
         Warning: MISTY-DC02 is the Domain Owner, but is not responding to DS

         RPC Bind.

         [MISTY-DC02] LDAP bind failed with error 8341,

         A directory service error has occurred..
         Warning: MISTY-DC02 is the Domain Owner, but is not responding to LDAP

         Bind.

         Warning: MISTY-DC02 is the PDC Owner, but is not responding to DS RPC

         Bind.

         Warning: MISTY-DC02 is the PDC Owner, but is not responding to LDAP

         Bind.

         Warning: MISTY-DC02 is the Rid Owner, but is not responding to DS RPC

         Bind.

         Warning: MISTY-DC02 is the Rid Owner, but is not responding to LDAP

         Bind.

         Warning: MISTY-DC02 is the Infrastructure Update Owner, but is not

         responding to DS RPC Bind.

         Warning: MISTY-DC02 is the Infrastructure Update Owner, but is not

         responding to LDAP Bind.

         ......................... MISTY-DC01 failed test KnowsOfRoleHolders

      Starting test: MachineAccount

         ......................... MISTY-DC01 passed test MachineAccount

      Starting test: NCSecDesc

         ......................... MISTY-DC01 passed test NCSecDesc

      Starting test: NetLogons

         Unable to connect to the NETLOGON share! (\\MISTY-DC01\netlogon)

         [MISTY-DC01] An net use or LsaPolicy operation failed with error 67,

         The network name cannot be found..

         ......................... MISTY-DC01 failed test NetLogons

      Starting test: ObjectsReplicated

         ......................... MISTY-DC01 passed test ObjectsReplicated

      Starting test: Replications

         [MISTYMAIL] DsBindWithSpnEx() failed with error 5,

         Access is denied..
         [Replications Check,MISTY-DC01] A recent replication attempt failed:

            From MISTYMAIL to MISTY-DC01

            Naming Context: CN=Schema,CN=Configuration,DC=misty,DC=home

            The replication generated an error (1908):

            Could not find the domain controller for this domain.

            The failure occurred at 2011-04-30 07:07:39.

            The last success occurred at 2011-04-30 06:52:58.

            2 failures have occurred since the last success.

            Kerberos Error.

            A KDC was not found to authenticate the call.

            Check that sufficient domain controllers are available.

         [Replications Check,MISTY-DC01] A recent replication attempt failed:

            From MISTY-DC02 to MISTY-DC01

            Naming Context: CN=Schema,CN=Configuration,DC=misty,DC=home

            The replication generated an error (1908):

            Could not find the domain controller for this domain.

            The failure occurred at 2011-04-30 07:09:52.

            The last success occurred at 2011-04-30 06:52:58.

            2 failures have occurred since the last success.

            Kerberos Error.

            A KDC was not found to authenticate the call.

            Check that sufficient domain controllers are available.

         [Replications Check,MISTY-DC01] A recent replication attempt failed:

            From MISTY-MAS to MISTY-DC01

            Naming Context: CN=Schema,CN=Configuration,DC=misty,DC=home

            The replication generated an error (1908):

            Could not find the domain controller for this domain.

            The failure occurred at 2011-04-30 07:12:17.

            The last success occurred at 2011-04-30 06:52:58.

            2 failures have occurred since the last success.

            Kerberos Error.

            A KDC was not found to authenticate the call.

            Check that sufficient domain controllers are available.

         [MISTY-MAS] DsBindWithSpnEx() failed with error 5,

         Access is denied..
         ......................... MISTY-DC01 failed test Replications

      Starting test: RidManager

         ......................... MISTY-DC01 failed test RidManager

      Starting test: Services

         ......................... MISTY-DC01 passed test Services

      Starting test: SystemLog

         A warning event occurred.  EventID: 0x8000001D

            Time Generated: 04/30/2011   06:56:37

            Event String:

            The Key Distribution Center (KDC) cannot find a suitable certificate to use for smart card logons, or the KDC certificate could not be verified. Smart card logon may not function correctly if this problem is not resolved. To correct this problem, either verify the existing KDC certificate using certutil.exe or enroll for a new KDC certificate.

         A warning event occurred.  EventID: 0x00000C18

            Time Generated: 04/30/2011   06:56:44

            Event String:

            The primary Domain Controller for this domain could not be located.

         A warning event occurred.  EventID: 0x000003F6

            Time Generated: 04/30/2011   06:57:08

            Event String:

            Name resolution for the name _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs.misty.home timed out after none of the configured DNS servers responded.

         An error event occurred.  EventID: 0xC00038D6

            Time Generated: 04/30/2011   06:57:09

            Event String:

            The DFS Namespace service could not initialize cross forest trust information on this domain controller, but it will periodically retry the operation. The return code is in the record data.

         A warning event occurred.  EventID: 0x000003F6

            Time Generated: 04/30/2011   06:57:35

            Event String:

            Name resolution for the name misty.home timed out after none of the configured DNS servers responded.

         An error event occurred.  EventID: 0xC00038D6

            Time Generated: 04/30/2011   06:57:36

            Event String:

            The DFS Namespace service could not initialize cross forest trust information on this domain controller, but it will periodically retry the operation. The return code is in the record data.

         An error event occurred.  EventID: 0xC00038D6

            Time Generated: 04/30/2011   06:58:03

            Event String:

            The DFS Namespace service could not initialize cross forest trust information on this domain controller, but it will periodically retry the operation. The return code is in the record data.

         A warning event occurred.  EventID: 0x00000081

            Time Generated: 04/30/2011   06:58:08

            Event String:

            NtpClient was unable to set a domain peer to use as a time source because of discovery error. NtpClient will try again in 3473457 minutes and double the reattempt interval thereafter. The error was: The entry is not found. (0x800706E1)

         A warning event occurred.  EventID: 0x00000081

            Time Generated: 04/30/2011   06:58:10

            Event String:

            NtpClient was unable to set a domain peer to use as a time source because of discovery error. NtpClient will try again in 3473457 minutes and double the reattempt interval thereafter. The error was: The entry is not found. (0x800706E1)

         An error event occurred.  EventID: 0xC00038D6

            Time Generated: 04/30/2011   06:58:30

            Event String:

            The DFS Namespace service could not initialize cross forest trust information on this domain controller, but it will periodically retry the operation. The return code is in the record data.

         An error event occurred.  EventID: 0xC00038D6

            Time Generated: 04/30/2011   06:58:57

            Event String:

            The DFS Namespace service could not initialize cross forest trust information on this domain controller, but it will periodically retry the operation. The return code is in the record data.

         An error event occurred.  EventID: 0xC00038D6

            Time Generated: 04/30/2011   06:59:24

            Event String:

            The DFS Namespace service could not initialize cross forest trust information on this domain controller, but it will periodically retry the operation. The return code is in the record data.

         An error event occurred.  EventID: 0xC00038D6

            Time Generated: 04/30/2011   06:59:51

            Event String:

            The DFS Namespace service could not initialize cross forest trust information on this domain controller, but it will periodically retry the operation. The return code is in the record data.

         A warning event occurred.  EventID: 0x000727AA

            Time Generated: 04/30/2011   07:00:07

            Event String:

            The WinRM service failed to create the following SPNs: WSMAN/Misty-DC01.misty.home; WSMAN/Misty-DC01.


         An error event occurred.  EventID: 0xC00038D6

            Time Generated: 04/30/2011   07:00:18

            Event String:

            The DFS Namespace service could not initialize cross forest trust information on this domain controller, but it will periodically retry the operation. The return code is in the record data.

         An error event occurred.  EventID: 0xC00038D6

            Time Generated: 04/30/2011   07:00:45

            Event String:

            The DFS Namespace service could not initialize cross forest trust information on this domain controller, but it will periodically retry the operation. The return code is in the record data.

         An error event occurred.  EventID: 0xC00038D6

            Time Generated: 04/30/2011   07:01:12

            Event String:

            The DFS Namespace service could not initialize cross forest trust information on this domain controller, but it will periodically retry the operation. The return code is in the record data.

         An error event occurred.  EventID: 0xC00038D6

            Time Generated: 04/30/2011   07:01:39

            Event String:

            The DFS Namespace service could not initialize cross forest trust information on this domain controller, but it will periodically retry the operation. The return code is in the record data.

         An error event occurred.  EventID: 0x00000469

            Time Generated: 04/30/2011   07:01:50

            Event String:

            The processing of Group Policy failed because of lack of network connectivity to a domain controller. This may be a transient condition. A success message would be generated once the machine gets connected to the domain controller and Group Policy has succesfully processed. If you do not see a success message for several hours, then contact your administrator.

         A warning event occurred.  EventID: 0x00001695

            Time Generated: 04/30/2011   07:03:01

            Event String:

            Dynamic registration or deletion of one or more DNS records associated with DNS domain 'misty.home.' failed.  These records are used by other computers to locate this server as a domain controller (if the specified domain is an Active Directory domain) or as an LDAP server (if the specified domain is an application partition).  


         An error event occurred.  EventID: 0xC000042B

            Time Generated: 04/30/2011   07:16:25

            Event String:

            The terminal server cannot register 'TERMSRV' Service Principal Name to be used for server authentication. The following error occured: The system detected a possible attempt to compromise security. Please ensure that you can contact the server that authenticated you.


         An error event occurred.  EventID: 0x40000005

            Time Generated: 04/30/2011   07:17:13

            Event String:

            The kerberos client received a KRB_AP_ERR_TKT_NYV error from the server misty-dc01$. This indicates that the ticket used against that server is not yet valid (in relationship to that server time).  Contact your system administrator to make sure the client and server times are in sync, and that the KDC in realm MISTY.HOME is in sync with the KDC in the client realm.

         A warning event occurred.  EventID: 0x00001695

            Time Generated: 04/30/2011   07:17:15

            Event String:

            Dynamic registration or deletion of one or more DNS records associated with DNS domain 'DomainDnsZones.misty.home.' failed.  These records are used by other computers to locate this server as a domain controller (if the specified domain is an Active Directory domain) or as an LDAP server (if the specified domain is an application partition).  


         A warning event occurred.  EventID: 0x00001695

            Time Generated: 04/30/2011   07:17:51

            Event String:

            Dynamic registration or deletion of one or more DNS records associated with DNS domain 'ForestDnsZones.misty.home.' failed.  These records are used by other computers to locate this server as a domain controller (if the specified domain is an Active Directory domain) or as an LDAP server (if the specified domain is an application partition).  


         An error event occurred.  EventID: 0x40000005

            Time Generated: 04/30/2011   07:17:55

            Event String:

            The kerberos client received a KRB_AP_ERR_TKT_NYV error from the server misty-dc02$. This indicates that the ticket used against that server is not yet valid (in relationship to that server time).  Contact your system administrator to make sure the client and server times are in sync, and that the KDC in realm MISTY.HOME is in sync with the KDC in the client realm.

         An error event occurred.  EventID: 0x0000168E

            Time Generated: 04/30/2011   07:18:56

            Event String:

            The dynamic registration of the DNS record '_ldap._tcp.gc._msdcs.misty.home. 600 IN SRV 0 100 3268 Misty-DC01.misty.home.' failed on the following DNS server:  


         An error event occurred.  EventID: 0x000003EE

            Time Generated: 04/30/2011   07:18:58

            Event String:

            The processing of Group Policy failed. Windows could not authenticate to the Active Directory service on a domain controller. (LDAP Bind function call failed). Look in the details tab for error code and description.

         A warning event occurred.  EventID: 0x000003F6

            Time Generated: 04/30/2011   07:18:58

            Event String:

            Name resolution for the name secure.logmein.com timed out after none of the configured DNS servers responded.

         A warning event occurred.  EventID: 0x00000081

            Time Generated: 04/30/2011   07:19:00

            Event String:

            NtpClient was unable to set a domain peer to use as a time source because of discovery error. NtpClient will try again in 3473457 minutes and double the reattempt interval thereafter. The error was: The entry is not found. (0x800706E1)

         A warning event occurred.  EventID: 0x00000081

            Time Generated: 04/30/2011   07:19:06

            Event String:

            NtpClient was unable to set a domain peer to use as a time source because of discovery error. NtpClient will try again in 3473457 minutes and double the reattempt interval thereafter. The error was: The entry is not found. (0x800706E1)

         An error event occurred.  EventID: 0x00000457

            Time Generated: 04/30/2011   07:19:12

            Event String:

            Driver Amyuni Document Converter 2.50 required for printer Extended Solutions PDF Converter is unknown. Contact the administrator to install the driver before you log in again.

         ......................... MISTY-DC01 failed test SystemLog

      Starting test: VerifyReferences

         ......................... MISTY-DC01 passed test VerifyReferences

   
   
   Running partition tests on : ForestDnsZones

      Starting test: CheckSDRefDom

         ......................... ForestDnsZones passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... ForestDnsZones passed test

         CrossRefValidation

   
   Running partition tests on : DomainDnsZones

      Starting test: CheckSDRefDom

         ......................... DomainDnsZones passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... DomainDnsZones passed test

         CrossRefValidation

   
   Running partition tests on : Schema

      Starting test: CheckSDRefDom

         ......................... Schema passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... Schema passed test CrossRefValidation

   
   Running partition tests on : Configuration

      Starting test: CheckSDRefDom

         ......................... Configuration passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... Configuration passed test CrossRefValidation

   
   Running partition tests on : misty

      Starting test: CheckSDRefDom

         ......................... misty passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... misty passed test CrossRefValidation

   
   Running enterprise tests on : misty.home

      Starting test: LocatorCheck

         [MISTYMAIL] LDAP bind failed with error 8341,

         A directory service error has occurred..
         ......................... misty.home passed test LocatorCheck

      Starting test: Intersite

         ......................... misty.home passed test Intersite

Is it possible I missed some of the domain prep steps when joining the new DC's? If that were the cause AD wouldnt have let me add them would it?

no it wouldn't have.

Again, what other roles are these servers performing?

Topology: Server 2003 DC with exchange installed on it (bad idea) trying to normalize the environment. So I installed two server 2008 R2 DC's and exchange 2010 on seperate servers. New DC's only have AD and DNS on them. exchange only has exchange. when shutting down 2003 DC I'm getting GC contact failure from OWA.

OK, suggestion then.

Follow these steps:

-> on the server you configured them all to use for DNS, open the DNS console, on the type of zone click Change and uncheck the box to store the zone in Active Directory.

DO NOT Change the DNS configuration on the 3 Domain Controllers.

This will make the zone a regular zone and not an Active Directory Zone.

-> Transfer all 5 FSMO roles back to the 2003 server.
-> Make sure the 2 2008 Servers are not Global Catalog servers.
-> 1 by 1 DCPROMO the 2008 servers so they are no longer Domain Controllers.

As long as the above all goes well.

On the 2008 Server that is not the DNS server they are all pointing to, remove it from the domain, delete the computer account and then re-join it to the domain.

Run DCPROMO to make it a domain controller again.

Once done, check the DC health using DCDIAG, what do we get?

We don't want to move on until this DC is healthy.

It will take me some time to complete this, I will post back when complete. thanks for the assistance.

that's OK, it's time for me to head to bed anyway :)

What do you think about starting over from scratch with a new Server 2008 R2 DC?

When attempting to add the new 2008 DC I get this message, this new DC is pointing to the other 2008 DC for DNS. I recall I got these errors with the other DC installations (2008) as well, they were pointing to the server 2003 DC for DNS at that time.
dns-error.png

Interesting.

So both the 2003 server and the one you are trying to promote are both pointing to the other 2008 server for DNS?

In the DNS console.  Right click the forward lookup zone and select properties.  Make sure the update is set to secure and insecure.

Check the zone, does it have any subfolders? Does it have DNS records? Can you post a screen shot?

Correct, both 2003 and 2008 DC's are pointing to the 2008 server for DNS. I believe that was one of your diagnostics steps previously. We wanted to rule out any DNS wackiness.

The zone looks fully enumerated and healthy.  

Notice the Updates are set to only secure, I'll change this.
dns-config.png

Can you try doing the ADprep once

Sure I'll give that a try.

This is on the brand new DC I just installed. Same issue.


Performing initial setup:
   Trying to find home server...
   Home Server = Misty-dc03
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\MISTY-DC03
      Starting test: Connectivity
         ......................... MISTY-DC03 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\MISTY-DC03
      Starting test: Advertising
         Warning: DsGetDcName returned information for \\mistymail.misty.home,
         when we were trying to reach MISTY-DC03.
         SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.
         ......................... MISTY-DC03 failed test Advertising
      Starting test: FrsEvent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... MISTY-DC03 passed test FrsEvent
      Starting test: DFSREvent
         ......................... MISTY-DC03 passed test DFSREvent
      Starting test: SysVolCheck
         ......................... MISTY-DC03 passed test SysVolCheck
      Starting test: KccEvent
         ......................... MISTY-DC03 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         ......................... MISTY-DC03 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         ......................... MISTY-DC03 passed test MachineAccount
      Starting test: NCSecDesc
         ......................... MISTY-DC03 passed test NCSecDesc
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\MISTY-DC03\netlogon)
         [MISTY-DC03] An net use or LsaPolicy operation failed with error 67,
         The network name cannot be found..
         ......................... MISTY-DC03 failed test NetLogons
      Starting test: ObjectsReplicated
         ......................... MISTY-DC03 passed test ObjectsReplicated
      Starting test: Replications
         REPLICATION LATENCY WARNING
         ERROR: Expected notification link is missing.
         Source MISTYMAIL
         Replication of new changes along this path will be delayed.
         This problem should self-correct on the next periodic sync.
         REPLICATION LATENCY WARNING

what do you think about doing this:

Resolution 2

   1. Copy the contents from Ntfrs_Preexisting folder to the %Windows%Sysvol\Sysvol\Domain Name folder.
   2. Start Registry Editor. Locate to the following subkey:
      HKLM\SYSTEM\CurrentControlSet\Services\Ntfrs\Parameters\Backup/Restore\Process at Startup
      .
   3. Set the value of the BurFlags registry entry to D4.

      Note Before you change the registry, see the "More Information" section.
   4. Restart the Ntfrs service, and then wait until the Sysvol and the Netlogon folders are shared.

How did you join that to the domain?

What DNS settings did you give it?

The sysvol share is not present on any of the new 2008 DC's so it seems they are not promoting themselves properly.

That was going to be my next suggestion.

But I am concerned that DNS still may not be working properly.

All DNS is pointing to the first server 2008 DC.

I'm confident something is not promoting correctly, is the easiest way to do that burflags fix? or should I do something else. the Burflags seems to only apply to 2003 DC's will it work in 08?

Yep, I'm in process of doing that now. I'll let you know what the outcome is. Thanks.

this worked, thanks for all your help!

Currently your DC is not fully promoted to a DC if you run the burflag this will sync the two DCs then create the SYSVOL which why the DC is not fully functioning.

Took backup of the policies and script folders from both the servers from c:\Windows\Sysvol\domain
Stopped NTFRS service on both DCs.
Made one of the DC authoritative server by modifying registry setting : Navigate to registry HKLM\System\CCS\Services\NTFRS\Parameters\CumlativeReplicaSets and Set the Burflags value to D4. This should be done with server which has the Updated information available or correct data.
Went to other DC and made that Non-authoritative by navigating to same registry location HKLM\System\CCS\Services\NTFRS\Parameters\CumlativeReplicaSets and Set the Burflags value to D2.
Restarted Ntfrs service on both servers and forced replication to see event 13516 in event viewer for FRS.