DsReplicaSync() failed with status 1753 (0x6d9)

I did a repadmin.exe /showrepl dc* /verbose /all /intersite > c:\repl.txt

and this was the outcome:
Repadmin experienced the following error trying to resolve the DC_NAME: dc*

Error: An error occured:

    Win32 Error 8419(0x20e3): The DSA object could not be found.

Here is an output of DCDIAG /V /C /D /E /s:DC1 > c:\dcdiag.log:
Command Line: "dcdiag.exe /V /C /D /E /s:DC1"

Domain Controller Diagnosis

Performing initial setup:
   * Connecting to directory service on server DC1.
   DC1.currentTime = 20110725092533.0Z
   DC1.highestCommittedUSN = 5447743
   DC1.isSynchronized = 1
   DC1.isGlobalCatalogReady = 1
   Failure Analysis: DC1 ... OK.
   [DC1] Directory Binding Error 1753:
   There are no more endpoints available from the endpoint mapper.
   This may limit some of the tests that can be performed.
   * Collecting site info.
   * Identifying all servers.
   DC1.currentTime = 20110725092615.0Z
   DC1.highestCommittedUSN = 5447743
   DC1.isSynchronized = 1
   DC1.isGlobalCatalogReady = 1
   * Identifying all NC cross-refs.
   * Found 3 DC(s). Testing 3 of them.
   Done gathering initial info.


===============================================Printing out pDsInfo

GLOBAL:
      ulNumServers=3
      pszRootDomain=DOMAIN.local
      pszNC=
      pszRootDomainFQDN=DC=DOMAIN,DC=local
      pszConfigNc=CN=Configuration,DC=DOMAIN,DC=local
      pszPartitionsDn=CN=Partitions,CN=Configuration,DC=DOMAIN,DC=local
      iSiteOptions=0
      dwTombstoneLifeTimeDays=60

      dwForestBehaviorVersion=2

      HomeServer=0, DC1

      SERVER: pServer[0].pszName=DC1
            pServer[0].pszGuidDNSName=86b881a6-4b3e-424b-adca-ad1aff078296._msdcs.DOMAIN.local
            pServer[0].pszDNSName=DC1.DOMAIN.local
            pServer[0].pszDn=CN=NTDS Settings,CN=DC1,CN=Servers,CN=Main Office,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pServer[0].pszComputerAccountDn=CN=DC1,OU=Domain Controllers,DC=DOMAIN,DC=local
            pServer[0].uuidObjectGuid=86b881a6-4b3e-424b-adca-ad1aff078296
            pServer[0].uuidInvocationId=86b881a6-4b3e-424b-adca-ad1aff078296
            pServer[0].iSite=0 (Main Office)
            pServer[0].iOptions=1
            pServer[0].ftLocalAcquireTime=e7273b00 01cc4aac

            pServer[0].ftRemoteConnectTime=e6911580 01cc4aac

            pServer[0].ppszMasterNCs:
                  ppszMasterNCs[0]=DC=ForestDnsZones,DC=DOMAIN,DC=local
                  ppszMasterNCs[1]=DC=DomainDnsZones,DC=DOMAIN,DC=local
                  ppszMasterNCs[2]=CN=Schema,CN=Configuration,DC=DOMAIN,DC=local
                  ppszMasterNCs[3]=CN=Configuration,DC=DOMAIN,DC=local
                  ppszMasterNCs[4]=DC=DOMAIN,DC=local

      SERVER: pServer[1].pszName=BranchDC
            pServer[1].pszGuidDNSName=e641d3dd-4581-4e92-81ef-8fd161c6d016._msdcs.DOMAIN.local
            pServer[1].pszDNSName=BranchDC.DOMAIN.local
            pServer[1].pszDn=CN=NTDS Settings,CN=BranchDC,CN=Servers,CN=Branch Office,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pServer[1].pszComputerAccountDn=CN=BranchDC,OU=Domain Controllers,DC=DOMAIN,DC=local
            pServer[1].uuidObjectGuid=e641d3dd-4581-4e92-81ef-8fd161c6d016
            pServer[1].uuidInvocationId=472f6f39-1e82-40b7-ab76-762fd0a8fae5
            pServer[1].iSite=1 (Branch Office)
            pServer[1].iOptions=1
            pServer[1].ftLocalAcquireTime=00000000 00000000

            pServer[1].ftRemoteConnectTime=00000000 00000000

            pServer[1].ppszMasterNCs:
                  ppszMasterNCs[0]=DC=ForestDnsZones,DC=DOMAIN,DC=local
                  ppszMasterNCs[1]=DC=DomainDnsZones,DC=DOMAIN,DC=local
                  ppszMasterNCs[2]=CN=Schema,CN=Configuration,DC=DOMAIN,DC=local
                  ppszMasterNCs[3]=CN=Configuration,DC=DOMAIN,DC=local
                  ppszMasterNCs[4]=DC=DOMAIN,DC=local

      SERVER: pServer[2].pszName=DC2
            pServer[2].pszGuidDNSName=b1db76d8-5b7f-4e88-b9f9-a3fba490bfba._msdcs.DOMAIN.local
            pServer[2].pszDNSName=DC2.DOMAIN.local
            pServer[2].pszDn=CN=NTDS Settings,CN=DC2,CN=Servers,CN=Main Office,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pServer[2].pszComputerAccountDn=CN=DC2,OU=Domain Controllers,DC=DOMAIN,DC=local
            pServer[2].uuidObjectGuid=b1db76d8-5b7f-4e88-b9f9-a3fba490bfba
            pServer[2].uuidInvocationId=05b1c28f-ebc1-448a-ab6e-b0f71cbf8211
            pServer[2].iSite=0 (Main Office)
            pServer[2].iOptions=1
            pServer[2].ftLocalAcquireTime=00000000 00000000

            pServer[2].ftRemoteConnectTime=00000000 00000000

            pServer[2].ppszMasterNCs:
                  ppszMasterNCs[0]=DC=ForestDnsZones,DC=DOMAIN,DC=local
                  ppszMasterNCs[1]=DC=DomainDnsZones,DC=DOMAIN,DC=local
                  ppszMasterNCs[2]=CN=Schema,CN=Configuration,DC=DOMAIN,DC=local
                  ppszMasterNCs[3]=CN=Configuration,DC=DOMAIN,DC=local
                  ppszMasterNCs[4]=DC=DOMAIN,DC=local

      SITES:  pSites[0].pszName=Main Office
            pSites[0].pszSiteSettings=CN=NTDS Site Settings,CN=Main Office,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pSites[0].pszISTG=CN=NTDS Settings,CN=DC1,CN=Servers,CN=Main Office,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pSites[0].iSiteOption=0

            pSites[0].cServers=2

      SITES:  pSites[1].pszName=Branch Office
            pSites[1].pszSiteSettings=CN=NTDS Site Settings,CN=Branch Office,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pSites[1].pszISTG=CN=NTDS Settings,CN=BranchDC,CN=Servers,CN=Branch Office,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pSites[1].iSiteOption=0

            pSites[1].cServers=1

      NC:     pNCs[0].pszName=ForestDnsZones
            pNCs[0].pszDn=DC=ForestDnsZones,DC=DOMAIN,DC=local

                  pNCs[0].aCrInfo[0].dwFlags=0x00000201
                  pNCs[0].aCrInfo[0].pszDn=CN=d35fa9b5-b39d-43d7-a886-8d0758f5ba64,CN=Partitions,CN=Configuration,DC=DOMAIN,DC=local
                  pNCs[0].aCrInfo[0].pszDnsRoot=ForestDnsZones.DOMAIN.local
                  pNCs[0].aCrInfo[0].iSourceServer=0
                  pNCs[0].aCrInfo[0].pszSourceServer=(null)
                  pNCs[0].aCrInfo[0].ulSystemFlags=0x00000005
                  pNCs[0].aCrInfo[0].bEnabled=TRUE
                  pNCs[0].aCrInfo[0].ftWhenCreated=00000000 00000000                   pNCs[0].aCrInfo[0].pszSDReferenceDomain=(null)
                  pNCs[0].aCrInfo[0].pszNetBiosName=(null)
                  pNCs[0].aCrInfo[0].cReplicas=-1
                  pNCs[0].aCrInfo[0].aszReplicas=


      NC:     pNCs[1].pszName=DomainDnsZones
            pNCs[1].pszDn=DC=DomainDnsZones,DC=DOMAIN,DC=local

                  pNCs[1].aCrInfo[0].dwFlags=0x00000201
                  pNCs[1].aCrInfo[0].pszDn=CN=3e9ef1d8-1909-43a2-9b4d-819166c5e9e2,CN=Partitions,CN=Configuration,DC=DOMAIN,DC=local
                  pNCs[1].aCrInfo[0].pszDnsRoot=DomainDnsZones.DOMAIN.local
                  pNCs[1].aCrInfo[0].iSourceServer=0
                  pNCs[1].aCrInfo[0].pszSourceServer=(null)
                  pNCs[1].aCrInfo[0].ulSystemFlags=0x00000005
                  pNCs[1].aCrInfo[0].bEnabled=TRUE
                  pNCs[1].aCrInfo[0].ftWhenCreated=00000000 00000000                   pNCs[1].aCrInfo[0].pszSDReferenceDomain=(null)
                  pNCs[1].aCrInfo[0].pszNetBiosName=(null)
                  pNCs[1].aCrInfo[0].cReplicas=-1
                  pNCs[1].aCrInfo[0].aszReplicas=


      NC:     pNCs[2].pszName=Schema
            pNCs[2].pszDn=CN=Schema,CN=Configuration,DC=DOMAIN,DC=local

                  pNCs[2].aCrInfo[0].dwFlags=0x00000201
                  pNCs[2].aCrInfo[0].pszDn=CN=Enterprise Schema,CN=Partitions,CN=Configuration,DC=DOMAIN,DC=local
                  pNCs[2].aCrInfo[0].pszDnsRoot=DOMAIN.local
                  pNCs[2].aCrInfo[0].iSourceServer=0
                  pNCs[2].aCrInfo[0].pszSourceServer=(null)
                  pNCs[2].aCrInfo[0].ulSystemFlags=0x00000001
                  pNCs[2].aCrInfo[0].bEnabled=TRUE
                  pNCs[2].aCrInfo[0].ftWhenCreated=00000000 00000000                   pNCs[2].aCrInfo[0].pszSDReferenceDomain=(null)
                  pNCs[2].aCrInfo[0].pszNetBiosName=(null)
                  pNCs[2].aCrInfo[0].cReplicas=-1
                  pNCs[2].aCrInfo[0].aszReplicas=


      NC:     pNCs[3].pszName=Configuration
            pNCs[3].pszDn=CN=Configuration,DC=DOMAIN,DC=local

                  pNCs[3].aCrInfo[0].dwFlags=0x00000201
                  pNCs[3].aCrInfo[0].pszDn=CN=Enterprise Configuration,CN=Partitions,CN=Configuration,DC=DOMAIN,DC=local
                  pNCs[3].aCrInfo[0].pszDnsRoot=DOMAIN.local
                  pNCs[3].aCrInfo[0].iSourceServer=0
                  pNCs[3].aCrInfo[0].pszSourceServer=(null)
                  pNCs[3].aCrInfo[0].ulSystemFlags=0x00000001
                  pNCs[3].aCrInfo[0].bEnabled=TRUE
                  pNCs[3].aCrInfo[0].ftWhenCreated=00000000 00000000                   pNCs[3].aCrInfo[0].pszSDReferenceDomain=(null)
                  pNCs[3].aCrInfo[0].pszNetBiosName=(null)
                  pNCs[3].aCrInfo[0].cReplicas=-1
                  pNCs[3].aCrInfo[0].aszReplicas=


      NC:     pNCs[4].pszName=DOMAIN
            pNCs[4].pszDn=DC=DOMAIN,DC=local

                  pNCs[4].aCrInfo[0].dwFlags=0x00000201
                  pNCs[4].aCrInfo[0].pszDn=CN=DOMAIN,CN=Partitions,CN=Configuration,DC=DOMAIN,DC=local
                  pNCs[4].aCrInfo[0].pszDnsRoot=DOMAIN.local
                  pNCs[4].aCrInfo[0].iSourceServer=0
                  pNCs[4].aCrInfo[0].pszSourceServer=(null)
                  pNCs[4].aCrInfo[0].ulSystemFlags=0x00000003
                  pNCs[4].aCrInfo[0].bEnabled=TRUE
                  pNCs[4].aCrInfo[0].ftWhenCreated=00000000 00000000                   pNCs[4].aCrInfo[0].pszSDReferenceDomain=(null)
                  pNCs[4].aCrInfo[0].pszNetBiosName=(null)
                  pNCs[4].aCrInfo[0].cReplicas=-1
                  pNCs[4].aCrInfo[0].aszReplicas=


      5 NC TARGETS: ForestDnsZones, DomainDnsZones, Schema, Configuration, DOMAIN,
      3 TARGETS: DC1, BranchDC, DC2,

=============================================Done Printing pDsInfo

Doing initial required tests
   
   Testing server: Main Office\DC1
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         Failure Analysis: DC1 ... OK.
         * Active Directory RPC Services Check
         [DC1] DsBindWithSpnEx() failed with error 1753,
         There are no more endpoints available from the endpoint mapper..
         Printing RPC Extended Error Info:
         Error Record 1, ProcessID is 1056 (DcDiag)        
            System Time is: 7/25/2011 9:26:42:812
            Generating component is 2 (RPC runtime)
            Status is 1753: There are no more endpoints available from the endpoint mapper.

            Detection location is 500
            NumberOfParameters is 4
            Unicode string: ncacn_ip_tcp
            Unicode string: 86b881a6-4b3e-424b-adca-ad1aff078296._msdcs.DOMAIN.local
            Long val: -481213899
            Long val: 65537
         Error Record 2, ProcessID is 1056 (DcDiag)        
            System Time is: 7/25/2011 9:26:42:812
            Generating component is 2 (RPC runtime)
            Status is 1722: The RPC server is unavailable.

            Detection location is 761
            NumberOfParameters is 1
            Unicode string: 1025
         Error Record 3, ProcessID is 1056 (DcDiag)        
            System Time is: 7/25/2011 9:26:42:812
            Generating component is 8 (winsock)
            Status is 1722: The RPC server is unavailable.

            Detection location is 323
         Error Record 4, ProcessID is 1056 (DcDiag)        
            System Time is: 7/25/2011 9:26:42:812
            Generating component is 8 (winsock)
            Status is 1237: The operation could not be completed. A retry should be performed.

            Detection location is 313
         Error Record 5, ProcessID is 1056 (DcDiag)        
            System Time is: 7/25/2011 9:26:42:812
            Generating component is 8 (winsock)
            Status is 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

            Detection location is 311
            NumberOfParameters is 3
            Long val: 1025
            Pointer val: 0
            Pointer val: 0
         Error Record 6, ProcessID is 1056 (DcDiag)        
            System Time is: 7/25/2011 9:26:42:812
            Generating component is 8 (winsock)
            Status is 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

            Detection location is 318
         ......................... DC1 failed test Connectivity
   
   Testing server: Branch Office\BranchDC
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         BranchDC.currentTime = 20110725092643.0Z
         BranchDC.highestCommittedUSN = 307261
         BranchDC.isSynchronized = 1
         BranchDC.isGlobalCatalogReady = 1
         Failure Analysis: BranchDC ... OK.
         * Active Directory RPC Services Check
         ......................... BranchDC passed test Connectivity
   
   Testing server: Main Office\DC2
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         [DC2] LDAP search failed with error 58,
         The specified server cannot perform the requested operation..
         ***Error: The machine, DC2 could not be contacted, because of a

         bad net  response.  Check to make sure that this machine is a Domain

         Controller.
         ......................... DC2 failed test Connectivity

Doing primary tests
   
   Testing server: Main Office\DC1
      Skipping all tests, because server DC1 is
      not responding to directory service requests
   
   Testing server: Branch Office\BranchDC
      Starting test: Replications
         * Replications Check
         DC=ForestDnsZones,DC=DOMAIN,DC=local has 6 cursors.
         [Replications Check,BranchDC] A recent replication attempt failed:
            From DC1 to BranchDC
            Naming Context: DC=ForestDnsZones,DC=DOMAIN,DC=local
            The replication generated an error (1256):
            The remote system is not available. For information about network troubleshooting, see Windows Help.
            The failure occurred at 2011-07-25 04:48:58.
            The last success occurred at 2011-07-22 04:58:50.
            1 failures have occurred since the last success.
         DC=DomainDnsZones,DC=DOMAIN,DC=local has 6 cursors.
         [Replications Check,BranchDC] A recent replication attempt failed:
            From DC1 to BranchDC
            Naming Context: DC=DomainDnsZones,DC=DOMAIN,DC=local
            The replication generated an error (1256):
            The remote system is not available. For information about network troubleshooting, see Windows Help.
            The failure occurred at 2011-07-25 04:48:58.
            The last success occurred at 2011-07-22 04:58:50.
            1 failures have occurred since the last success.
         CN=Schema,CN=Configuration,DC=DOMAIN,DC=local has 9 cursors.
         [Replications Check,BranchDC] A recent replication attempt failed:
            From DC1 to BranchDC
            Naming Context: CN=Schema,CN=Configuration,DC=DOMAIN,DC=local
            The replication generated an error (1753):
            There are no more endpoints available from the endpoint mapper.
            The failure occurred at 2011-07-25 04:50:01.
            The last success occurred at 2011-07-22 04:58:50.
            1 failures have occurred since the last success.
            The directory on DC1 is in the process.
            of starting up or shutting down, and is not available.
            Verify machine is not hung during boot.
         CN=Configuration,DC=DOMAIN,DC=local has 9 cursors.
         [Replications Check,BranchDC] A recent replication attempt failed:
            From DC1 to BranchDC
            Naming Context: CN=Configuration,DC=DOMAIN,DC=local
            The replication generated an error (1753):
            There are no more endpoints available from the endpoint mapper.
            The failure occurred at 2011-07-25 04:49:19.
            The last success occurred at 2011-07-22 04:58:49.
            1 failures have occurred since the last success.
            The directory on DC1 is in the process.
            of starting up or shutting down, and is not available.
            Verify machine is not hung during boot.
         DC=DOMAIN,DC=local has 9 cursors.
         [Replications Check,BranchDC] A recent replication attempt failed:
            From DC1 to BranchDC
            Naming Context: DC=DOMAIN,DC=local
            The replication generated an error (1753):
            There are no more endpoints available from the endpoint mapper.
            The failure occurred at 2011-07-25 11:01:27.
            The last success occurred at 2011-07-22 04:58:49.
            13 failures have occurred since the last success.
            The directory on DC1 is in the process.
            of starting up or shutting down, and is not available.
            Verify machine is not hung during boot.
         * Replication Latency Check
         REPLICATION-RECEIVED LATENCY WARNING
         BranchDC:  Current time is 2011-07-25 11:26:43.
            DC=ForestDnsZones,DC=DOMAIN,DC=local
               Last replication recieved from DC2 at 2011-07-22 01:59:51.
               Last replication recieved from DC1 at 2011-07-22 04:58:34.
               Latency information for 3 entries in the vector were ignored.
                  3 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=DomainDnsZones,DC=DOMAIN,DC=local
               Last replication recieved from DC2 at 2011-07-22 01:59:51.
               Last replication recieved from DC1 at 2011-07-22 04:58:33.
               Latency information for 3 entries in the vector were ignored.
                  3 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Schema,CN=Configuration,DC=DOMAIN,DC=local
               Last replication recieved from DC2 at 2011-07-22 01:59:51.
               Last replication recieved from DC1 at 2011-07-22 04:58:33.
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Configuration,DC=DOMAIN,DC=local
               Last replication recieved from DC2 at 2011-07-22 01:59:51.
               Last replication recieved from DC1 at 2011-07-22 04:58:33.
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=DOMAIN,DC=local
               Last replication recieved from DC2 at 2011-07-22 02:02:02.
               Last replication recieved from DC1 at 2011-07-22 04:58:32.
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
         ......................... BranchDC passed test Replications
      Starting test: Topology
         * Configuration Topology Integrity Check
         * Analyzing the connection topology for DC=ForestDnsZones,DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for DC=DomainDnsZones,DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for CN=Schema,CN=Configuration,DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for CN=Configuration,DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.

My bad. Here is the complete log:
Command Line: "dcdiag.exe /V /C /D /E /s:DC1"

Domain Controller Diagnosis

Performing initial setup:
   * Connecting to directory service on server DC1.
   DC1.currentTime = 20110725092533.0Z
   DC1.highestCommittedUSN = 5447743
   DC1.isSynchronized = 1
   DC1.isGlobalCatalogReady = 1
   Failure Analysis: DC1 ... OK.
   [DC1] Directory Binding Error 1753:
   There are no more endpoints available from the endpoint mapper.
   This may limit some of the tests that can be performed.
   * Collecting site info.
   * Identifying all servers.
   DC1.currentTime = 20110725092615.0Z
   DC1.highestCommittedUSN = 5447743
   DC1.isSynchronized = 1
   DC1.isGlobalCatalogReady = 1
   * Identifying all NC cross-refs.
   * Found 3 DC(s). Testing 3 of them.
   Done gathering initial info.


===============================================Printing out pDsInfo

GLOBAL:
      ulNumServers=3
      pszRootDomain=DOMAIN.local
      pszNC=
      pszRootDomainFQDN=DC=DOMAIN,DC=local
      pszConfigNc=CN=Configuration,DC=DOMAIN,DC=local
      pszPartitionsDn=CN=Partitions,CN=Configuration,DC=DOMAIN,DC=local
      iSiteOptions=0
      dwTombstoneLifeTimeDays=60

      dwForestBehaviorVersion=2

      HomeServer=0, DC1

      SERVER: pServer[0].pszName=DC1
            pServer[0].pszGuidDNSName=86b881a6-4b3e-424b-adca-ad1aff078296._msdcs.DOMAIN.local
            pServer[0].pszDNSName=DC1.DOMAIN.local
            pServer[0].pszDn=CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pServer[0].pszComputerAccountDn=CN=DC1,OU=Domain Controllers,DC=DOMAIN,DC=local
            pServer[0].uuidObjectGuid=86b881a6-4b3e-424b-adca-ad1aff078296
            pServer[0].uuidInvocationId=86b881a6-4b3e-424b-adca-ad1aff078296
            pServer[0].iSite=0 (MainOffice)
            pServer[0].iOptions=1
            pServer[0].ftLocalAcquireTime=e7273b00 01cc4aac

            pServer[0].ftRemoteConnectTime=e6911580 01cc4aac

            pServer[0].ppszMasterNCs:
                  ppszMasterNCs[0]=DC=ForestDnsZones,DC=DOMAIN,DC=local
                  ppszMasterNCs[1]=DC=DomainDnsZones,DC=DOMAIN,DC=local
                  ppszMasterNCs[2]=CN=Schema,CN=Configuration,DC=DOMAIN,DC=local
                  ppszMasterNCs[3]=CN=Configuration,DC=DOMAIN,DC=local
                  ppszMasterNCs[4]=DC=DOMAIN,DC=local

      SERVER: pServer[1].pszName=BranchDC
            pServer[1].pszGuidDNSName=e641d3dd-4581-4e92-81ef-8fd161c6d016._msdcs.DOMAIN.local
            pServer[1].pszDNSName=BranchDC.DOMAIN.local
            pServer[1].pszDn=CN=NTDS Settings,CN=BranchDC,CN=Servers,CN=BranchOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pServer[1].pszComputerAccountDn=CN=BranchDC,OU=Domain Controllers,DC=DOMAIN,DC=local
            pServer[1].uuidObjectGuid=e641d3dd-4581-4e92-81ef-8fd161c6d016
            pServer[1].uuidInvocationId=472f6f39-1e82-40b7-ab76-762fd0a8fae5
            pServer[1].iSite=1 (BranchOffice)
            pServer[1].iOptions=1
            pServer[1].ftLocalAcquireTime=00000000 00000000

            pServer[1].ftRemoteConnectTime=00000000 00000000

            pServer[1].ppszMasterNCs:
                  ppszMasterNCs[0]=DC=ForestDnsZones,DC=DOMAIN,DC=local
                  ppszMasterNCs[1]=DC=DomainDnsZones,DC=DOMAIN,DC=local
                  ppszMasterNCs[2]=CN=Schema,CN=Configuration,DC=DOMAIN,DC=local
                  ppszMasterNCs[3]=CN=Configuration,DC=DOMAIN,DC=local
                  ppszMasterNCs[4]=DC=DOMAIN,DC=local

      SERVER: pServer[2].pszName=DC2
            pServer[2].pszGuidDNSName=b1db76d8-5b7f-4e88-b9f9-a3fba490bfba._msdcs.DOMAIN.local
            pServer[2].pszDNSName=DC2.DOMAIN.local
            pServer[2].pszDn=CN=NTDS Settings,CN=DC2,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pServer[2].pszComputerAccountDn=CN=DC2,OU=Domain Controllers,DC=DOMAIN,DC=local
            pServer[2].uuidObjectGuid=b1db76d8-5b7f-4e88-b9f9-a3fba490bfba
            pServer[2].uuidInvocationId=05b1c28f-ebc1-448a-ab6e-b0f71cbf8211
            pServer[2].iSite=0 (MainOffice)
            pServer[2].iOptions=1
            pServer[2].ftLocalAcquireTime=00000000 00000000

            pServer[2].ftRemoteConnectTime=00000000 00000000

            pServer[2].ppszMasterNCs:
                  ppszMasterNCs[0]=DC=ForestDnsZones,DC=DOMAIN,DC=local
                  ppszMasterNCs[1]=DC=DomainDnsZones,DC=DOMAIN,DC=local
                  ppszMasterNCs[2]=CN=Schema,CN=Configuration,DC=DOMAIN,DC=local
                  ppszMasterNCs[3]=CN=Configuration,DC=DOMAIN,DC=local
                  ppszMasterNCs[4]=DC=DOMAIN,DC=local

      SITES:  pSites[0].pszName=MainOffice
            pSites[0].pszSiteSettings=CN=NTDS Site Settings,CN=MainOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pSites[0].pszISTG=CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pSites[0].iSiteOption=0

            pSites[0].cServers=2

      SITES:  pSites[1].pszName=BranchOffice
            pSites[1].pszSiteSettings=CN=NTDS Site Settings,CN=BranchOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pSites[1].pszISTG=CN=NTDS Settings,CN=BranchDC,CN=Servers,CN=BranchOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
            pSites[1].iSiteOption=0

            pSites[1].cServers=1

      NC:     pNCs[0].pszName=ForestDnsZones
            pNCs[0].pszDn=DC=ForestDnsZones,DC=DOMAIN,DC=local

                  pNCs[0].aCrInfo[0].dwFlags=0x00000201
                  pNCs[0].aCrInfo[0].pszDn=CN=d35fa9b5-b39d-43d7-a886-8d0758f5ba64,CN=Partitions,CN=Configuration,DC=DOMAIN,DC=local
                  pNCs[0].aCrInfo[0].pszDnsRoot=ForestDnsZones.DOMAIN.local
                  pNCs[0].aCrInfo[0].iSourceServer=0
                  pNCs[0].aCrInfo[0].pszSourceServer=(null)
                  pNCs[0].aCrInfo[0].ulSystemFlags=0x00000005
                  pNCs[0].aCrInfo[0].bEnabled=TRUE
                  pNCs[0].aCrInfo[0].ftWhenCreated=00000000 00000000                   pNCs[0].aCrInfo[0].pszSDReferenceDomain=(null)
                  pNCs[0].aCrInfo[0].pszNetBiosName=(null)
                  pNCs[0].aCrInfo[0].cReplicas=-1
                  pNCs[0].aCrInfo[0].aszReplicas=


      NC:     pNCs[1].pszName=DomainDnsZones
            pNCs[1].pszDn=DC=DomainDnsZones,DC=DOMAIN,DC=local

                  pNCs[1].aCrInfo[0].dwFlags=0x00000201
                  pNCs[1].aCrInfo[0].pszDn=CN=3e9ef1d8-1909-43a2-9b4d-819166c5e9e2,CN=Partitions,CN=Configuration,DC=DOMAIN,DC=local
                  pNCs[1].aCrInfo[0].pszDnsRoot=DomainDnsZones.DOMAIN.local
                  pNCs[1].aCrInfo[0].iSourceServer=0
                  pNCs[1].aCrInfo[0].pszSourceServer=(null)
                  pNCs[1].aCrInfo[0].ulSystemFlags=0x00000005
                  pNCs[1].aCrInfo[0].bEnabled=TRUE
                  pNCs[1].aCrInfo[0].ftWhenCreated=00000000 00000000                   pNCs[1].aCrInfo[0].pszSDReferenceDomain=(null)
                  pNCs[1].aCrInfo[0].pszNetBiosName=(null)
                  pNCs[1].aCrInfo[0].cReplicas=-1
                  pNCs[1].aCrInfo[0].aszReplicas=


      NC:     pNCs[2].pszName=Schema
            pNCs[2].pszDn=CN=Schema,CN=Configuration,DC=DOMAIN,DC=local

                  pNCs[2].aCrInfo[0].dwFlags=0x00000201
                  pNCs[2].aCrInfo[0].pszDn=CN=Enterprise Schema,CN=Partitions,CN=Configuration,DC=DOMAIN,DC=local
                  pNCs[2].aCrInfo[0].pszDnsRoot=DOMAIN.local
                  pNCs[2].aCrInfo[0].iSourceServer=0
                  pNCs[2].aCrInfo[0].pszSourceServer=(null)
                  pNCs[2].aCrInfo[0].ulSystemFlags=0x00000001
                  pNCs[2].aCrInfo[0].bEnabled=TRUE
                  pNCs[2].aCrInfo[0].ftWhenCreated=00000000 00000000                   pNCs[2].aCrInfo[0].pszSDReferenceDomain=(null)
                  pNCs[2].aCrInfo[0].pszNetBiosName=(null)
                  pNCs[2].aCrInfo[0].cReplicas=-1
                  pNCs[2].aCrInfo[0].aszReplicas=


      NC:     pNCs[3].pszName=Configuration
            pNCs[3].pszDn=CN=Configuration,DC=DOMAIN,DC=local

                  pNCs[3].aCrInfo[0].dwFlags=0x00000201
                  pNCs[3].aCrInfo[0].pszDn=CN=Enterprise Configuration,CN=Partitions,CN=Configuration,DC=DOMAIN,DC=local
                  pNCs[3].aCrInfo[0].pszDnsRoot=DOMAIN.local
                  pNCs[3].aCrInfo[0].iSourceServer=0
                  pNCs[3].aCrInfo[0].pszSourceServer=(null)
                  pNCs[3].aCrInfo[0].ulSystemFlags=0x00000001
                  pNCs[3].aCrInfo[0].bEnabled=TRUE
                  pNCs[3].aCrInfo[0].ftWhenCreated=00000000 00000000                   pNCs[3].aCrInfo[0].pszSDReferenceDomain=(null)
                  pNCs[3].aCrInfo[0].pszNetBiosName=(null)
                  pNCs[3].aCrInfo[0].cReplicas=-1
                  pNCs[3].aCrInfo[0].aszReplicas=


      NC:     pNCs[4].pszName=DOMAIN
            pNCs[4].pszDn=DC=DOMAIN,DC=local

                  pNCs[4].aCrInfo[0].dwFlags=0x00000201
                  pNCs[4].aCrInfo[0].pszDn=CN=DOMAIN,CN=Partitions,CN=Configuration,DC=DOMAIN,DC=local
                  pNCs[4].aCrInfo[0].pszDnsRoot=DOMAIN.local
                  pNCs[4].aCrInfo[0].iSourceServer=0
                  pNCs[4].aCrInfo[0].pszSourceServer=(null)
                  pNCs[4].aCrInfo[0].ulSystemFlags=0x00000003
                  pNCs[4].aCrInfo[0].bEnabled=TRUE
                  pNCs[4].aCrInfo[0].ftWhenCreated=00000000 00000000                   pNCs[4].aCrInfo[0].pszSDReferenceDomain=(null)
                  pNCs[4].aCrInfo[0].pszNetBiosName=(null)
                  pNCs[4].aCrInfo[0].cReplicas=-1
                  pNCs[4].aCrInfo[0].aszReplicas=


      5 NC TARGETS: ForestDnsZones, DomainDnsZones, Schema, Configuration, DOMAIN,
      3 TARGETS: DC1, BranchDC, DC2,

=============================================Done Printing pDsInfo

Doing initial required tests
   
   Testing server: MainOffice\DC1
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         Failure Analysis: DC1 ... OK.
         * Active Directory RPC Services Check
         [DC1] DsBindWithSpnEx() failed with error 1753,
         There are no more endpoints available from the endpoint mapper..
         Printing RPC Extended Error Info:
         Error Record 1, ProcessID is 1056 (DcDiag)        
            System Time is: 7/25/2011 9:26:42:812
            Generating component is 2 (RPC runtime)
            Status is 1753: There are no more endpoints available from the endpoint mapper.

            Detection location is 500
            NumberOfParameters is 4
            Unicode string: ncacn_ip_tcp
            Unicode string: 86b881a6-4b3e-424b-adca-ad1aff078296._msdcs.DOMAIN.local
            Long val: -481213899
            Long val: 65537
         Error Record 2, ProcessID is 1056 (DcDiag)        
            System Time is: 7/25/2011 9:26:42:812
            Generating component is 2 (RPC runtime)
            Status is 1722: The RPC server is unavailable.

            Detection location is 761
            NumberOfParameters is 1
            Unicode string: 1025
         Error Record 3, ProcessID is 1056 (DcDiag)        
            System Time is: 7/25/2011 9:26:42:812
            Generating component is 8 (winsock)
            Status is 1722: The RPC server is unavailable.

            Detection location is 323
         Error Record 4, ProcessID is 1056 (DcDiag)        
            System Time is: 7/25/2011 9:26:42:812
            Generating component is 8 (winsock)
            Status is 1237: The operation could not be completed. A retry should be performed.

            Detection location is 313
         Error Record 5, ProcessID is 1056 (DcDiag)        
            System Time is: 7/25/2011 9:26:42:812
            Generating component is 8 (winsock)
            Status is 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

            Detection location is 311
            NumberOfParameters is 3
            Long val: 1025
            Pointer val: 0
            Pointer val: 0
         Error Record 6, ProcessID is 1056 (DcDiag)        
            System Time is: 7/25/2011 9:26:42:812
            Generating component is 8 (winsock)
            Status is 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

            Detection location is 318
         ......................... DC1 failed test Connectivity
   
   Testing server: BranchOffice\BranchDC
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         BranchDC.currentTime = 20110725092643.0Z
         BranchDC.highestCommittedUSN = 307261
         BranchDC.isSynchronized = 1
         BranchDC.isGlobalCatalogReady = 1
         Failure Analysis: BranchDC ... OK.
         * Active Directory RPC Services Check
         ......................... BranchDC passed test Connectivity
   
   Testing server: MainOffice\DC2
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         [DC2] LDAP search failed with error 58,
         The specified server cannot perform the requested operation..
         ***Error: The machine, DC2 could not be contacted, because of a

         bad net  response.  Check to make sure that this machine is a Domain

         Controller.
         ......................... DC2 failed test Connectivity

Doing primary tests
   
   Testing server: MainOffice\DC1
      Skipping all tests, because server DC1 is
      not responding to directory service requests
   
   Testing server: BranchOffice\BranchDC
      Starting test: Replications
         * Replications Check
         DC=ForestDnsZones,DC=DOMAIN,DC=local has 6 cursors.
         [Replications Check,BranchDC] A recent replication attempt failed:
            From DC1 to BranchDC
            Naming Context: DC=ForestDnsZones,DC=DOMAIN,DC=local
            The replication generated an error (1256):
            The remote system is not available. For information about network troubleshooting, see Windows Help.
            The failure occurred at 2011-07-25 04:48:58.
            The last success occurred at 2011-07-22 04:58:50.
            1 failures have occurred since the last success.
         DC=DomainDnsZones,DC=DOMAIN,DC=local has 6 cursors.
         [Replications Check,BranchDC] A recent replication attempt failed:
            From DC1 to BranchDC
            Naming Context: DC=DomainDnsZones,DC=DOMAIN,DC=local
            The replication generated an error (1256):
            The remote system is not available. For information about network troubleshooting, see Windows Help.
            The failure occurred at 2011-07-25 04:48:58.
            The last success occurred at 2011-07-22 04:58:50.
            1 failures have occurred since the last success.
         CN=Schema,CN=Configuration,DC=DOMAIN,DC=local has 9 cursors.
         [Replications Check,BranchDC] A recent replication attempt failed:
            From DC1 to BranchDC
            Naming Context: CN=Schema,CN=Configuration,DC=DOMAIN,DC=local
            The replication generated an error (1753):
            There are no more endpoints available from the endpoint mapper.
            The failure occurred at 2011-07-25 04:50:01.
            The last success occurred at 2011-07-22 04:58:50.
            1 failures have occurred since the last success.
            The directory on DC1 is in the process.
            of starting up or shutting down, and is not available.
            Verify machine is not hung during boot.
         CN=Configuration,DC=DOMAIN,DC=local has 9 cursors.
         [Replications Check,BranchDC] A recent replication attempt failed:
            From DC1 to BranchDC
            Naming Context: CN=Configuration,DC=DOMAIN,DC=local
            The replication generated an error (1753):
            There are no more endpoints available from the endpoint mapper.
            The failure occurred at 2011-07-25 04:49:19.
            The last success occurred at 2011-07-22 04:58:49.
            1 failures have occurred since the last success.
            The directory on DC1 is in the process.
            of starting up or shutting down, and is not available.
            Verify machine is not hung during boot.
         DC=DOMAIN,DC=local has 9 cursors.
         [Replications Check,BranchDC] A recent replication attempt failed:
            From DC1 to BranchDC
            Naming Context: DC=DOMAIN,DC=local
            The replication generated an error (1753):
            There are no more endpoints available from the endpoint mapper.
            The failure occurred at 2011-07-25 11:01:27.
            The last success occurred at 2011-07-22 04:58:49.
            13 failures have occurred since the last success.
            The directory on DC1 is in the process.
            of starting up or shutting down, and is not available.
            Verify machine is not hung during boot.
         * Replication Latency Check
         REPLICATION-RECEIVED LATENCY WARNING
         BranchDC:  Current time is 2011-07-25 11:26:43.
            DC=ForestDnsZones,DC=DOMAIN,DC=local
               Last replication recieved from DC2 at 2011-07-22 01:59:51.
               Last replication recieved from DC1 at 2011-07-22 04:58:34.
               Latency information for 3 entries in the vector were ignored.
                  3 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=DomainDnsZones,DC=DOMAIN,DC=local
               Last replication recieved from DC2 at 2011-07-22 01:59:51.
               Last replication recieved from DC1 at 2011-07-22 04:58:33.
               Latency information for 3 entries in the vector were ignored.
                  3 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Schema,CN=Configuration,DC=DOMAIN,DC=local
               Last replication recieved from DC2 at 2011-07-22 01:59:51.
               Last replication recieved from DC1 at 2011-07-22 04:58:33.
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Configuration,DC=DOMAIN,DC=local
               Last replication recieved from DC2 at 2011-07-22 01:59:51.
               Last replication recieved from DC1 at 2011-07-22 04:58:33.
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=DOMAIN,DC=local
               Last replication recieved from DC2 at 2011-07-22 02:02:02.
               Last replication recieved from DC1 at 2011-07-22 04:58:32.
               Latency information for 6 entries in the vector were ignored.
                  6 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
         ......................... BranchDC passed test Replications
      Starting test: Topology
         * Configuration Topology Integrity Check
         * Analyzing the connection topology for DC=ForestDnsZones,DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for DC=DomainDnsZones,DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for CN=Schema,CN=Configuration,DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for CN=Configuration,DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         ......................... BranchDC passed test Topology
      Starting test: CutoffServers
         * Configuration Topology Aliveness Check
         * Analyzing the alive system replication topology for DC=ForestDnsZones,DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for DC=DomainDnsZones,DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for CN=Schema,CN=Configuration,DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for CN=Configuration,DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for DC=DOMAIN,DC=local.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         ......................... BranchDC passed test CutoffServers
      Starting test: NCSecDesc
         * Security Permissions check for all NC's on DC BranchDC.
         * Security Permissions Check for
           DC=ForestDnsZones,DC=DOMAIN,DC=local
            (NDNC,Version 2)
         * Security Permissions Check for
           DC=DomainDnsZones,DC=DOMAIN,DC=local
            (NDNC,Version 2)
         * Security Permissions Check for
           CN=Schema,CN=Configuration,DC=DOMAIN,DC=local
            (Schema,Version 2)
         * Security Permissions Check for
           CN=Configuration,DC=DOMAIN,DC=local
            (Configuration,Version 2)
         * Security Permissions Check for
           DC=DOMAIN,DC=local
            (Domain,Version 2)
         ......................... BranchDC passed test NCSecDesc
      Starting test: NetLogons
         * Network Logons Privileges Check
         Verified share \\BranchDC\netlogon
         Verified share \\BranchDC\sysvol
         ......................... BranchDC passed test NetLogons
      Starting test: Advertising
         The DC BranchDC is advertising itself as a DC and having a DS.
         The DC BranchDC is advertising as an LDAP server
         The DC BranchDC is advertising as having a writeable directory
         The DC BranchDC is advertising as a Key Distribution Center
         Warning: BranchDC is not advertising as a time server.
         The DS BranchDC is advertising as a GC.
         ......................... BranchDC failed test Advertising
      Starting test: KnowsOfRoleHolders
         Role Schema Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
         Warning: DC1 is the Schema Owner, but is not responding to DS RPC Bind.
         RPC Extended Error Info not available. Use group policy on the local machine at "Computer Configuration/Administrative Templates/System/Remote Procedure Call" to enable it.
         Role Domain Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
         Warning: DC1 is the Domain Owner, but is not responding to DS RPC Bind.
         RPC Extended Error Info not available. Use group policy on the local machine at "Computer Configuration/Administrative Templates/System/Remote Procedure Call" to enable it.
         Role PDC Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
         Warning: DC1 is the PDC Owner, but is not responding to DS RPC Bind.
         RPC Extended Error Info not available. Use group policy on the local machine at "Computer Configuration/Administrative Templates/System/Remote Procedure Call" to enable it.
         Role Rid Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
         Warning: DC1 is the Rid Owner, but is not responding to DS RPC Bind.
         RPC Extended Error Info not available. Use group policy on the local machine at "Computer Configuration/Administrative Templates/System/Remote Procedure Call" to enable it.
         Role Infrastructure Update Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
         Warning: DC1 is the Infrastructure Update Owner, but is not responding to DS RPC Bind.
         RPC Extended Error Info not available. Use group policy on the local machine at "Computer Configuration/Administrative Templates/System/Remote Procedure Call" to enable it.
         ......................... BranchDC failed test KnowsOfRoleHolders
      Starting test: RidManager
         ridManagerReference = CN=RID Manager$,CN=System,DC=DOMAIN,DC=local
         * Available RID Pool for the Domain is 6603 to 1073741823
         fSMORoleOwner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local
         * DC1.DOMAIN.local is the RID Master
         ......................... BranchDC failed test RidManager
      Starting test: MachineAccount
         Checking machine account for DC BranchDC on DC BranchDC.
         * SPN found :LDAP/BranchDC.DOMAIN.local/DOMAIN.local
         * SPN found :LDAP/BranchDC.DOMAIN.local
         * SPN found :LDAP/BranchDC
         * SPN found :LDAP/BranchDC.DOMAIN.local/DOMAIN
         * SPN found :LDAP/e641d3dd-4581-4e92-81ef-8fd161c6d016._msdcs.DOMAIN.local
         * SPN found :E3514235-4B06-11D1-AB04-00C04FC2DCD2/e641d3dd-4581-4e92-81ef-8fd161c6d016/DOMAIN.local
         * SPN found :HOST/BranchDC.DOMAIN.local/DOMAIN.local
         * SPN found :HOST/BranchDC.DOMAIN.local
         * SPN found :HOST/BranchDC
         * SPN found :HOST/BranchDC.DOMAIN.local/DOMAIN
         * SPN found :GC/BranchDC.DOMAIN.local/DOMAIN.local
         ......................... BranchDC passed test MachineAccount
      Starting test: Services
         * Checking Service: Dnscache
         * Checking Service: NtFrs
         * Checking Service: IsmServ
         * Checking Service: kdc
         * Checking Service: SamSs
         * Checking Service: LanmanServer
         * Checking Service: LanmanWorkstation
         * Checking Service: RpcSs
         * Checking Service: w32time
         * Checking Service: NETLOGON
         ......................... BranchDC passed test Services
      Starting test: OutboundSecureChannels
         * The Outbound Secure Channels test
         ** Did not run Outbound Secure Channels test
         because /testdomain: was not entered
         ......................... BranchDC passed test OutboundSecureChannels
      Starting test: ObjectsReplicated
         BranchDC is in domain DC=DOMAIN,DC=local
         Checking for CN=BranchDC,OU=Domain Controllers,DC=DOMAIN,DC=local in domain DC=DOMAIN,DC=local on 1 servers
            Object is up-to-date on all servers.
         Checking for CN=NTDS Settings,CN=BranchDC,CN=Servers,CN=BranchOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local in domain CN=Configuration,DC=DOMAIN,DC=local on 1 servers
            Object is up-to-date on all servers.
         ......................... BranchDC passed test ObjectsReplicated
      Starting test: frssysvol
         * The File Replication Service SYSVOL ready test
         File Replication Service's SYSVOL is ready
         ......................... BranchDC passed test frssysvol
      Starting test: frsevent
         * The File Replication Service Event log test
         There are warning or error events within the last 24 hours after the

         SYSVOL has been shared.  Failing SYSVOL replication problems may cause

         Group Policy problems.
         An Warning Event occured.  EventID: 0x800034C4
            Time Generated: 07/25/2011   11:03:17
            (Event String could not be retrieved)
         ......................... BranchDC failed test frsevent
      Starting test: kccevent
         * The KCC Event log test
         An Warning Event occured.  EventID: 0x8025082D
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x8025082D
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x8025082D
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80000746
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80250829
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x8025082D
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80000746
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80250829
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x8025082D
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80000746
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80250829
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80000746
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80250829
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80000746
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80250829
            Time Generated: 07/25/2011   11:28:35
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80000785
            Time Generated: 07/25/2011   11:36:44
            Event String: The attempt to establish a replication link for

the following writable directory partition

failed.

 

Directory partition:

DC=DOMAIN,DC=local

Source domain controller:

CN=NTDS Settings,CN=DC2,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local

 

Source domain controller address:

b1db76d8-5b7f-4e88-b9f9-a3fba490bfba._msdcs.DOMAIN.local

 

Intersite transport (if any):

CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local

 

 

This domain controller will be unable to

replicate with the source domain controller until

this problem is corrected.  

 

User Action

Verify if the source domain controller is

accessible or network connectivity is available.

 

Additional Data

Error value:

1722 The RPC server is unavailable.
         ......................... BranchDC failed test kccevent
      Starting test: systemlog
         * The System Event log test
         Found no errors in System Event log in the last 60 minutes.
         ......................... BranchDC passed test systemlog
      Starting test: VerifyReplicas
         ......................... BranchDC passed test VerifyReplicas
      Starting test: VerifyReferences
         The system object reference (serverReference)

         CN=BranchDC,OU=Domain Controllers,DC=DOMAIN,DC=local and backlink on

         CN=BranchDC,CN=Servers,CN=BranchOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local

          are correct.
         The system object reference (frsComputerReferenceBL)

         CN=BranchDC,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=DOMAIN,DC=local

         and backlink on CN=BranchDC,OU=Domain Controllers,DC=DOMAIN,DC=local

         are correct.
         The system object reference (serverReferenceBL)

         CN=BranchDC,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=DOMAIN,DC=local

         and backlink on

         CN=NTDS Settings,CN=BranchDC,CN=Servers,CN=BranchOffice,CN=Sites,CN=Configuration,DC=DOMAIN,DC=local

         are correct.
         ......................... BranchDC passed test VerifyReferences
      Starting test: VerifyEnterpriseReferences
         ......................... BranchDC passed test VerifyEnterpriseReferences
      Starting test: CheckSecurityError
         * Dr Auth:  Beginning security errors check!
         Found KDC BranchDC for domain DOMAIN.local in site BranchOffice
         Checking machine account for DC BranchDC on DC BranchDC.
         * SPN found :LDAP/BranchDC.DOMAIN.local/DOMAIN.local
         * SPN found :LDAP/BranchDC.DOMAIN.local
         * SPN found :LDAP/BranchDC
         * SPN found :LDAP/BranchDC.DOMAIN.local/DOMAIN
         * SPN found :LDAP/e641d3dd-4581-4e92-81ef-8fd161c6d016._msdcs.DOMAIN.local
         * SPN found :E3514235-4B06-11D1-AB04-00C04FC2DCD2/e641d3dd-4581-4e92-81ef-8fd161c6d016/DOMAIN.local
         * SPN found :HOST/BranchDC.DOMAIN.local/DOMAIN.local
         * SPN found :HOST/BranchDC.DOMAIN.local
         * SPN found :HOST/BranchDC
         * SPN found :HOST/BranchDC.DOMAIN.local/DOMAIN
         * SPN found :GC/BranchDC.DOMAIN.local/DOMAIN.local
         Source DC DC2 has possible security error (1722).  Diagnosing...
               Found KDC DC2 for domain DOMAIN.local in site MainOffice
               Checking time skew between servers:
                     DC2
                     BranchDC
               Getting time for \\DC2.DOMAIN.local
               Error 53 querying time on DC DC2.  Ignoring this DC and continuing...
               Getting time for \\BranchDC.DOMAIN.local
               Time is 1311586800 on \\BranchDC.DOMAIN.local
               Time skew error between client and 1 DCs!  ERROR_ACCESS_DENIED or down machine recieved by:
                           DC2
         Ignoring DC DC1 in the convergence test of object CN=BranchDC,OU=Domain Controllers,DC=DOMAIN,DC=local, because we cannot connect!
         Ignoring DC DC2 in the convergence test of object CN=BranchDC,OU=Domain Controllers,DC=DOMAIN,DC=local, because we cannot connect!
         Checking for CN=BranchDC,OU=Domain Controllers,DC=DOMAIN,DC=local in domain DC=DOMAIN,DC=local on 1 servers
            Object is up-to-date on all servers.
         ......................... BranchDC failed test CheckSecurityError
   
   Testing server: MainOffice\DC2
      Skipping all tests, because server DC2 is
      not responding to directory service requests

DNS Tests are running and not hung. Please wait a few minutes...
   
   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
   
   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
   
   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
   
   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
   
   Running partition tests on : DOMAIN
      Starting test: CrossRefValidation
         ......................... DOMAIN passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... DOMAIN passed test CheckSDRefDom
   
   Running enterprise tests on : DOMAIN.local
      Starting test: Intersite
         Doing intersite inbound replication test on site MainOffice:
            Locating & Contacting Intersite Topology Generator (ISTG) ...
               *Warning: Currest ISTG (DC1) is down.  Looking for a new

               ISTG.
               ***Error: The current ISTG is down in site MainOffice and further

               dcdiag could not contact any other servers in the site that

               could take the ISTG role.  Ensure there is at least one up DC.

               Must abandon inbound intersite replication test for this site.
         Doing intersite inbound replication test on site BranchOffice:
            Locating & Contacting Intersite Topology Generator (ISTG) ...
               The ISTG for site BranchOffice is: BranchDC.
               ISTG (BranchDC) Failure Parameters:

                   Failover Tries: 1

                   Failover Time: 120
            Checking for down bridgeheads ...
               *Warning: Remote bridgehead MainOffice\DC1 is not eligible as a

               bridgehead due to too many failures.  Replication may be

               disrupted into the local site BranchOffice.
               Remote bridgehead MainOffice\DC1 also couldn't be contacted by

               dcdiag.  Check this server.
               Bridghead BranchOffice\BranchDC is up and replicating fine.
               *Warning: Remote bridgehead MainOffice\DC2 has some

               replication syncs failing.  It will  be 0 hours 52 minutes

               before the bridgehead is considered ineligible to be a

               bridgehead.
               Remote bridgehead MainOffice\DC2 also couldn't be contacted

               by dcdiag.  Check this server.
            Doing in depth site analysis ...
               Checking writeable NC: ForestDnsZones on remote site MainOffice
               Remote site MainOffice is replicating to the local site BranchOffice

               the writeable NC ForestDnsZones correctly.
               Checking writeable NC: DomainDnsZones on remote site MainOffice
               Remote site MainOffice is replicating to the local site BranchOffice

               the writeable NC DomainDnsZones correctly.
               Checking writeable NC: Schema on remote site MainOffice
               Remote site MainOffice is replicating to the local site BranchOffice

               the writeable NC Schema correctly.
               Checking writeable NC: Configuration on remote site MainOffice
               Remote site MainOffice is replicating to the local site BranchOffice

               the writeable NC Configuration correctly.
               Checking writeable NC: DOMAIN on remote site MainOffice
               Remote site MainOffice is replicating to the local site BranchOffice

               the writeable NC DOMAIN correctly.
         ......................... DOMAIN.local failed test Intersite
      Starting test: FsmoCheck
         GC Name: \\DC1.DOMAIN.local
         Locator Flags: 0xe00003fd
         Warning: Couldn't verify this server as a PDC using DsListRoles()
         PDC Name: \\DC1.DOMAIN.local
         Locator Flags: 0xe00003fd
         Time Server Name: \\DC1.DOMAIN.local
         Locator Flags: 0xe00003fd
         Preferred Time Server Name: \\DC1.DOMAIN.local
         Locator Flags: 0xe00003fd
         KDC Name: \\DC1.DOMAIN.local
         Locator Flags: 0xe00003fd
         ......................... DOMAIN.local passed test FsmoCheck
      Starting test: DNS
         Test results for domain controllers:
           
            DC: BranchDC.DOMAIN.local
            Domain: DOMAIN.local

                 
               TEST: Authentication (Auth)
                  Authentication test: Successfully completed
                 
               TEST: Basic (Basc)
                   Microsoft(R) Windows(R) Server 2003 Standard x64 Edition (Service Pack level: 2.0) is supported
                  NETLOGON service is running
                  kdc service is running
                  DNSCACHE service is running
                  DNS service is running
                  DC is a DNS server
                  Network adapters information:
                  Adapter [00000007] HP NC105i PCIe Gigabit Server Adapter:
                     MAC address is 00:25:B3:99:B4:B1
                     IP address is static
                     IP address: 192.168.10.1
                     DNS servers:
                        192.168.10.1 (<name unavailable>) [Valid]
                        192.168.0.1 (DC1.DOMAIN.local.) [Valid]
                  The A record for this DC was found
                  The SOA record for the Active Directory zone was found
                  The Active Directory zone on this DC/DNS server was found (primary)
                  Root zone on this DC/DNS server was not found
                 
               TEST: Forwarders/Root hints (Forw)
                  Recursion is enabled
                  Forwarders Information:
                     192.168.0.1 (DC1.DOMAIN.local.) [Valid]
                 
               TEST: Delegations (Del)
                  Delegation information for the zone: DOMAIN.local.
                     Delegated domain name: _msdcs.DOMAIN.local.
                        DNS server: DC1.DOMAIN.local. IP:192.168.0.1 [Valid]
                 
               TEST: Dynamic update (Dyn)
                  Warning: Dynamic update is enabled on the zone but not secure DOMAIN.local.
                  Test record _dcdiag_test_record added successfully in zone DOMAIN.local.
                  Test record _dcdiag_test_record deleted successfully in zone DOMAIN.local.
                 
               TEST: Records registration (RReg)
                  Network Adapter [00000007] HP NC105i PCIe Gigabit Server Adapter:
                     Matching A record found at DNS server 192.168.10.1:
                     BranchDC.DOMAIN.local

                     Matching CNAME record found at DNS server 192.168.10.1:
                     e641d3dd-4581-4e92-81ef-8fd161c6d016._msdcs.DOMAIN.local

                     Matching DC SRV record found at DNS server 192.168.10.1:
                     _ldap._tcp.dc._msdcs.DOMAIN.local

                     Matching GC SRV record found at DNS server 192.168.10.1:
                     _ldap._tcp.gc._msdcs.DOMAIN.local

               Total query time:0 min. 1 sec.. Total RPC connection time:0 min. 0 sec.
               Total WMI connection time:0 min. 3 sec. Total Netuse connection time:0 min. 0 sec.
         
           
            DC: DC2.DOMAIN.local
            Domain: DOMAIN.local

                 
               TEST: Authentication (Auth)
                  Error: Authentication failed with specified credentials
                  [Error details: 1203 (Type: Win32 - Description: No network provider accepted the given network path.) - Add connection failed]
                 
               TEST: Basic (Basc)
                  Error: No LDAP connectivity
                  Error: No WMI connectivity
                  [Error details: 0x800706ba (Type: HRESULT - Facility: Win32, Description: The RPC server is unavailable.) - Connection to WMI server failed]
               Total query time:0 min. 0 sec.. Total RPC connection time:0 min. 0 sec.
               Total WMI connection time:0 min. 42 sec. Total Netuse connection time:0 min. 23 sec.
         
           
            DC: DC1.DOMAIN.local
            Domain: DOMAIN.local

                 
               TEST: Authentication (Auth)
                  Authentication test: Successfully completed
                 
               TEST: Basic (Basc)
                  Error: No DS RPC connectivity
                  Error: No WMI connectivity
                  [Error details: 0x800706ba (Type: HRESULT - Facility: Win32, Description: The RPC server is unavailable.) - Connection to WMI server failed]
               Total query time:0 min. 0 sec.. Total RPC connection time:0 min. 0 sec.
               Total WMI connection time:0 min. 2 sec. Total Netuse connection time:0 min. 1 sec.
         
         Summary of test results for DNS servers used by the above domain controllers:

            DNS server: 192.168.0.1 (DC1.DOMAIN.local.)
               All tests passed on this DNS server
               This is a valid DNS server.
               Name resolution is funtional. _ldap._tcp SRV record for the forest root domain is registered
               Delegation to the domain _msdcs.DOMAIN.local. is operational
               Total query time:0 min. 0 sec., Total WMI connection time:0 min. 3 sec.
               
            DNS server: 192.168.10.1 (<name unavailable>)
               All tests passed on this DNS server
               This is a valid DNS server.
               Name resolution is funtional. _ldap._tcp SRV record for the forest root domain is registered
               Total query time:0 min. 0 sec., Total WMI connection time:0 min. 0 sec.
               
         Summary of DNS test results:
         
                                            Auth Basc Forw Del  Dyn  RReg Ext  
               ________________________________________________________________
            Domain: DOMAIN.local
               BranchDC                       PASS PASS PASS PASS WARN PASS n/a  
               DC2                   FAIL FAIL n/a  n/a  n/a  n/a  n/a  
               DC1                       PASS FAIL n/a  n/a  n/a  n/a  n/a  
         
         Total Time taken to test all the DCs:1 min. 15 sec.
         ......................... DOMAIN.local failed test DNS

You said replication stopped on Friday. Do you know if there was any changes on your network that day?

You also said you opened some ports, but you also need to add the "TCP high ports" used by RPC if you don't have a fixed port for that (or IPSec).

http://technet.microsoft.com/en-us/library/bb727063.aspx

Hi snusgubben

There was no changes on friday. I have added some udp ports that I missed out for kerberos and LDAP ping.

How do I check what is my fixed port for rpc?

thanks

Default, the RPC uses dynamic ports. If you want it fixed, you have to add two registry keys on all DCs you have, like described in the Technet article (see "Limited RPC"):

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\TCP/IP Port

HKEY_LOCAL_MACHINESYSTEM\CurrentControlSet\Services\NTFRS\Parameters\RPC TCP/IP Port Assignment

Thanks

Will this reg fix work with Windows Server 2008 R2 as DC2 is running that OS?

The second key is for FRS replication. If you use DFSR it's not valid.

This is not a "fix", but a description of open ports needed by replication behind FWs. If nothing has changed since Friday, I would not set a fixed RPC port, but look for what has happend.

Setting a fixed port makes it harder to maintain in the future (i.e. adding a new DC and do you/other admin remember the fixed port. You could offcourse use GPP for this...)

First I would check if the needed ports are open between the branch and main office.

Port Query is useful in such a troubleshooting:

http://blogs.technet.com/b/abizerh/archive/2009/06/11/troubleshooting-rpc-server-is-unavailable-error-reported-in-failing-ad-replication-scenario.aspx


If i.e. TCP 1026 is dropping, open 1024-65535 and see if it solves your problem. When things are running ok, you could consider thighten things a little.

Thanks for the help snus.

I was actually reffering to the registry fix of the tcp ports assignments for dfrs and ntds.

I now have replication between DC1 and DC2 but on BranchDC. I think the problem is with branchDC because it cannot replicate between both DC's. Here are the errors that I receive from BranchDC:
Event Type:      Error
Event Source:      NTDS KCC
Event Category:      Knowledge Consistency Checker
Event ID:      1311
Date:            2011/07/25
Time:            05:50:15 PM
User:            NT AUTHORITY\ANONYMOUS LOGON
Computer:      BranchDC
Description:
The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
 
Directory partition:
DC=DomainDnsZones,DC=domain,DC=local
 
There is insufficient site connectivity information in Active Directory Sites and Services for the KCC to create a spanning tree replication topology. Or, one or more domain controllers with this directory partition are unable to replicate the directory partition information. This is probably due to inaccessible domain controllers.
 
User Action
Use Active Directory Sites and Services to perform one of the following actions:
- Publish sufficient site connectivity information so that the KCC can determine a route by which this directory partition can reach this site. This is the preferred option.
- Add a Connection object to a domain controller that contains the directory partition in this site from a domain controller that contains the same directory partition in another site.
 
If neither of the Active Directory Sites and Services tasks correct this condition, see previous events logged by the KCC that identify the inaccessible domain controllers.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
---------------------------------------------------------------------------
Event Type:      Warning
Event Source:      NTDS KCC
Event Category:      Knowledge Consistency Checker
Event ID:      1566
Date:            2011/07/25
Time:            05:50:15 PM
User:            NT AUTHORITY\ANONYMOUS LOGON
Computer:      BranchDC
Description:
All domain controllers in the following site that can replicate the directory partition over this transport are currently unavailable.
 
Site:
CN=MainOffice,CN=Sites,CN=Configuration,DC=domain,DC=local
Directory partition:
DC=domain,DC=local
Transport:
CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=domain,DC=local

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
--------------------------------------------------------------------------
Event Type:      Error
Event Source:      NTDS KCC
Event Category:      Knowledge Consistency Checker
Event ID:      1311
Date:            2011/07/25
Time:            05:50:15 PM
User:            NT AUTHORITY\ANONYMOUS LOGON
Computer:      BranchDC
Description:
The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
 
Directory partition:
DC=domain,DC=local
 
There is insufficient site connectivity information in Active Directory Sites and Services for the KCC to create a spanning tree replication topology. Or, one or more domain controllers with this directory partition are unable to replicate the directory partition information. This is probably due to inaccessible domain controllers.
 
User Action
Use Active Directory Sites and Services to perform one of the following actions:
- Publish sufficient site connectivity information so that the KCC can determine a route by which this directory partition can reach this site. This is the preferred option.
- Add a Connection object to a domain controller that contains the directory partition in this site from a domain controller that contains the same directory partition in another site.
 
If neither of the Active Directory Sites and Services tasks correct this condition, see previous events logged by the KCC that identify the inaccessible domain controllers.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
------------------------------------------------------------------------

So DC1 and DC2 is replicating fine and they are both in the main office?

BranchDC will not replicate with main office DCs?

Please verify that the sites is correct in AD Sites and Services. Does the branchDC use itself as DNS or is it pointing to the main office DCs for DNS?

Have you verified that all needed ports are open refer the previous posted article?

dc1 and dc2 are in the main office.
branchdc uses itself as dns.
i have verified ports on all dc's.
the sites are correct. should i create new connection objects?

Verify that this runs ok both directions: repadmin /bind dcname

i.e.
from branchDC: repadmin /bind DC2
from main site: repadmin /bind branchDC

The MS DS Team made a blog entry about troubleshooting those KCC errors you got. You should give it close look.

http://blogs.technet.com/b/askds/archive/2008/10/31/troubleshooting-kcc-event-log-errors.aspx

I have verified this.
I could successfully bind to the other servers from either one of them.
I have got FRS and DFRS enable on all DC's as I have Windows Server 2003 R2 on DC1 and BranchDC and Windows Server 2008 R2 on DC2. Is this affecting the replication?
I have not had this problem before and replication has been taking place thru FRS.

FRS/DFSR is used to replicate SYSVOL, but the Naming Contexts are replicated with AD Replication. They are two different things.

Can you run "repadmin /replsum" from one of the main office DC? Post the output.

Also run "dcdiag /v /e /c /f:dcdiag.txt" (attach the file)

From DC1:
C:\Documents and Settings\Administrator>repadmin /replsum
Replication Summary Start Time: 2011-07-26 08:35:14

Beginning data collection for replication summary, this may take awhile:
  ......


Source DC           largest delta  fails/total  %%  error
 DC1                    48m:08s    0 /  10    0
 DC2                38m:45s    0 /   5    0
 BranchDC                    38m:45s    0 /   5    0


Destination DC    largest delta    fails/total  %%  error
 DC1                    38m:45s    0 /  10    0
 DC2                48m:09s    0 /   5    0
 BranchDC                    40m:41s    0 /   5    0

From DC2:
Replication Summary Start Time: 2011-07-26 08:36:53

Beginning data collection for replication summary, this may take awhile:
  ......


Source DSA          largest delta    fails/total %%   error
 DC1                    49m:47s    0 /   5    0
 DC2                40m:24s    0 /   5    0
 BranchDC                    40m:24s    0 /   5    0


Destination DSA     largest delta    fails/total %%   error
 DC1                    40m:23s    0 /  10    0
 DC2                50m:10s    0 /   5    0


Experienced the following operational errors trying to retrieve replication info
rmation:
        1722 - BranchDC

Here is dcdiag.txt from DC1
dcdiag.txt

And here is dcdiagDC2.txt from DC2
dcdiagDC2.txt

Replication seems fine from DC1, but not from DC2 <-> BranchDC.

DC1 bridgehead replicates fine with the branch bridgehead.

The problems seems to be DC2.

From DCdiag:

Make sure the firewall accepts packets on TCP/UDP 53 (DNS) and UDP 123 (time).

Is the time correct on DC2?
Is the Windows firewall enabled on DC2?
Do you have any anti-virus prog with a firewall on DC2? (ie. Symantec EP. If so disable during troubleshooting)

Change the prefered DNS on DC2 to point to DC1 during troubleshooting and itself as secondary, and run these on DC2:

ipconfig /flushdns
ipconfig /registerdns
netstop netlogon && netstart netlogon

When the above is checked, run a new DCdiag on DC2. (the wrapping was messed on the last one, so it was hard to read).

I have checked all inbound rules and they are all enabled. I have changed the preferred dns of DC2 to DC1.
But now I cannot bind to BranchDC from DC2 and vice versa.
WIll post the new dcdiag now of DC2.

I tried to bind from BranchDC vice versa and I get the same error:

DsBindWithCred to DC2.domain.local failed with status 1722 (0x6ba):
    The RPC server is unavailable.

Was this after you changed the DNS settings on DC2?

Yes

Here is dcdiag from DC2
dcdiagDC2-New.txt

Is the time correct on DC2?
Is the Windows firewall enabled on DC2?
Do you have any anti-virus prog with a firewall on DC2? (ie. Symantec EP. If so disable during troubleshooting)

The wrapping in that dcdiag is messed up. Do you run the command with the f-switch? (/f:dcdiag.txt)

The time is correct on DC2 and the firewall is enabled. I will disable AVG Business Security and try it out again.

Thanks

I have disabled the Anti-Virus and the result is still the same.
Will run dcdiag with the f-switch. I apologise for that. I wanted to save it on C:\

Ok. Will do that now. Thanks

just use "/f:c:\dcdiag.txt" then

Problem was the firewall. I disabled it and the binding works from both sides. How do I find out which port it is blocking?

If you want to keep the Windows FW enabled for the domain;

Open "Windows Firewall with Advanced Security"
In the Action pane: Properties
Hit the "Cusomize" button in the Logging section -> "Log dropped packets" -> Yes

The default location of the log is located: %systemroot%\system32\LogFiles\Firewall\pfirewall.log

You need to apply the same rules as you have stated in the site-to-site FW.

#Version: 1.5
#Software: Microsoft Windows Firewall
#Time Format: Local
#Fields: date time action protocol src-ip dst-ip src-port dst-port size tcpflags tcpsyn tcpack tcpwin icmptype icmpcode info path

                                                                                                                                                                                                                2011-07-26 12:02:41 DROP ICMP 192.168.0.10 192.168.0.2 - - 80 - - - - 5 0 - RECEIVE
2011-07-26 12:02:42 DROP ICMP 192.168.0.10 192.168.0.2 - - 80 - - - - 5 0 - RECEIVE
2011-07-26 12:02:43 DROP ICMP 192.168.0.10 192.168.0.2 - - 1463 - - - - 5 0 - RECEIVE
2011-07-26 12:02:45 DROP ICMP 192.168.0.10 192.168.0.2 - - 1436 - - - - 5 0 - RECEIVE
2011-07-26 12:02:46 DROP ICMP 192.168.0.10 192.168.0.2 - - 68 - - - - 5 0 - RECEIVE
2011-07-26 12:02:49 DROP ICMP 192.168.0.10 192.168.0.2 - - 80 - - - - 5 0 - RECEIVE
2011-07-26 12:02:53 DROP ICMP 192.168.0.10 192.168.0.2 - - 76 - - - - 5 0 - RECEIVE

This might be a silly question. But how do I enable ICMP in Windows Server 2008 R2? :-)

open cmd: netsh firewall set icmpsetting 8 enable

Found it. Thanks So Much snus

Can I change the Primary DNS on DC2 back to itself?

Yep