Windows 2003 Server Remote Sites not Replicating with main location

mcrossland
mcrossland used Ask the Experts™
on
I have 2 A/D servers at my main location that replication is working on correctly.
I have 2 remote locations each with one AD DC (Windows 2003 Server).
Replication is not working between the main site and the remote sites.
The sites are connected VPN using Watchguard Fireboxes with wide open access.  
Please advise as to what I can provide you with to troubleshoot this issue and get it resolved.

Thanks!
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Top Expert 2012

Commented:
Run dcdiag on the DC post results.

Make sure they are pointing to themselves for DNS there should be no external DNS servers listed in the TCP\IP including the router.

Author

Commented:
Here you go.  Dcdiag on the main location DC
C:\Program Files\Support Tools>dcdiag

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: Main-Office\RCCAD1
      Starting test: Connectivity
         ......................... RCCAD1 passed test Connectivity

Doing primary tests

   Testing server: Main-Office\RCCAD1
      Starting test: Replications
         [Replications Check,RCCAD1] A recent replication attempt failed:
            From RCCMSTR-WW to RCCAD1
            Naming Context: DC=royalcase,DC=local
            The replication generated an error (8606):
            Insufficient attributes were given to create an object.  This object
 may not exist because it may have been deleted and already garbage collected.
            The failure occurred at 2010-07-05 13:38:57.
            The last success occurred at (never).
            2991 failures have occurred since the last success.
         REPLICATION-RECEIVED LATENCY WARNING
         RCCAD1:  Current time is 2010-07-05 13:39:08.
            DC=ForestDnsZones,DC=royalcase,DC=local
               Last replication recieved from RCCMSTR2 at 2010-03-06 15:45:39.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!

            DC=DomainDnsZones,DC=royalcase,DC=local
               Last replication recieved from RCCMSTR2 at 2010-03-06 15:45:39.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!

            CN=Schema,CN=Configuration,DC=royalcase,DC=local
               Last replication recieved from RCCMSTR2 at 2010-03-06 15:45:35.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!

            CN=Configuration,DC=royalcase,DC=local
               Last replication recieved from RCCMSTR2 at 2010-03-06 15:45:35.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!

            DC=royalcase,DC=local
               Last replication recieved from RCCMSTR2 at 2010-03-06 15:45:39.
               WARNING:  This latency is over the Tombstone Lifetime of 60 days!

               Last replication recieved from RCCMSTR-WW at 2010-05-13 10:11:51.

         ......................... RCCAD1 passed test Replications
      Starting test: NCSecDesc
         ......................... RCCAD1 passed test NCSecDesc
      Starting test: NetLogons
         ......................... RCCAD1 passed test NetLogons
      Starting test: Advertising
         ......................... RCCAD1 passed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... RCCAD1 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... RCCAD1 passed test RidManager
      Starting test: MachineAccount
         ......................... RCCAD1 passed test MachineAccount
      Starting test: Services
         ......................... RCCAD1 passed test Services
      Starting test: ObjectsReplicated
         ......................... RCCAD1 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... RCCAD1 passed test frssysvol
      Starting test: frsevent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... RCCAD1 failed test frsevent
      Starting test: kccevent
         An Error Event occured.  EventID: 0xC00007FA
            Time Generated: 07/05/2010   13:37:10
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80000785
            Time Generated: 07/05/2010   13:37:10
            Event String: The attempt to establish a replication link for
         An Error Event occured.  EventID: 0xC00007FA
            Time Generated: 07/05/2010   13:37:11
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80000785
            Time Generated: 07/05/2010   13:37:11
            Event String: The attempt to establish a replication link for
         An Error Event occured.  EventID: 0xC00007FA
            Time Generated: 07/05/2010   13:37:11
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80000785
            Time Generated: 07/05/2010   13:37:11
            Event String: The attempt to establish a replication link for
         An Error Event occured.  EventID: 0xC00007FA
            Time Generated: 07/05/2010   13:37:11
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80000785
            Time Generated: 07/05/2010   13:37:11
            Event String: The attempt to establish a replication link for
         An Error Event occured.  EventID: 0xC00007FA
            Time Generated: 07/05/2010   13:37:11
            (Event String could not be retrieved)
         An Warning Event occured.  EventID: 0x80000785
            Time Generated: 07/05/2010   13:37:11
            Event String: The attempt to establish a replication link for
         An Error Event occured.  EventID: 0xC02507C4
            Time Generated: 07/05/2010   13:38:57
            (Event String could not be retrieved)
         ......................... RCCAD1 failed test kccevent
      Starting test: systemlog
         An Error Event occured.  EventID: 0xC0001B7A
            Time Generated: 07/05/2010   13:06:44
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 07/05/2010   13:09:49
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 07/05/2010   13:09:49
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 07/05/2010   13:09:49
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 07/05/2010   13:09:50
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 07/05/2010   13:09:51
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 07/05/2010   13:09:52
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 07/05/2010   13:09:52
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 07/05/2010   13:09:53
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0x00000457
            Time Generated: 07/05/2010   13:09:53
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/05/2010   13:28:44
            (Event String could not be retrieved)
         An Error Event occured.  EventID: 0xC0002719
            Time Generated: 07/05/2010   13:29:26
            (Event String could not be retrieved)
         ......................... RCCAD1 failed test systemlog
      Starting test: VerifyReferences
         ......................... RCCAD1 passed test VerifyReferences

   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom

   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom

   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom

   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom

   Running partition tests on : royalcase
      Starting test: CrossRefValidation
         ......................... royalcase passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... royalcase passed test CheckSDRefDom

   Running enterprise tests on : royalcase.local
      Starting test: Intersite
         ......................... royalcase.local passed test Intersite
      Starting test: FsmoCheck
         ......................... royalcase.local passed test FsmoCheck

C:\Program Files\Support Tools>

Open in new window

Top Expert 2012
Commented:
DC is past the tombstone life which means it is dead and you must remove the DC.

Run dcpromo /forceremoval on DC.
Run metadata cleanup on HQ DCs http://www.petri.co.il/delete_failed_dcs_from_ad.htm
Delete all DNS records for the failed DC.

Once you have done the above you can dpromo the DC again.
Amazon Web Services

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

Author

Commented:
I understand.
The server that is not tombstoned is the primary one that needs the most urgent care and brought back online.  I'm assuming that I can do the same steps on it, right?
It is the one in the dcdiag log named rccmstr-ww
Top Expert 2012

Commented:
You should keep your HQ DCs those are going to have the most updated data. Also, they should be holding the FSMO roles.

Is your primary DC down?

Whatever DCs state they are tombstone they must be removed from the system before you can proceed on working on functioning DCs.

Author

Commented:
Primary DC's are UP and replicating at HQ.
I'm forcing dcpromo on the 2 remote locations' DC's now.
I will then do the metadatacleanup on the HQ DCs.
Then, if you like, I can try to promote one of the remote ones back to a DC unless you want me to stop there and check something first.  Just let me know what your thoughts are at that step.
Top Expert 2012

Commented:
Nope sounds good but I would force replication between the 2 HQ DCs to make sure they have the same data before trying to promote the remote DCs again.

Author

Commented:
OK.  I have promoted one of the remote DC's and it is showing up in Sites and Services under the correct Site.  Under NTDS Settings it has an automatically generated with the name of one of the HQ DC's.  How to I confirm that Replication is working now before proceeding to my other remote site DC.
I don't want to get ahead of myself and would like to confirm this is all working so far.

Author

Commented:
Here is the DCDIAG output from the HQ DC.  Looks like Replication Errors.
Please advise as to what to do next.
C:\Program Files\Support Tools>dcdiag

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: Main-Office\RCCAD1
      Starting test: Connectivity
         ......................... RCCAD1 passed test Connectivity

Doing primary tests

   Testing server: Main-Office\RCCAD1
      Starting test: Replications
         REPLICATION LATENCY WARNING
         RCCAD1: This replication path was preempted by higher priority work.
            from RCCMSTR-WW to RCCAD1
            Reason: The operation completed successfully.
            The last success occurred at (never).
            Replication of new changes along this path will be delayed.
            Progress is occurring normally on this path.
         REPLICATION LATENCY WARNING
         RCCAD1: This replication path was preempted by higher priority work.
            from RCCMSTR-WW to RCCAD1
            Reason: The operation completed successfully.
            The last success occurred at (never).
            Replication of new changes along this path will be delayed.
            Progress is occurring normally on this path.
         REPLICATION LATENCY WARNING
         RCCAD1: This replication path was preempted by higher priority work.
            from RCCMSTR-WW to RCCAD1
            Reason: The operation completed successfully.
            The last success occurred at (never).
            Replication of new changes along this path will be delayed.
            Progress is occurring normally on this path.
         ......................... RCCAD1 passed test Replications
      Starting test: NCSecDesc
         ......................... RCCAD1 passed test NCSecDesc
      Starting test: NetLogons
         ......................... RCCAD1 passed test NetLogons
      Starting test: Advertising
         ......................... RCCAD1 passed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... RCCAD1 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... RCCAD1 passed test RidManager
      Starting test: MachineAccount
         ......................... RCCAD1 passed test MachineAccount
      Starting test: Services
         ......................... RCCAD1 passed test Services
      Starting test: ObjectsReplicated
         ......................... RCCAD1 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... RCCAD1 passed test frssysvol
      Starting test: frsevent
         ......................... RCCAD1 passed test frsevent
      Starting test: kccevent
         ......................... RCCAD1 passed test kccevent
      Starting test: systemlog
         ......................... RCCAD1 passed test systemlog
      Starting test: VerifyReferences
         ......................... RCCAD1 passed test VerifyReferences

   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom

   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation

      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom

   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom

   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom

   Running partition tests on : royalcase
      Starting test: CrossRefValidation
         ......................... royalcase passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... royalcase passed test CheckSDRefDom

   Running enterprise tests on : royalcase.local
      Starting test: Intersite
         ......................... royalcase.local passed test Intersite
      Starting test: FsmoCheck
         ......................... royalcase.local passed test FsmoCheck

Open in new window

Top Expert 2012

Commented:
Wait it out you will see that replication will fully take place but there are other processes running. Wait it out everything looks good. The first replication is still taking place.

Author

Commented:
Will do.  I'll just check on it tomorrow.   Remind me to follow up here if you don't hear from me.  :)
Top Expert 2012

Commented:
I will try.
Top Expert 2012

Commented:
Let me know if you have any trouble

Author

Commented:
Sure!  I appreciate your help.   I think we're probably ok but wanted to award you the points now.

Author

Commented:
No real troubles.  Everything is replicating main HQ servers (2) and both remote Servers.
One issue is that when I run repadmin /showreps on one of the remote servers, it shows that it is trying to replicate with the other remote site which it doesn't have access to due to our VPN setup.
How can I get it to stop trying to replicate with that other remote server and just rep with the main HQ?
If I delete the automatically generated one won't it just come back?
Here is the error:
    Main-Office\RCCAD1 via RPC
        DC object GUID: 09263355-fcda-4568-984d-4e57babccd4b
        Last attempt @ 2010-07-06 14:51:45 was successful.

CN=Schema,CN=Configuration,DC=royalcase,DC=local
    Main-Office\RCCAD1 via RPC
        DC object GUID: 09263355-fcda-4568-984d-4e57babccd4b
        Last attempt @ 2010-07-06 14:51:45 was successful.

DC=DomainDnsZones,DC=royalcase,DC=local
    Main-Office\RCCAD1 via RPC
        DC object GUID: 09263355-fcda-4568-984d-4e57babccd4b
        Last attempt @ 2010-07-06 14:51:46 was successful.

DC=ForestDnsZones,DC=royalcase,DC=local
    Main-Office\RCCAD1 via RPC
        DC object GUID: 09263355-fcda-4568-984d-4e57babccd4b
        Last attempt @ 2010-07-06 14:51:46 was successful.

Source: Whitewright\RCCMSTR-WW
******* 90 CONSECUTIVE FAILURES since 2010-07-05 15:53:09
Last error: 1722 (0x6ba):
            The RPC server is unavailable.

Naming Context: DC=DomainDnsZones,DC=royalcase,DC=local
Source: Whitewright\RCCMSTR-WW
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: DC=ForestDnsZones,DC=royalcase,DC=local
Source: Whitewright\RCCMSTR-WW
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: CN=Configuration,DC=royalcase,DC=local
Source: Whitewright\RCCMSTR-WW
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: DC=royalcase,DC=local
Source: Whitewright\RCCMSTR-WW
******* WARNING: KCC could not add this REPLICA LINK due to error.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial