Link to home
Start Free TrialLog in
Avatar of Simba01
Simba01

asked on

Replication Issues

Hi We have tow sites, Site a Primary Site with Two Domain controllers, Melbad01 running all the FSMO roles and MelbAD02 a secondary DC.

The Remote Site has again two domain controllers SYDAD01 and SYDAD02.

Replication is broken and the Netlogon services are not running.

I ran repadmin and it displayed the following results.


H:\>repadmin /showrepl

Repadmin: running command /showrepl against full DC localhost
Melbourne\MELBAD01
DSA Options: IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL
Site Options: (none)
DSA object GUID: ee5149a7-87b4-4ffe-8765-b90a979a0e4f
DSA invocationID: ed2b9bb2-46b9-4aa3-acc7-c07249c2af7f

==== INBOUND NEIGHBORS ======================================

DC=frontline,DC=local
    Sydney\SYDAD01 via RPC
        DSA object GUID: 44fb4ec7-8c38-4118-80d1-844608e38b2b
        Last attempt @ 2016-01-14 10:30:59 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        188 consecutive failure(s).
        Last success @ 2016-01-10 21:00:17.
    Melbourne\MELBAD02 via RPC
        DSA object GUID: 76f5af27-4d67-47db-9a0b-4b4b5bc9a67e
        Last attempt @ 2016-01-14 10:33:48 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        410 consecutive failure(s).
        Last success @ 2016-01-10 20:45:17.

CN=Configuration,DC=xxx,DC=local
    Sydney\SYDAD01 via RPC
        DSA object GUID: 44fb4ec7-8c38-4118-80d1-844608e38b2b
        Last attempt @ 2016-01-14 10:30:59 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        188 consecutive failure(s).
        Last success @ 2016-01-10 21:00:17.
    Melbourne\MELBAD02 via RPC
        DSA object GUID: 76f5af27-4d67-47db-9a0b-4b4b5bc9a67e
        Last attempt @ 2016-01-14 10:36:25 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        233 consecutive failure(s).
        Last success @ 2016-01-10 20:45:17.

CN=Schema,CN=Configuration,DC=xxx,DC=local
    Melbourne\MELBAD02 via RPC
        DSA object GUID: 76f5af27-4d67-47db-9a0b-4b4b5bc9a67e
        Last attempt @ 2016-01-14 09:45:59 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        48 consecutive failure(s).
        Last success @ 2016-01-10 20:45:17.
    Sydney\SYDAD01 via RPC
        DSA object GUID: 44fb4ec7-8c38-4118-80d1-844608e38b2b
        Last attempt @ 2016-01-14 10:30:59 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        188 consecutive failure(s).
        Last success @ 2016-01-10 21:00:17.

DC=DomainDnsZones,DC=xxx,DC=local
    Melbourne\MELBAD02 via RPC
        DSA object GUID: 76f5af27-4d67-47db-9a0b-4b4b5bc9a67e
        Last attempt @ 2016-01-14 09:45:59 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        58 consecutive failure(s).
        Last success @ 2016-01-10 20:45:17.
    Sydney\SYDAD01 via RPC
        DSA object GUID: 44fb4ec7-8c38-4118-80d1-844608e38b2b
        Last attempt @ 2016-01-14 10:30:59 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        186 consecutive failure(s).
        Last success @ 2016-01-10 21:00:17.

DC=ForestDnsZones,DC=xxx,DC=local
    Melbourne\MELBAD02 via RPC
        DSA object GUID: 76f5af27-4d67-47db-9a0b-4b4b5bc9a67e
        Last attempt @ 2016-01-14 09:45:59 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        48 consecutive failure(s).
        Last success @ 2016-01-10 20:45:17.
    Sydney\SYDAD01 via RPC
        DSA object GUID: 44fb4ec7-8c38-4118-80d1-844608e38b2b
        Last attempt @ 2016-01-14 10:30:59 failed, result 8457 (0x2109):
            The destination server is currently rejecting replication requests.
        186 consecutive failure(s).
        Last success @ 2016-01-10 21:00:17.

Source: Melbourne\MELBAD02
******* 410 CONSECUTIVE FAILURES since 2016-01-10 20:45:17
Last error: 8457 (0x2109):
            The destination server is currently rejecting replication requests.

Source: Sydney\SYDAD02
******* 1 CONSECUTIVE FAILURES since 2016-01-14 10:37:54
Last error: 8457 (0x2109):
            The destination server is currently rejecting replication requests.

Naming Context: DC=xxx,DC=local
Source: Sydney\SYDAD02
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: DC=DomainDnsZones,DC=xxx,DC=local
Source: Sydney\SYDAD02
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: DC=ForestDnsZones,DC=xxx,DC=local
Source: Sydney\SYDAD02
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: CN=Configuration,DC=xxx,DC=local
Source: Sydney\SYDAD02
******* WARNING: KCC could not add this REPLICA LINK due to error.

Source: Sydney\SYDAD01
******* 188 CONSECUTIVE FAILURES since 2016-01-10 21:00:17
Last error: 8457 (0x2109):
            The destination server is currently rejecting replication requests.


And Ran DCDIAG with the following Results.
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

H:\>dcdiag

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = MELBAD01
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Melbourne\MELBAD01
      Starting test: Connectivity
         ......................... MELBAD01 passed test Connectivity

Doing primary tests

   Testing server: Melbourne\MELBAD01
      Starting test: Advertising
         Warning: DsGetDcName returned information for
         \\SYDAD02.xxx.local, when we were trying to reach MELBAD01.
         SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.
         ......................... MELBAD01 failed test Advertising
      Starting test: FrsEvent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... MELBAD01 passed test FrsEvent
      Starting test: DFSREvent
         ......................... MELBAD01 passed test DFSREvent
      Starting test: SysVolCheck
         ......................... MELBAD01 passed test SysVolCheck
      Starting test: KccEvent
         A warning event occurred.  EventID: 0x80000785
            Time Generated: 01/14/2016   11:07:54
            Event String:
            The attempt to establish a replication link for the following writab
le directory partition failed.
         A warning event occurred.  EventID: 0x80000785
            Time Generated: 01/14/2016   11:07:54
            Event String:
            The attempt to establish a replication link for the following writab
le directory partition failed.
         A warning event occurred.  EventID: 0x80000785
            Time Generated: 01/14/2016   11:07:54
            Event String:
            The attempt to establish a replication link for the following writab
le directory partition failed.
         A warning event occurred.  EventID: 0x80000785
            Time Generated: 01/14/2016   11:07:54
            Event String:
            The attempt to establish a replication link for the following writab
le directory partition failed.
         A warning event occurred.  EventID: 0x80000785
            Time Generated: 01/14/2016   11:07:54
            Event String:
            The attempt to establish a replication link for the following writab
le directory partition failed.
         A warning event occurred.  EventID: 0x8000082C
            Time Generated: 01/14/2016   11:16:59
            Event String:
         ......................... MELBAD01 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         ......................... MELBAD01 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         ......................... MELBAD01 passed test MachineAccount
      Starting test: NCSecDesc
         Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
            Replicating Directory Changes In Filtered Set
         access rights for the naming context:
         DC=ForestDnsZones,DC=xxx,DC=local
         Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
            Replicating Directory Changes In Filtered Set
         access rights for the naming context:
         DC=DomainDnsZones,DC=xxx,DC=local
         ......................... MELBAD01 failed test NCSecDesc
      Starting test: NetLogons
         ......................... MELBAD01 passed test NetLogons
      Starting test: ObjectsReplicated
         ......................... MELBAD01 passed test ObjectsReplicated
      Starting test: Replications
         [Replications Check,Replications Check] Inbound replication is
         disabled.
         To correct, run "repadmin /options MELBAD01 -DISABLE_INBOUND_REPL"
         [Replications Check,MELBAD01] Outbound replication is disabled.
         To correct, run "repadmin /options MELBAD01 -DISABLE_OUTBOUND_REPL"
         ......................... MELBAD01 failed test Replications
      Starting test: RidManager
         ......................... MELBAD01 passed test RidManager
      Starting test: Services
            NETLOGON Service is paused on [MELBAD01]
         ......................... MELBAD01 failed test Services
      Starting test: SystemLog
         An error event occurred.  EventID: 0xC0002719
            Time Generated: 01/14/2016   10:24:38
            Event String:
            DCOM was unable to communicate with the computer 203.134.64.66 using
 any of the configured protocols.
         An error event occurred.  EventID: 0xC0002719
            Time Generated: 01/14/2016   10:24:39
            Event String:
            DCOM was unable to communicate with the computer 203.134.65.66 using
 any of the configured protocols.
         ......................... MELBAD01 failed test SystemLog
      Starting test: VerifyReferences
         ......................... MELBAD01 passed test VerifyReferences


   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test
         CrossRefValidation

   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test
         CrossRefValidation

   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

   Running partition tests on : xxx
      Starting test: CheckSDRefDom
         ......................... frontline passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... frontline passed test CrossRefValidation

   Running enterprise tests on : xxx.local
      Starting test: LocatorCheck


Please advise how I can fix this issue.
Avatar of Matt Minor
Matt Minor
Flag of Canada image

Any significant administrative changes made on the domain prior to this issue presenting?
Avatar of Simba01
Simba01

ASKER

Hi Matt,

Yes, The Melb01 Domain controller and Sydad01 are Virtual Servers. The Raid controller Failed and we restored the Servers from Veeam Backup the ran successfully the night before.
Avatar of Simba01

ASKER

Sorry Both Melbad01 and Melbad02 were recovered and not Sydad01 and Sydad02.
Server running all the fSMO roles is Melbad01, that was restore from Image backups..
Aha! Well it looks like inbound/outbound replication is disabled on MELBAD01

DSA Options: IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL

To resolve, issue the following commands on MELBAD01:
repadmin /options localhost -DISABLE_OUTBOUND_REPL
epadmin /options localhost -DISABLE_INBOUND_REPL


Then verify using
repadmin /OPTIONS *

You should now see:
DSA Options: IS_GC
Avatar of Simba01

ASKER

Hi Matt,

Below are the results.

H:\>repadmin /OPTIONS *

Repadmin: running command /OPTIONS against full DC MELBAD01.xxx.local
Current DSA Options: IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL

Repadmin: running command /OPTIONS against full DC MELBAD02.xxx.local
Current DSA Options: IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL

Repadmin: running command /OPTIONS against full DC SYDAD01.xxx.local
Current DSA Options: IS_GC

Repadmin: running command /OPTIONS against full DC SYDAD02.xxx.local
Current DSA Options: IS_GC
Ok so SYDAD01/02 are good, execute the same on the remaining 2 servers and re-verify.

All should report back IS_GC
Then give things 30 minutes to re-converge and test your domain services - ie user login, shared folder access, etc - and you should be back in business.

For your reference: this happens quite often as more and more people are virtualizing resources.
Here is a great article for reference should this ever occur again:
https://guylabs.ch/2013/11/06/vmware-snapshot-and-recovery-fix-active-directory-replication/
Avatar of Simba01

ASKER

Thank you Matt,

I enabled replication and got the following results.
H:\>repadmin /OPTIONS *

Repadmin: running command /OPTIONS against full DC MELBAD01.frontline.local
Current DSA Options: IS_GC

Repadmin: running command /OPTIONS against full DC MELBAD02.frontline.local
Current DSA Options: IS_GC

Repadmin: running command /OPTIONS against full DC SYDAD01.frontline.local
Current DSA Options: IS_GC

Repadmin: running command /OPTIONS against full DC SYDAD02.frontline.local
Current DSA Options: IS_GC

But after a while the Replication gets disabled again.
As below:
H:\>repadmin /OPTIONS *

Repadmin: running command /OPTIONS against full DC MELBAD01.frontline.local
Current DSA Options: IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL

Repadmin: running command /OPTIONS against full DC MELBAD02.frontline.local
Current DSA Options: IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL

Repadmin: running command /OPTIONS against full DC SYDAD01.frontline.local
Current DSA Options: IS_GC

Repadmin: running command /OPTIONS against full DC SYDAD02.frontline.local
Current DSA Options: IS_GC
Avatar of Simba01

ASKER

I re Ran Dcdiag, and Get Melbad01 has replication disabled.


H:\>dcdiag

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = MELBAD01
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Melbourne\MELBAD01
      Starting test: Connectivity
         ......................... MELBAD01 passed test Connectivity

Doing primary tests

   Testing server: Melbourne\MELBAD01
      Starting test: Advertising
         Warning: DsGetDcName returned information for
         \\SYDAD01.frontline.local, when we were trying to reach MELBAD01.
         SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.
         ......................... MELBAD01 failed test Advertising
      Starting test: FrsEvent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... MELBAD01 passed test FrsEvent
      Starting test: DFSREvent
         ......................... MELBAD01 passed test DFSREvent
      Starting test: SysVolCheck
         ......................... MELBAD01 passed test SysVolCheck
      Starting test: KccEvent
         An error event occurred.  EventID: 0xC000082F
            Time Generated: 01/14/2016   14:31:00
            Event String:
            During an Active Directory Domain Services replication request, the
local domain controller (DC) identified a remote DC which has received replicati
on data from the local DC using already-acknowledged USN tracking numbers.
         A warning event occurred.  EventID: 0x80000459
            Time Generated: 01/14/2016   14:31:00
            Event String: Inbound replication has been disabled by the user.
         A warning event occurred.  EventID: 0x8000045B
            Time Generated: 01/14/2016   14:31:00
            Event String: Outbound replication has been disabled by the user.
         A warning event occurred.  EventID: 0x80000495
            Time Generated: 01/14/2016   14:31:00
            Event String:
            Internal event: Active Directory Domain Services has encountered the
 following exception and associated parameters.
         A warning event occurred.  EventID: 0x80000785
            Time Generated: 01/14/2016   14:37:59
            Event String:
            The attempt to establish a replication link for the following writab
le directory partition failed.
         A warning event occurred.  EventID: 0x80000785
            Time Generated: 01/14/2016   14:37:59
            Event String:
            The attempt to establish a replication link for the following writab
le directory partition failed.
         A warning event occurred.  EventID: 0x80000785
            Time Generated: 01/14/2016   14:37:59
            Event String:
            The attempt to establish a replication link for the following writab
le directory partition failed.
         A warning event occurred.  EventID: 0x80000785
            Time Generated: 01/14/2016   14:37:59
            Event String:
            The attempt to establish a replication link for the following writab
le directory partition failed.
         A warning event occurred.  EventID: 0x80000785
            Time Generated: 01/14/2016   14:37:59
            Event String:
            The attempt to establish a replication link for the following writab
le directory partition failed.
         ......................... MELBAD01 failed test KccEvent
      Starting test: KnowsOfRoleHolders
         ......................... MELBAD01 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         ......................... MELBAD01 passed test MachineAccount
      Starting test: NCSecDesc
         Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
            Replicating Directory Changes In Filtered Set
         access rights for the naming context:
         DC=ForestDnsZones,DC=frontline,DC=local
         Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
            Replicating Directory Changes In Filtered Set
         access rights for the naming context:
         DC=DomainDnsZones,DC=frontline,DC=local
         ......................... MELBAD01 failed test NCSecDesc
      Starting test: NetLogons
         ......................... MELBAD01 passed test NetLogons
      Starting test: ObjectsReplicated
         ......................... MELBAD01 passed test ObjectsReplicated
      Starting test: Replications
         [Replications Check,Replications Check] Inbound replication is
         disabled.
         To correct, run "repadmin /options MELBAD01 -DISABLE_INBOUND_REPL"
         [Replications Check,MELBAD01] Outbound replication is disabled.
         To correct, run "repadmin /options MELBAD01 -DISABLE_OUTBOUND_REPL"
         ......................... MELBAD01 failed test Replications
      Starting test: RidManager
         ......................... MELBAD01 passed test RidManager
      Starting test: Services
            NETLOGON Service is paused on [MELBAD01]
         ......................... MELBAD01 failed test Services
      Starting test: SystemLog
         A warning event occurred.  EventID: 0x8000001D
            Time Generated: 01/14/2016   14:00:53
            Event String:
            The Key Distribution Center (KDC) cannot find a suitable certificate
 to use for smart card logons, or the KDC certificate could not be verified. Sma
rt card logon may not function correctly if this problem is not resolved. To cor
rect this problem, either verify the existing KDC certificate using certutil.exe
 or enroll for a new KDC certificate.
         ......................... MELBAD01 passed test SystemLog
      Starting test: VerifyReferences
         ......................... MELBAD01 passed test VerifyReferences


   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test
         CrossRefValidation

   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test
         CrossRefValidation

   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

   Running partition tests on : xxx
      Starting test: CheckSDRefDom
         ......................... xxx passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... frontline passed test CrossRefValidation

   Running enterprise tests on : xxx.local
      Starting test: LocatorCheck
ASKER CERTIFIED SOLUTION
Avatar of Matt Minor
Matt Minor
Flag of Canada image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Things are working again for you?
Avatar of Simba01

ASKER

Thanks Matt, I  Restore the Active Directory form Symantec backups, and the Replication is working fine now.