AD Domain Controller not replicating

I have 3 AD servers,all virtual but on two different boxes:
1 and 3 are replicating to each other and now to 2
2 is not replicating to 1 or 3 and also holds DHCP ( I do not want  to  lose this, but I do have DHCP running on 3 also with different zones)H
1 and 3 will replicate to 2 but not the other way around.
1 and 3 are 2008 R2
2 is 2012
I have lingering objects error messages that I can not seem to clean up.

1 is the RID Master
3 is the PDC
2 does not hold a role

Same forest/same domain
All are GCs

Help please!!
eliaexpertsAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

MaheshArchitectCommented:
on PDC server run dcdiag /v and post output here
eliaexpertsAuthor Commented:
Here you go!

            HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Strict Replic
ation Consistency

             Replication errors between DCs sharing a common partition can preve
nt user and compter acounts, trust relationships, their passwords, security grou
ps, security group memberships and other Active Directory Domain Services config
uration data to vary between DCs, affecting the ability to log on, find objects
of interest and perform other critical operations. These inconsistencies are res
olved once replication errors are resolved.  DCs that fail to inbound replicate
deleted objects within tombstone lifetime number of days will remain inconsisten
t until lingering objects are manually removed by an administrator from each loc
al DC.

             Lingering objects may be prevented by ensuring that all domain cont
rollers in the forest are running Active Directory Domain Services, are connecte
d by a spanning tree connection topology and perform inbound replication before
Tombstone Live number of days pass.
         An error event occurred.  EventID: 0xC00007C4
            Time Generated: 03/26/2018   06:59:28
            Event String:
            Active Directory Domain Services Replication encountered the existen
ce of objects in the following partition that have been deleted from the local d
omain controllers (DCs) Active Directory Domain Services database.  Not all dire
ct or transitive replication partners replicated in the deletion before the tomb
stone lifetime number of days passed.  Objects that have been deleted and garbag
e collected from an Active Directory Domain Services partition but still exist i
n the writable partitions of other DCs in the same domain, or read-only partitio
ns of global catalog servers in other domains in the forest are known as "linger
ing objects".


            Source domain controller:
            164b078e-6952-4065-9874-7a3c99250264._msdcs.stalcuin.com
            Object:
            DC=16.4,DC=16.172.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=
stalcuin,DC=com
            Object GUID:
            a9de8b80-c86a-4c3b-ae6b-9b366bbc1219  This event is being logged bec
ause the source DC contains a lingering object which does not exist on the local
 DCs Active Directory Domain Services database.  This replication attempt has be
en blocked.

             The best solution to this problem is to identify and remove all lin
gering objects in the forest.


            User Action:

            Remove Lingering Objects:

             The action plan to recover from this error can be found at http://s
upport.microsoft.com/?id=314282.

             If both the source and destination DCs are Windows Server 2003 DCs,
 then install the support tools included on the installation CD.  To see which o
bjects would be deleted without actually performing the deletion run "repadmin /
removelingeringobjects <Source DC> <Destination DC DSA GUID> <NC> /ADVISORY_MODE
". The eventlogs on the source DC will enumerate all lingering objects.  To remo
ve lingering objects from a source domain controller run "repadmin /removelinger
ingobjects <Source DC> <Destination DC DSA GUID> <NC>".

             If either source or destination DC is a Windows 2000 Server DC, the
n more information on how to remove lingering objects on the source DC can be fo
und at http://support.microsoft.com/?id=314282 or from your Microsoft support pe
rsonnel.

             If you need Active Directory Domain Services replication to functio
n immediately at all costs and don't have time to remove lingering objects, enab
le loose replication consistency by unsetting the following registry key:

            Registry Key:
            HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Strict Replic
ation Consistency

             Replication errors between DCs sharing a common partition can preve
nt user and compter acounts, trust relationships, their passwords, security grou
ps, security group memberships and other Active Directory Domain Services config
uration data to vary between DCs, affecting the ability to log on, find objects
of interest and perform other critical operations. These inconsistencies are res
olved once replication errors are resolved.  DCs that fail to inbound replicate
deleted objects within tombstone lifetime number of days will remain inconsisten
t until lingering objects are manually removed by an administrator from each loc
al DC.

             Lingering objects may be prevented by ensuring that all domain cont
rollers in the forest are running Active Directory Domain Services, are connecte
d by a spanning tree connection topology and perform inbound replication before
Tombstone Live number of days pass.
         ......................... SADC3 failed test KccEvent
      Starting test: KnowsOfRoleHolders
         Role Schema Owner = CN=NTDS Settings,CN=SADC1,CN=Servers,CN=Default-Fir
st-Site-Name,CN=Sites,CN=Configuration,DC=stalcuin,DC=com
         Role Domain Owner = CN=NTDS Settings,CN=SADC1,CN=Servers,CN=Default-Fir
st-Site-Name,CN=Sites,CN=Configuration,DC=stalcuin,DC=com
         Role PDC Owner = CN=NTDS Settings,CN=SADC3,CN=Servers,CN=Default-First-
Site-Name,CN=Sites,CN=Configuration,DC=stalcuin,DC=com
         Role Rid Owner = CN=NTDS Settings,CN=SADC1,CN=Servers,CN=Default-First-
Site-Name,CN=Sites,CN=Configuration,DC=stalcuin,DC=com
         Role Infrastructure Update Owner = CN=NTDS Settings,CN=SADC1,CN=Servers
,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=stalcuin,DC=com
         ......................... SADC3 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         Checking machine account for DC SADC3 on DC SADC3.
         * SPN found :LDAP/SADC3.stalcuin.com/stalcuin.com
         * SPN found :LDAP/SADC3.stalcuin.com
         * SPN found :LDAP/SADC3
         * SPN found :LDAP/SADC3.stalcuin.com/STALCUIN
         * SPN found :LDAP/e09dbf38-0fb3-405f-96df-d3574a240187._msdcs.stalcuin.
com
         * SPN found :E3514235-4B06-11D1-AB04-00C04FC2DCD2/e09dbf38-0fb3-405f-96
df-d3574a240187/stalcuin.com
         * SPN found :HOST/SADC3.stalcuin.com/stalcuin.com
         * SPN found :HOST/SADC3.stalcuin.com
         * SPN found :HOST/SADC3
         * SPN found :HOST/SADC3.stalcuin.com/STALCUIN
         * SPN found :GC/SADC3.stalcuin.com/stalcuin.com
         ......................... SADC3 passed test MachineAccount
      Starting test: NCSecDesc
         * Security Permissions check for all NC's on DC SADC3.
         The forest is not ready for RODC. Will skip checking ERODC ACEs.
         * Security Permissions Check for
           DC=ForestDnsZones,DC=stalcuin,DC=com
            (NDNC,Version 3)
         Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
            Replicating Directory Changes In Filtered Set
         access rights for the naming context:
         DC=ForestDnsZones,DC=stalcuin,DC=com
         * Security Permissions Check for
           DC=DomainDnsZones,DC=stalcuin,DC=com
            (NDNC,Version 3)
         Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
            Replicating Directory Changes In Filtered Set
         access rights for the naming context:
         DC=DomainDnsZones,DC=stalcuin,DC=com
         * Security Permissions Check for
           CN=Schema,CN=Configuration,DC=stalcuin,DC=com
            (Schema,Version 3)
         * Security Permissions Check for
           CN=Configuration,DC=stalcuin,DC=com
            (Configuration,Version 3)
         * Security Permissions Check for
           DC=stalcuin,DC=com
            (Domain,Version 3)
         ......................... SADC3 failed test NCSecDesc
      Starting test: NetLogons
         * Network Logons Privileges Check
         Verified share \\SADC3\netlogon
         Verified share \\SADC3\sysvol
         ......................... SADC3 passed test NetLogons
      Starting test: ObjectsReplicated
         SADC3 is in domain DC=stalcuin,DC=com
         Checking for CN=SADC3,OU=Domain Controllers,DC=stalcuin,DC=com in domai
n DC=stalcuin,DC=com on 1 servers
            Object is up-to-date on all servers.
         Checking for CN=NTDS Settings,CN=SADC3,CN=Servers,CN=Default-First-Site
-Name,CN=Sites,CN=Configuration,DC=stalcuin,DC=com in domain CN=Configuration,DC
=stalcuin,DC=com on 1 servers
            Object is up-to-date on all servers.
         ......................... SADC3 passed test ObjectsReplicated
      Test omitted by user request: OutboundSecureChannels
      Starting test: Replications
         * Replications Check
         [Replications Check,SADC3] A recent replication attempt failed:
            From SADC2 to SADC3
            Naming Context: DC=DomainDnsZones,DC=stalcuin,DC=com
            The replication generated an error (8606):
            Insufficient attributes were given to create an object. This object
may not exist because it may have been deleted and already garbage collected.

            The failure occurred at 2018-03-26 06:59:28.
            The last success occurred at (never).
            198 failures have occurred since the last success.
         [Replications Check,SADC3] A recent replication attempt failed:
            From SADC2 to SADC3
            Naming Context: DC=stalcuin,DC=com
            The replication generated an error (8606):
            Insufficient attributes were given to create an object. This object
may not exist because it may have been deleted and already garbage collected.

            The failure occurred at 2018-03-26 06:59:22.
            The last success occurred at (never).
            3179 failures have occurred since the last success.
         ......................... SADC3 failed test Replications
      Starting test: RidManager
         * Available RID Pool for the Domain is 5601 to 1073741823
         * SADC1.stalcuin.com is the RID Master
         * DsBind with RID Master was successful
         * rIDAllocationPool is 3101 to 3600
         * rIDPreviousAllocationPool is 3101 to 3600
         * rIDNextRID: 3266
         ......................... SADC3 passed test RidManager
      Starting test: Services
         * Checking Service: EventSystem
         * Checking Service: RpcSs
         * Checking Service: NTDS
         * Checking Service: DnsCache
         * Checking Service: NtFrs
         * Checking Service: IsmServ
         * Checking Service: kdc
         * Checking Service: SamSs
         * Checking Service: LanmanServer
         * Checking Service: LanmanWorkstation
         * Checking Service: w32time
         * Checking Service: NETLOGON
         ......................... SADC3 passed test Services
      Starting test: SystemLog
         * The System Event log test
         A warning event occurred.  EventID: 0x8000001D
            Time Generated: 03/26/2018   06:10:54
            Event String:
            The Key Distribution Center (KDC) cannot find a suitable certificate
 to use for smart card logons, or the KDC certificate could not be verified. Sma
rt card logon may not function correctly if this problem is not resolved. To cor
rect this problem, either verify the existing KDC certificate using certutil.exe
 or enroll for a new KDC certificate.
         An error event occurred.  EventID: 0xC0002719
            Time Generated: 03/26/2018   06:11:12
            Event String:
            DCOM was unable to communicate with the computer alcuin-dc1.stalcuin
.com using any of the configured protocols.
         ......................... SADC3 failed test SystemLog
      Test omitted by user request: Topology
      Test omitted by user request: VerifyEnterpriseReferences
      Starting test: VerifyReferences
         The system object reference (serverReference)
         CN=SADC3,OU=Domain Controllers,DC=stalcuin,DC=com and backlink on
         CN=SADC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuratio
n,DC=stalcuin,DC=com
         are correct.
         The system object reference (serverReferenceBL)
         CN=SADC3,CN=Domain System Volume (SYSVOL share),CN=File Replication Ser
vice,CN=System,DC=stalcuin,DC=com
         and backlink on
         CN=NTDS Settings,CN=SADC3,CN=Servers,CN=Default-First-Site-Name,CN=Site
s,CN=Configuration,DC=stalcuin,DC=com
         are correct.
         The system object reference (frsComputerReferenceBL)
         CN=SADC3,CN=Domain System Volume (SYSVOL share),CN=File Replication Ser
vice,CN=System,DC=stalcuin,DC=com
         and backlink on CN=SADC3,OU=Domain Controllers,DC=stalcuin,DC=com are
         correct.
         ......................... SADC3 passed test VerifyReferences
      Test omitted by user request: VerifyReplicas

      Test omitted by user request: DNS
      Test omitted by user request: DNS

   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test
         CrossRefValidation

   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test
         CrossRefValidation

   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

   Running partition tests on : stalcuin
      Starting test: CheckSDRefDom
         ......................... stalcuin passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... stalcuin passed test CrossRefValidation

   Running enterprise tests on : stalcuin.com
      Test omitted by user request: DNS
      Test omitted by user request: DNS
      Starting test: LocatorCheck
         GC Name: \\SADC3.stalcuin.com
         Locator Flags: 0xe00033fd
         PDC Name: \\SADC3.stalcuin.com
         Locator Flags: 0xe00033fd
         Time Server Name: \\SADC3.stalcuin.com
         Locator Flags: 0xe00033fd
         Preferred Time Server Name: \\SADC3.stalcuin.com
         Locator Flags: 0xe00033fd
         KDC Name: \\SADC3.stalcuin.com
         Locator Flags: 0xe00033fd
         ......................... stalcuin.com passed test LocatorCheck
      Starting test: Intersite
         Skipping site Default-First-Site-Name, this site is outside the scope
         provided by the command line arguments provided.
         ......................... stalcuin.com passed test Intersite

C:\Windows\system32>
MaheshArchitectCommented:
Ping below and find out which dc is this (source), it is having lingering objects
164b078e-6952-4065-9874-7a3c99250264._msdcs.stalcuin.com

Now ensure that you have PDC role on another DC and then run repadmin command from elevated cmd on PDC to remove lingering objects
use tool - download link - https://www.microsoft.com/en-us/download/details.aspx?id=56051
How to - https://blogs.technet.microsoft.com/askds/2014/09/15/remove-lingering-objects-that-cause-ad-replication-error-8606-and-friends/

Else manually do with command:
The command is highlighted in below article:
http://www.dell.com/support/article/in/en/indhs1/sln283355/windows-server-how-to-detect-and-remove-lingering-objects-from-an-active-directory-domain-controller?lang=en

Once you completely removed lingering objects,
decommission DC with lingering objects from AD gracefully or forcefully (dcpromo /forceremoval and followed by metadata cleanup)  and promote it again
OR
Set below registry on PDC server and ensure AD replication is happening between both servers
https://support.microsoft.com/en-in/help/2020053/troubleshooting-ad-replication-error-8614-the-active-directory-cannot
OR
Cloud as a Security Delivery Platform for MSSPs

Every Managed Security Service Provider (MSSP) needs a platform to deliver effective and efficient security-as-a-service to their customers. Scale, elasticity and profitability are a few of the many features that a Cloud platform offers. View our on-demand webinar to learn more!

eliaexpertsAuthor Commented:
I ran the LOL again (have done that before) and this is what I get:

[03/26 09:21:44] Welcome to Lingering Object Liquidator version 2.0.21!
[03/26 09:21:44] Status messages will be logged in this window as well as in C:\Users\eliajohn\Desktop\LOL_20180326092129.log.
[03/26 09:21:44] Could not read HKLM\SOFTWARE\LingeringObjectLiquidator => DetectionTimeoutPerDCSeconds. Using the default value of 300 seconds. (HKLM.OpenSubKey returned null.)
[03/26 09:21:44] Could not read HKLM\SOFTWARE\LingeringObjectLiquidator => ThreadCount. Using the default value of 8 threads. (HKLM.OpenSubKey returned null.)
[03/26 09:21:44] Click the "Detect AD Topology" button to begin.
[03/26 09:22:29] Detecting AD Topology; Please wait...
[03/26 09:22:31] Local Domain Name: stalcuin.com
[03/26 09:22:31] Forest Name: stalcuin.com
[03/26 09:22:32] Forest contains 1 domains.
[03/26 09:22:32] Domain stalcuin.com contains 3 domain controllers.
[03/26 09:22:32] SADC1.stalcuin.com is a writable global catalog for the stalcuin.com domain. Using it to acquire a list of naming contexts...
[03/26 09:22:32] 4 naming contexts found. (Omitting schema partition.)
[03/26 09:22:32] AD Topology Detection finished in 3.14 seconds. Success.
[03/26 09:23:02] Detecting Lingering Objects using SADC1.stalcuin.com as the Reference DC; Please wait...
[03/26 09:23:02] DSA GUID of the Reference DC SADC1.stalcuin.com is ea940e1e-054c-4e62-aba5-eb11ca2debcc.
[03/26 09:23:03] Created event log subscription to SADC2.stalcuin.com.
[03/26 09:23:03] DsBind success to DC SADC2.stalcuin.com.
[03/26 09:23:12] Call to DsReplicaVerifyObjects against DC SADC2.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com succeeded.
[03/26 09:23:12] UnBind from DC SADC2.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (1 more NCs to complete.)
[03/26 09:23:12] Lingering object detection has completed for a naming context on SADC2.stalcuin.com.
[03/26 09:23:13] All requested naming contexts on domain controller SADC2.stalcuin.com have completed lingering object detection.
[03/26 09:23:13] Lingering Object Detection finished in 10.72 seconds.
[03/26 09:23:13] 0 lingering objects were detected.
[03/26 09:23:26] Detecting Lingering Objects using SADC3.stalcuin.com as the Reference DC; Please wait...
[03/26 09:23:26] DSA GUID of the Reference DC SADC3.stalcuin.com is e09dbf38-0fb3-405f-96df-d3574a240187.
[03/26 09:23:26] Created event log subscription to SADC2.stalcuin.com.
[03/26 09:23:26] DsBind success to DC SADC2.stalcuin.com.
[03/26 09:23:34] Call to DsReplicaVerifyObjects against DC SADC2.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com succeeded.
[03/26 09:23:34] UnBind from DC SADC2.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (1 more NCs to complete.)
[03/26 09:23:35] Lingering object detection has completed for a naming context on SADC2.stalcuin.com.
[03/26 09:23:35] All requested naming contexts on domain controller SADC2.stalcuin.com have completed lingering object detection.
[03/26 09:23:35] Lingering Object Detection finished in 9.57 seconds.
[03/26 09:23:35] 0 lingering objects were detected.
eliaexpertsAuthor Commented:
I recently set "strict replication" on DC2 (the one with the issues).  should I set the registry key to 0 to allow for the clean up of lingering objects?

Thank you for your help!!
MaheshArchitectCommented:
Yes, that's right

Are you getting success in removing lingering objects?

the output u written only detection did,
eliaexpertsAuthor Commented:
no, the logs from the LOL state 0 lingering objects, that is why I was asking if I should set "strict replication" to 0 on DC2 to enable replication.
MaheshArchitectCommented:
yes
After that check if tool is able to find any lingering objects
If found cleanup it
After that check if ad replication is working
If required, set registry on PDC server as mentioned earlier and check if replication happening between two
If still you face issues, demote this dc  gracefully or forcefully, do metadata cleanup and promote it again
DrDave242Senior Support EngineerCommented:
no, the logs from the LOL state 0 lingering objects

When you ran the LOL, did you select [Scan all NCs] from the Naming Context dropdown at the top before kicking off the scan? The output you posted appears to show that only one naming context was checked.
eliaexpertsAuthor Commented:
Yes, I selected "scan entire forest" (that was my choice.  Running it on the PDC (3).  Also tried every other combination I could think of..

LOL-run-on-SADC3.JPG
eliaexpertsAuthor Commented:
Sorry the pic before was run on SADC2..this one is from SADC3LOL-run-on-SADC3.JPG
DrDave242Senior Support EngineerCommented:
The dropdown I'm asking about is the one at the very top that's currently set to the Configuration naming context. Set that to [Scan all NCs] and run it again. The events you posted above appear to indicate that lingering objects were found in the Domain and DomainDnsZones naming contexts, so it's no surprise that the Configuration naming context comes up clean.
eliaexpertsAuthor Commented:
I turned off strict replication on DC2 and ran LOL on DC3 with the same results.

[03/26 10:01:34] Welcome to Lingering Object Liquidator version 2.0.21!
[03/26 10:01:34] Status messages will be logged in this window as well as in C:\Users\eliajohn\Desktop\LOL_20180326100133.log.
[03/26 10:01:34] Could not read HKLM\SOFTWARE\LingeringObjectLiquidator => DetectionTimeoutPerDCSeconds. Using the default value of 300 seconds. (HKLM.OpenSubKey returned null.)
[03/26 10:01:34] Could not read HKLM\SOFTWARE\LingeringObjectLiquidator => ThreadCount. Using the default value of 8 threads. (HKLM.OpenSubKey returned null.)
[03/26 10:01:34] Click the "Detect AD Topology" button to begin.
[03/26 10:01:43] Detecting AD Topology; Please wait...
[03/26 10:01:43] Local Domain Name: stalcuin.com
[03/26 10:01:43] Forest Name: stalcuin.com
[03/26 10:01:44] Forest contains 1 domains.
[03/26 10:01:44] Domain stalcuin.com contains 3 domain controllers.
[03/26 10:01:44] SADC1.stalcuin.com is a writable global catalog for the stalcuin.com domain. Using it to acquire a list of naming contexts...
[03/26 10:01:44] 4 naming contexts found. (Omitting schema partition.)
[03/26 10:01:44] AD Topology Detection finished in 0.45 seconds. Success.
[03/26 10:01:51] Detecting Lingering Objects using SADC3.stalcuin.com as the Reference DC; Please wait...
[03/26 10:01:51] DSA GUID of the Reference DC SADC3.stalcuin.com is e09dbf38-0fb3-405f-96df-d3574a240187.
[03/26 10:01:51] Created event log subscription to SADC2.stalcuin.com.
[03/26 10:01:51] DsBind success to DC SADC2.stalcuin.com.
[03/26 10:01:58] Call to DsReplicaVerifyObjects against DC SADC2.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com succeeded.
[03/26 10:01:58] UnBind from DC SADC2.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (1 more NCs to complete.)
[03/26 10:01:59] Lingering object detection has completed for a naming context on SADC2.stalcuin.com.
[03/26 10:01:59] All requested naming contexts on domain controller SADC2.stalcuin.com have completed lingering object detection.
[03/26 10:01:59] Lingering Object Detection finished in 8.16 seconds.
[03/26 10:01:59] 0 lingering objects were detected.
[03/26 10:02:17] Detecting Lingering Objects using SADC2.stalcuin.com as the Reference DC; Please wait...
[03/26 10:02:17] DSA GUID of the Reference DC SADC2.stalcuin.com is 164b078e-6952-4065-9874-7a3c99250264.
[03/26 10:02:17] Created event log subscription to SADC3.stalcuin.com.
[03/26 10:02:17] DsBind success to DC SADC3.stalcuin.com.
[03/26 10:02:17] WARNING: Call to DsReplicaVerifyObjects against DC SADC3.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com failed! (WARNING: DsReplicaVerifyObjects returned Win32 Error: 8453)
[03/26 10:02:17] UnBind from DC SADC3.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (0 more NCs to complete.)
[03/26 10:02:17] All requested naming contexts on domain controller SADC3.stalcuin.com have completed lingering object detection.
[03/26 10:02:17] Lingering Object Detection finished in 0.12 seconds.
[03/26 10:02:17] 0 lingering objects were detected.
[03/26 10:09:25] Detecting AD Topology; Please wait...
[03/26 10:09:25] Local Domain Name: stalcuin.com
[03/26 10:09:25] Forest Name: stalcuin.com
[03/26 10:09:25] Forest contains 1 domains.
[03/26 10:09:25] Domain stalcuin.com contains 3 domain controllers.
[03/26 10:09:25] SADC1.stalcuin.com is a writable global catalog for the stalcuin.com domain. Using it to acquire a list of naming contexts...
[03/26 10:09:25] 4 naming contexts found. (Omitting schema partition.)
[03/26 10:09:25] AD Topology Detection finished in 0.17 seconds. Success.
[03/26 10:17:56] Detecting Lingering Objects using SADC3.stalcuin.com as the Reference DC; Please wait...
[03/26 10:17:56] DSA GUID of the Reference DC SADC3.stalcuin.com is e09dbf38-0fb3-405f-96df-d3574a240187.
[03/26 10:17:56] Created event log subscription to SADC2.stalcuin.com.
[03/26 10:17:56] DsBind success to DC SADC2.stalcuin.com.
[03/26 10:18:02] Call to DsReplicaVerifyObjects against DC SADC2.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com succeeded.
[03/26 10:18:03] UnBind from DC SADC2.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (1 more NCs to complete.)
[03/26 10:18:03] Lingering object detection has completed for a naming context on SADC2.stalcuin.com.
[03/26 10:18:04] All requested naming contexts on domain controller SADC2.stalcuin.com have completed lingering object detection.
[03/26 10:18:04] Lingering Object Detection finished in 7.14 seconds.
[03/26 10:18:04] 0 lingering objects were detected.
[03/26 10:18:42] Detecting Lingering Objects using SADC1.stalcuin.com as the Reference DC; Please wait...
[03/26 10:18:42] DSA GUID of the Reference DC SADC1.stalcuin.com is ea940e1e-054c-4e62-aba5-eb11ca2debcc.
[03/26 10:18:42] Created event log subscription to SADC2.stalcuin.com.
[03/26 10:18:42] DsBind success to DC SADC2.stalcuin.com.
[03/26 10:18:49] Call to DsReplicaVerifyObjects against DC SADC2.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com succeeded.
[03/26 10:18:49] UnBind from DC SADC2.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (1 more NCs to complete.)
[03/26 10:18:50] Lingering object detection has completed for a naming context on SADC2.stalcuin.com.
[03/26 10:18:50] All requested naming contexts on domain controller SADC2.stalcuin.com have completed lingering object detection.
[03/26 10:18:50] Lingering Object Detection finished in 7.73 seconds.
[03/26 10:18:50] 0 lingering objects were detected.
[03/26 10:19:36] Detecting Lingering Objects using SADC1.stalcuin.com as the Reference DC; Please wait...
[03/26 10:19:36] DSA GUID of the Reference DC SADC1.stalcuin.com is ea940e1e-054c-4e62-aba5-eb11ca2debcc.
[03/26 10:19:36] Skipping Target DC SADC1.stalcuin.com because it is also the Reference DC.
[03/26 10:19:36] Created event log subscription to SADC2.stalcuin.com.
[03/26 10:19:36] Created event log subscription to SADC3.stalcuin.com.
[03/26 10:19:36] DsBind success to DC SADC3.stalcuin.com.
[03/26 10:19:36] WARNING: Call to DsReplicaVerifyObjects against DC SADC3.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com failed! (WARNING: DsReplicaVerifyObjects returned Win32 Error: 8453)
[03/26 10:19:36] UnBind from DC SADC3.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (0 more NCs to complete.)
[03/26 10:19:36] All requested naming contexts on domain controller SADC3.stalcuin.com have completed lingering object detection.
[03/26 10:19:36] DsBind success to DC SADC2.stalcuin.com.
[03/26 10:19:37] Call to DsReplicaVerifyObjects against DC SADC2.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com succeeded.
[03/26 10:19:37] UnBind from DC SADC2.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (1 more NCs to complete.)
[03/26 10:19:38] Lingering object detection has completed for a naming context on SADC2.stalcuin.com.
[03/26 10:19:38] All requested naming contexts on domain controller SADC2.stalcuin.com have completed lingering object detection.
[03/26 10:19:38] Lingering Object Detection finished in 2.31 seconds.
[03/26 10:19:38] 0 lingering objects were detected.
[03/26 12:34:31] Detecting AD Topology; Please wait...
[03/26 12:34:31] Local Domain Name: stalcuin.com
[03/26 12:34:31] Forest Name: stalcuin.com
[03/26 12:34:31] Forest contains 1 domains.
[03/26 12:34:31] Domain stalcuin.com contains 3 domain controllers.
[03/26 12:34:31] SADC1.stalcuin.com is a writable global catalog for the stalcuin.com domain. Using it to acquire a list of naming contexts...
[03/26 12:34:31] 4 naming contexts found. (Omitting schema partition.)
[03/26 12:34:31] AD Topology Detection finished in 0.16 seconds. Success.
[03/26 12:34:46] Detecting AD Topology; Please wait...
[03/26 12:34:46] Local Domain Name: stalcuin.com
[03/26 12:34:46] Forest Name: stalcuin.com
[03/26 12:34:46] Forest contains 1 domains.
[03/26 12:34:46] Domain stalcuin.com contains 3 domain controllers.
[03/26 12:34:46] SADC1.stalcuin.com is a writable global catalog for the stalcuin.com domain. Using it to acquire a list of naming contexts...
[03/26 12:34:46] 4 naming contexts found. (Omitting schema partition.)
[03/26 12:34:46] AD Topology Detection finished in 0.06 seconds. Success.
[03/26 12:34:53] Detecting Lingering Objects using SADC1.stalcuin.com as the Reference DC; Please wait...
[03/26 12:34:53] DSA GUID of the Reference DC SADC1.stalcuin.com is ea940e1e-054c-4e62-aba5-eb11ca2debcc.
[03/26 12:34:53] Skipping Target DC SADC1.stalcuin.com because it is also the Reference DC.
[03/26 12:34:53] Created event log subscription to SADC3.stalcuin.com.
[03/26 12:34:53] DsBind success to DC SADC3.stalcuin.com.
[03/26 12:34:53] WARNING: Call to DsReplicaVerifyObjects against DC SADC3.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com failed! (WARNING: DsReplicaVerifyObjects returned Win32 Error: 8453)
[03/26 12:34:53] UnBind from DC SADC3.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (0 more NCs to complete.)
[03/26 12:34:53] All requested naming contexts on domain controller SADC3.stalcuin.com have completed lingering object detection.
[03/26 12:34:53] Created event log subscription to SADC2.stalcuin.com.
[03/26 12:34:53] DsBind success to DC SADC2.stalcuin.com.
[03/26 12:34:55] Call to DsReplicaVerifyObjects against DC SADC2.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com succeeded.
[03/26 12:34:55] UnBind from DC SADC2.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (1 more NCs to complete.)
[03/26 12:34:56] Lingering object detection has completed for a naming context on SADC2.stalcuin.com.
[03/26 12:34:56] All requested naming contexts on domain controller SADC2.stalcuin.com have completed lingering object detection.
[03/26 12:34:56] Lingering Object Detection finished in 2.94 seconds.
[03/26 12:34:56] 0 lingering objects were detected.
[03/26 12:36:24] Detecting Lingering Objects using SADC1.stalcuin.com as the Reference DC; Please wait...
[03/26 12:36:24] DSA GUID of the Reference DC SADC1.stalcuin.com is ea940e1e-054c-4e62-aba5-eb11ca2debcc.
[03/26 12:36:24] Skipping Target DC SADC1.stalcuin.com because it is also the Reference DC.
[03/26 12:36:24] Created event log subscription to SADC3.stalcuin.com.
[03/26 12:36:24] Created event log subscription to SADC2.stalcuin.com.
[03/26 12:36:24] DsBind success to DC SADC3.stalcuin.com.
[03/26 12:36:24] DsBind success to DC SADC2.stalcuin.com.
[03/26 12:36:24] WARNING: Call to DsReplicaVerifyObjects against DC SADC3.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com failed! (WARNING: DsReplicaVerifyObjects returned Win32 Error: 8453)
[03/26 12:36:24] UnBind from DC SADC3.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (0 more NCs to complete.)
[03/26 12:36:24] All requested naming contexts on domain controller SADC3.stalcuin.com have completed lingering object detection.
[03/26 12:36:24] Call to DsReplicaVerifyObjects against DC SADC2.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com succeeded.
[03/26 12:36:24] UnBind from DC SADC2.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (1 more NCs to complete.)
[03/26 12:36:25] Lingering object detection has completed for a naming context on SADC2.stalcuin.com.
[03/26 12:36:25] All requested naming contexts on domain controller SADC2.stalcuin.com have completed lingering object detection.
[03/26 12:36:25] Lingering Object Detection finished in 1.2 seconds.
[03/26 12:36:25] 0 lingering objects were detected.
[03/26 12:41:31] Detecting Lingering Objects using SADC1.stalcuin.com as the Reference DC; Please wait...
[03/26 12:41:31] DSA GUID of the Reference DC SADC1.stalcuin.com is ea940e1e-054c-4e62-aba5-eb11ca2debcc.
[03/26 12:41:31] Skipping Target DC SADC1.stalcuin.com because it is also the Reference DC.
[03/26 12:41:31] Created event log subscription to SADC3.stalcuin.com.
[03/26 12:41:31] DsBind success to DC SADC3.stalcuin.com.
[03/26 12:41:31] WARNING: Call to DsReplicaVerifyObjects against DC SADC3.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com failed! (WARNING: DsReplicaVerifyObjects returned Win32 Error: 8453)
[03/26 12:41:31] UnBind from DC SADC3.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (0 more NCs to complete.)
[03/26 12:41:31] All requested naming contexts on domain controller SADC3.stalcuin.com have completed lingering object detection.
[03/26 12:41:31] Created event log subscription to SADC2.stalcuin.com.
[03/26 12:41:31] DsBind success to DC SADC2.stalcuin.com.
[03/26 12:41:32] Call to DsReplicaVerifyObjects against DC SADC2.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com succeeded.
[03/26 12:41:32] UnBind from DC SADC2.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (1 more NCs to complete.)
[03/26 12:41:32] Lingering object detection has completed for a naming context on SADC2.stalcuin.com.
[03/26 12:41:33] All requested naming contexts on domain controller SADC2.stalcuin.com have completed lingering object detection.
[03/26 12:41:33] Lingering Object Detection finished in 1.37 seconds.
[03/26 12:41:33] 0 lingering objects were detected.
[03/26 12:41:36] Detecting AD Topology; Please wait...
[03/26 12:41:36] Local Domain Name: stalcuin.com
[03/26 12:41:36] Forest Name: stalcuin.com
[03/26 12:41:36] Forest contains 1 domains.
[03/26 12:41:36] Domain stalcuin.com contains 3 domain controllers.
[03/26 12:41:36] SADC1.stalcuin.com is a writable global catalog for the stalcuin.com domain. Using it to acquire a list of naming contexts...
[03/26 12:41:36] 4 naming contexts found. (Omitting schema partition.)
[03/26 12:41:36] AD Topology Detection finished in 0.08 seconds. Success.
[03/26 12:41:44] Detecting Lingering Objects using SADC3.stalcuin.com as the Reference DC; Please wait...
[03/26 12:41:44] DSA GUID of the Reference DC SADC3.stalcuin.com is e09dbf38-0fb3-405f-96df-d3574a240187.
[03/26 12:41:44] Created event log subscription to SADC2.stalcuin.com.
[03/26 12:41:44] DsBind success to DC SADC2.stalcuin.com.
[03/26 12:41:49] Call to DsReplicaVerifyObjects against DC SADC2.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com succeeded.
[03/26 12:41:49] UnBind from DC SADC2.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (1 more NCs to complete.)
[03/26 12:41:50] Lingering object detection has completed for a naming context on SADC2.stalcuin.com.
[03/26 12:41:51] All requested naming contexts on domain controller SADC2.stalcuin.com have completed lingering object detection.
[03/26 12:41:51] Lingering Object Detection finished in 7.18 seconds.
[03/26 12:41:51] 0 lingering objects were detected.
[03/26 12:42:04] Detecting Lingering Objects using SADC1.stalcuin.com as the Reference DC; Please wait...
[03/26 12:42:04] DSA GUID of the Reference DC SADC1.stalcuin.com is ea940e1e-054c-4e62-aba5-eb11ca2debcc.
[03/26 12:42:04] Skipping Target DC SADC1.stalcuin.com because it is also the Reference DC.
[03/26 12:42:04] Created event log subscription to SADC2.stalcuin.com.
[03/26 12:42:04] Created event log subscription to SADC3.stalcuin.com.
[03/26 12:42:04] DsBind success to DC SADC2.stalcuin.com.
[03/26 12:42:04] DsBind success to DC SADC3.stalcuin.com.
[03/26 12:42:04] WARNING: Call to DsReplicaVerifyObjects against DC SADC3.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com failed! (WARNING: DsReplicaVerifyObjects returned Win32 Error: 8453)
[03/26 12:42:04] UnBind from DC SADC3.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (0 more NCs to complete.)
[03/26 12:42:04] All requested naming contexts on domain controller SADC3.stalcuin.com have completed lingering object detection.
[03/26 12:42:04] Call to DsReplicaVerifyObjects against DC SADC2.stalcuin.com, NC CN=Configuration,DC=stalcuin,DC=com succeeded.
[03/26 12:42:04] UnBind from DC SADC2.stalcuin.com, waiting for DsReplicaVerifyObjects to finish... (1 more NCs to complete.)
[03/26 12:42:05] Lingering object detection has completed for a naming context on SADC2.stalcuin.com.
[03/26 12:42:05] All requested naming contexts on domain controller SADC2.stalcuin.com have completed lingering object detection.
[03/26 12:42:05] Lingering Object Detection finished in 1.34 seconds.
[03/26 12:42:05] 0 lingering objects were detected.
eliaexpertsAuthor Commented:
Wow! What a difference!
eliaexpertsAuthor Commented:
LOL-on-SADC3-found-413.JPG
eliaexpertsAuthor Commented:
How safe is it to remove these? I have no experience with this!
DrDave242Senior Support EngineerCommented:
They've already been deleted on at least one DC (and most likely two, from the looks of things), so removing them should have no adverse effects. As always, it's a good idea to have a good system-state backup of your DCs before making significant changes to AD, though, on the off chance that something goes very wrong.
MaheshArchitectCommented:
Just remove them all
If you are able to replicate with same DC again, enable strict replication registry again
if you are forced to build new DC, it will automatically create registry value
eliaexpertsAuthor Commented:
Ok...so done. Still showing errors replicating from  DC2

            C:\Windows\system32>repadmin /showrepl
            
            Repadmin: running command /showrepl against full DC localhost
            Default-First-Site-Name\SADC3
            DSA Options: IS_GC
            Site Options: (none)
            DSA object GUID: e09dbf38-0fb3-405f-96df-d3574a240187
            DSA invocationID: 002813c3-4442-4db0-86a8-9c3075f1bb42
            
            ==== INBOUND NEIGHBORS ======================================
            
            DC=stalcuin,DC=com
                Default-First-Site-Name\SADC1 via RPC
                    DSA object GUID: ea940e1e-054c-4e62-aba5-eb11ca2debcc
                    Last attempt @ 2018-03-26 13:24:40 was successful.
                Default-First-Site-Name\SADC2 via RPC
                    DSA object GUID: 164b078e-6952-4065-9874-7a3c99250264
                    Last attempt @ 2018-03-26 13:24:50 failed, result 8614 (0x21a6):
                        The directory service cannot replicate with this server because the
            time since the last replication with this server has exceeded the tombstone life
            time.
                    4009 consecutive failure(s).
                    Last success @ (never).
            
            CN=Configuration,DC=stalcuin,DC=com
                Default-First-Site-Name\SADC2 via RPC
                    DSA object GUID: 164b078e-6952-4065-9874-7a3c99250264
                    Last attempt @ 2018-03-26 12:55:50 was successful.
                Default-First-Site-Name\SADC1 via RPC
                    DSA object GUID: ea940e1e-054c-4e62-aba5-eb11ca2debcc
                    Last attempt @ 2018-03-26 12:56:26 was successful.
            
            CN=Schema,CN=Configuration,DC=stalcuin,DC=com
                Default-First-Site-Name\SADC2 via RPC
                    DSA object GUID: 164b078e-6952-4065-9874-7a3c99250264
                    Last attempt @ 2018-03-26 12:55:50 was successful.
                Default-First-Site-Name\SADC1 via RPC
                    DSA object GUID: ea940e1e-054c-4e62-aba5-eb11ca2debcc
                    Last attempt @ 2018-03-26 12:56:35 was successful.
            
            DC=DomainDnsZones,DC=stalcuin,DC=com
                Default-First-Site-Name\SADC1 via RPC
                    DSA object GUID: ea940e1e-054c-4e62-aba5-eb11ca2debcc
                    Last attempt @ 2018-03-26 13:22:59 was successful.
                Default-First-Site-Name\SADC2 via RPC
                    DSA object GUID: 164b078e-6952-4065-9874-7a3c99250264
                    Last attempt @ 2018-03-26 13:23:20 failed, result 8614 (0x21a6):
                        The directory service cannot replicate with this server because the
            time since the last replication with this server has exceeded the tombstone life
            time.
                    499 consecutive failure(s).
                    Last success @ (never).
            
            DC=ForestDnsZones,DC=stalcuin,DC=com
                Default-First-Site-Name\SADC2 via RPC
                    DSA object GUID: 164b078e-6952-4065-9874-7a3c99250264
                    Last attempt @ 2018-03-26 12:55:50 was successful.
                Default-First-Site-Name\SADC1 via RPC
                    DSA object GUID: ea940e1e-054c-4e62-aba5-eb11ca2debcc
                    Last attempt @ 2018-03-26 12:56:47 was successful.
            
            Source: Default-First-Site-Name\SADC2
            ******* 3989 CONSECUTIVE FAILURES since (never)
            Last error: 8614 (0x21a6):
                        The directory service cannot replicate with this server because the
            time since the last replication with this server has exceeded the tombstone life
            time.
MaheshArchitectCommented:
I don't think it will allow you to replicate to DC2 even if you set divergent partner registry on PDC
Because AD will not allow replication beyond tombstone life
U need to demote it now, cleanup and promote it again
DrDave242Senior Support EngineerCommented:
I'm afraid DC2 is tombstoned now, as it hasn't been able to replicate in such a long time. Your quickest path to resolution at this point is going to be demoting DC2 and re-promoting it.

If a normal demotion won't complete, you'll need to force the demotion (there's a simple checkbox to do this during the demotion process) and perform a metadata cleanup to remove DC2 completely from Active Directory before it can be promoted again.
eliaexpertsAuthor Commented:
I was so afraid you would say that!   I did run Repadmin /syncall /Adep with no errors.  Any hope there?

If I have to force demote (tried running dcpromo last night...would not go gracefully, so I stopped), will it take it off the domain (I am thinking yes from experience).  If so, how will that effect my DHCP server that is located on DC2?
DrDave242Senior Support EngineerCommented:
I did run Repadmin /syncall /Adep with no errors.  Any hope there?

If repadmin /showrepl still shows replication faliures (and I'm assuming it does), then I'm afraid not.

A forced demotion will disjoin the DC from the domain; it'll be in a workgroup after the post-demotion reboot (which brings up a good point: you'll need to know the local admin password on that DC in order to log into it afterward). DHCP server functionality shouldn't be affected, though. The DHCP service won't be able to authorize itself in AD while it's not in the domain, but that shouldn't matter at all if it's the only DHCP server on the network.
MaheshArchitectCommented:
backup dhcp on DC2 and restore it on other DC server
http://www.ngcci.com/move-dhcp-one-server-another-dhcp-server/
ensure dhcp is working on another DC

use forceremoval switch to remove DC
https://www.interfacett.com/blogs/how-to-demote-a-domain-controller-dc-in-windows-server-2012-active-directory-domain-services-ad-ds/
Other way is:
just shut down it for ever
remove domain controller object from PDC master, while doing that select that DC will never come online and AD will clean up everything
You need to remove DC object from AD sites and services and from DNS NS records
now format the server and repurpose it
eliaexpertsAuthor Commented:
Need to keep it up and running.  So I am assuming after I add it back to the domain, DHCP will authorize with one of my other DCs?  I should not have to make any changes to it?  I also have important shared folders on it.
eliaexpertsAuthor Commented:
I do have another DHCP server running on DC3.  Different pools of the same scope.
MaheshArchitectCommented:
demote the server forcefully
it will add it to workgroup and do metadata cleanup
again join it to domain as member server and activate dhcp scope and share access will remain intact after joining to domain

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
DrDave242Senior Support EngineerCommented:
Yep, Mahesh is right. Also, the demote-cleanup-promote process shouldn't take very long, so it's unlikely that any of your DHCP clients will even notice that the DHCP server is down unless you've got a very short lease duration configured.
eliaexpertsAuthor Commented:
That is good to know...those are exactly the kind of questions I had.
eliaexpertsAuthor Commented:
Should I also remove DNS delegation?
DrDave242Senior Support EngineerCommented:
Nope, no need, especially since you'll be promoting it again.
eliaexpertsAuthor Commented:
Ok...looks like I am good!  Rejoined to domain, replication good, dhcp server authorized.

Thank you for your help!!!!
eliaexpertsAuthor Commented:
Both these contributors were an immense help.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
DHCP

From novice to tech pro — start learning today.