Link to home
Start Free TrialLog in
Avatar of James White
James White

asked on

Domain Controller not replicating

I'm getting these dcdiag errors when trying to replicate between DC1 and the other DC's:

Starting test: Replications
         [Replications Check,DC1] A recent replication attempt failed:
            From DC2 to DC1
            Naming Context: DC=domainame,DC=com
            The replication generated an error (8451):
            The replication operation encountered a database error.
            The failure occurred at 2019-01-02 09:25:25.
            The last success occurred at 2018-12-05 10:11:48.
            39398 failures have occurred since the last success.
            A serious error is preventing replication from continuing.
            Consult the error log for further information.
            If a particular object is named, it may be necessary to manually
            modify or delete the object.
            If the condition persists, contact Microsoft Support.
         [Replications Check:DC1] A recent replication attempt failed:
            From DC3 to DC1
            Naming Context: DC=domainame,DC=com
            The replication generated an error (8451):
            The replication operation encountered a database error.
            The failure occurred at 2019-01-02 09:25:39.
            The last success occurred at 2018-12-05 10:11:45.
            70279 failures have occurred since the last success.
            A serious error is preventing replication from continuing.
            Consult the error log for further information.
            If a particular object is named, it may be necessary to manually
            modify or delete the object.

I also got these:

Time Generated: 01/02/2019   09:25:39
            Event String:
            Internal event: Active Directory Domain Services could not update the following object with changes received from the following source directory service. This is because an error occurred during the application of the changes to Active Directory Domain Services on the directory service.
         An error event occurred.  EventID: 0xC000083C
            Time Generated: 01/02/2019   09:25:39
            Event String:
            This event contains REPAIR PROCEDURES for the 1084 event which has previously been logged. This message indicates a specific issue with the consistency of the Active Directory Domain Services database on this replication destination. A database error occurred while applying replicated changes to the following object. The database had unexpected contents, preventing the change from being made.

What can I do to fix this? I tried transferring roles to another DC but that ended up failing. Seizing roles failed too. It's just this one DC1 having this issue.
Avatar of Mahesh
Mahesh
Flag of India image

how many DCs you have?

and how many having same issue?

where you are seizing roles? seizing can be successful only if original role holder server is offline

If only particular DC have problem?

is this server also PDC master and what is your AD version?

If PDC is different server run dcdiag /v on that DC from elevated cmd and post results here
also run repadmin /replsum * /bysrc /bydest /sort:delta  and post results here

also run netdom query fsmo on all DCs and ensure output is same across all DCs

If problem exists with only specific DC, you can forcefully demote it and clean metadata clean-up, but before that you need to make sure that only it has problem, hence provide output of above cmdlets
I would recommend to increase diagnostic logging on the DC to see more of the errors being generated. This could be due to faulty hardware too.

https://support.microsoft.com/en-ca/help/2645996/active-directory-replication-error-8451-the-replication-operation-enco

Do you have any AV installed on the domain controllers? if so, can you confirm that the necessary exclusions have been configured for Active Directory related files and folders?
What OS are these DCs running?
Avatar of James White
James White

ASKER

We have 3 DC's. DC1, DC2, and DC3. Only DC1 is having this problem talking to the other DC's. I was trying to seize them through Active Directory Users and Computers. DC2 is the PDC.

Running your suggested command brought me this:
Source DSA          largest delta    fails/total %%   error
 DC3        28d.04h:00m:27s    1 /  10   10  (8451) The replication operation encountered a database error.
 DC2        28d.04h:00m:24s    1 /  10   10  (8451) The replication operation encountered a database error.
DC1                12m:48s    0 /  10    0

Destination DSA     largest delta    fails/total %%   error
 DC1        28d.04h:00m:27s    2 /  10   20  (8451) The replication operation encountered a database error.
 DC3                12m:48s    0 /  10    0
 DC2                   :34s    0 /  10    0

netdom query fsmo result is the same on all three. DC1 has all roles but PDC and DC2 has PDC
can you run dcdiag /v from elevated cmd on DC2 and post back results here for confirmation?

what is domain controller OS?

since DC is online, you must transfer fsmo, seize will notwork
Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   * Verifying that the local machine dc2, is a Directory Server.
   Home Server = dc2
   * Connecting to directory service on server dc2.
   * Identified AD Forest.
   Collecting AD specific global data
   * Collecting site info.
   Calling ldap_search_init_page(hld,CN=Sites,CN=Configuration,DC=domainname,DC=com,LDAP_SCOPE_SUBTREE,(objectCategory=ntDSSiteSettings),.......
   The previous call succeeded
   Iterating through the sites
   Looking at base site object: CN=NTDS Site Settings,CN=HostingProvider,CN=Sites,CN=Configuration,DC=domainname,DC=com
   Getting ISTG and options for the site
   Looking at base site object: CN=NTDS Site Settings,CN=MainOffice,CN=Sites,CN=Configuration,DC=domainname,DC=com
   Getting ISTG and options for the site
   * Identifying all servers.
   Calling ldap_search_init_page(hld,CN=Sites,CN=Configuration,DC=domainname,DC=com,LDAP_SCOPE_SUBTREE,(objectClass=ntDSDsa),.......
   The previous call succeeded....
   The previous call succeeded
   Iterating through the list of servers
   Getting information for the server CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=domainname,DC=com
   objectGuid obtained
   InvocationID obtained
   dnsHostname obtained
   site info obtained
   All the info for the server collected
   Getting information for the server CN=NTDS Settings,CN=DC3,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=domainname,DC=com
   objectGuid obtained
   InvocationID obtained
   dnsHostname obtained
   site info obtained
   All the info for the server collected
   Getting information for the server CN=NTDS Settings,CN=DC2,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=domainname,DC=com
   objectGuid obtained
   InvocationID obtained
   dnsHostname obtained
   site info obtained
   All the info for the server collected
   * Identifying all NC cross-refs.
   * Found 3 DC(s). Testing 1 of them.
   Done gathering initial info.

Doing initial required tests

   Testing server: MainOffice\DC2
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         Determining IP4 connectivity
         * Active Directory RPC Services Check
         ......................... DC2 passed test Connectivity

Doing primary tests

   Testing server: MainOffice\DC2
      Starting test: Advertising
         The DC DC2 is advertising itself as a DC and having a DS.
         The DC DC2 is advertising as an LDAP server
         The DC DC2 is advertising as having a writeable directory
         The DC DC2 is advertising as a Key Distribution Center
         Warning: DC2 is not advertising as a time server.
         The DS DC2 is advertising as a GC.
         ......................... DC2 failed test Advertising
      Test omitted by user request: CheckSecurityError
      Test omitted by user request: CutoffServers
      Starting test: FrsEvent
         * The File Replication Service Event log test
         There are warning or error events within the last 24 hours after the SYSVOL has been shared.  Failing SYSVOL replication problems may cause Group Policy problems
         A warning event occurred.  EventID: 0x800034C4
            Time Generated: 01/02/2019   08:54:32
            Event String:
            The File Replication Service is having trouble enabling replication from DC1 to DC2 for c:\windows\sysvol\domain using the DNS name DC1.domainname.com
             Following are some of the reasons you would see this warning.

             [1] FRS can not correctly resolve the DNS name DC1.domainname.com from this computer.
             [2] FRS is not running on DC1.domainname.com.
             [3] The topology information in the Active Directory Domain Services for this replica has not yet replicated to all the Domain Controllers.

             This event log message will appear once per connection, After the problem is fixed you will see another event log message indicating that the connection has
         ......................... DC2 passed test FrsEvent
      Starting test: DFSREvent
         The DFS Replication Event Log.
         Skip the test because the server is running FRS.
         ......................... DC2 passed test DFSREvent
      Starting test: SysVolCheck
         * The File Replication Service SYSVOL ready test
         File Replication Service's SYSVOL is ready
         ......................... DC2 passed test SysVolCheck
      Starting test: KccEvent
         * The KCC Event log test
         Found no KCC errors in "Directory Service" Event log in the last 15 minutes.
         .........................DC2 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         Role Schema Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=domainname,DC=com
         Role Domain Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=domainname,DC=com
         Role PDC Owner = CN=NTDS Settings,CN=DC2,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=domainname,DC=com
         Role Rid Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=domainname,DC=com
         Role Infrastructure Update Owner = CN=NTDS Settings,CN=DC1,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=domainname,DC=com
         ......................... DC2 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         Checking machine account for DC DC2 on DC DC2.
         * SPN found :LDAP/dc2.domainname.com/domainname.com
         * SPN found :LDAP/dc2.domainname.com
         * SPN found :LDAP/dc2
         * SPN found :LDAP/dc2.domainname.com/DOMAINNAME
         * SPN found :LDAP/7408dca1-adfb-452d-838a-c4444546464646._msdcs.domainname.com
         * SPN found :E3514235-4B06-11D1-AB04-00C535353532/7408dca1-adfb-452d-838a-b504c64255d5/domainname.com
         * SPN found :HOST/dc2.domainname.com/domainname.com
         * SPN found :HOST/dc2.domainname.com
         * SPN found :HOST/DC2
         * SPN found :HOST/dc2.domainname.com/DOMAINNAME
         * SPN found :GC/dc2.domainname.com/domainname.com
         ......................... DC2 passed test MachineAccount
      Starting test: NCSecDesc
         * Security Permissions check for all NC's on DC DC2.
         * Security Permissions Check for
           DC=ForestDnsZones,DC=domainname,DC=com
            (NDNC,Version 3)
         * Security Permissions Check for
           DC=DomainDnsZones,DC=domainname,DC=com
            (NDNC,Version 3)
         * Security Permissions Check for
           CN=Schema,CN=Configuration,DC=domainname,DC=com
            (Schema,Version 3)
         * Security Permissions Check for
           CN=Configuration,DC=domainname,DC=com
            (Configuration,Version 3)
         * Security Permissions Check for
           DC=domainname,DC=com
            (Domain,Version 3)
         .........................DC2 passed test NCSecDesc
      Starting test: NetLogons
         * Network Logons Privileges Check
         Verified share \\DC2\netlogon
         Verified share \\DC2\sysvol
         ......................... DC2 passed test NetLogons
      Starting test: ObjectsReplicated
         DC2 is in domain DC=domainname,DC=com
         Checking for CN=DC2,OU=Domain Controllers,DC=domainname,DC=com in domain DC=domainname,DC=com on 1 servers
            Object is up-to-date on all servers.
         Checking for CN=NTDS Settings,CN=DC2,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=domainname,DC=com in domain CN=Configuration,DC=domainname,DC
            Object is up-to-date on all servers.
         ......................... DC2 passed test ObjectsReplicated
      Test omitted by user request: OutboundSecureChannels
      Starting test: Replications
         * Replications Check
         * Replication Latency Check
            DC=ForestDnsZones,DC=domain,DC=com
               Latency information for 13 entries in the vector were ignored.
                  13 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency
            DC=DomainDnsZones,DC=domain,DC=com
               Latency information for 13 entries in the vector were ignored.
                  13 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency
            CN=Schema,CN=Configuration,DC=domainname,DC=com
               Latency information for 31 entries in the vector were ignored.
                  31 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency
            CN=Configuration,DC=domainname,DC=com
               Latency information for 31 entries in the vector were ignored.
                  31 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency
            DC=domainname,DC=com
               Latency information for 29 entries in the vector were ignored.
                  29 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency
         ......................... DC2 passed test Replications
      Starting test: RidManager
         * Available RID Pool for the Domain is 17603 to 1073741823
         * DC1.domainname.com is the RID Master
         * DsBind with RID Master was successful
         * rIDAllocationPool is 17103 to 17602
         * rIDPreviousAllocationPool is 17103 to 17602
         * rIDNextRID: 17121
         ......................... DC2 passed test RidManager
      Starting test: Services
         * Checking Service: EventSystem
         * Checking Service: RpcSs
         * Checking Service: NTDS
         * Checking Service: DnsCache
         * Checking Service: NtFrs
         * Checking Service: IsmServ
         * Checking Service: kdc
         * Checking Service: SamSs
         * Checking Service: LanmanServer
         * Checking Service: LanmanWorkstation
         * Checking Service: w32time
         * Checking Service: NETLOGON
         ......................... DC2 passed test Services
      Starting test: SystemLog
         * The System Event log test
         Found no errors in "System" Event log in the last 60 minutes.
         ......................... DC2 passed test SystemLog
      Test omitted by user request: Topology
      Test omitted by user request: VerifyEnterpriseReferences
      Starting test: VerifyReferences
         The system object reference (serverReference) CN=DC2,OU=Domain Controllers,DC=domainname,DC=com and backlink on CN=DC2,CN=Servers,CN=MainOffice,CN
         The system object reference (serverReferenceBL) CN=DC2,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=domainname,DC=com
         CN=NTDS Settings,CN=DC2,CN=Servers,CN=MainOffice,CN=Sites,CN=Configuration,DC=domainname,DC=com are correct.
         The system object reference (frsComputerReferenceBL) CN=DC2,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=domainname,DC
         CN=DC2,OU=Domain Controllers,DC=domainname,DC=com are correct.
         ......................... domainname passed test VerifyReferences
      Test omitted by user request: VerifyReplicas

      Test omitted by user request: DNS
      Test omitted by user request: DNS

   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidation

   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidation

   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

   Running partition tests on : domainname
      Starting test: CheckSDRefDom
         ......................... domainname passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... domainname passed test CrossRefValidation

   Running enterprise tests on : domainname.com
      Test omitted by user request: DNS
      Test omitted by user request: DNS
      Starting test: LocatorCheck
         GC Name: \\dc2.domainname.com
         Locator Flags: 0xe001f1bd
         PDC Name: \\dc2.domainname.com
         Locator Flags: 0xe001f1bd
         Time Server Name: \\DC3.domainname.com
         Locator Flags: 0xe001f1fc
         Preferred Time Server Name: \\DC3.domainname.com
         Locator Flags: 0xe001f1fc
         KDC Name: \\dc2.domainname.com
         Locator Flags: 0xe001f1bd
         ......................... domainname.com passed test LocatorCheck
      Starting test: Intersite
         Skipping site HostingProvider, this site is outside the scope provided by the command line arguments provided.
         Skipping site MainOffice, this site is outside the scope provided by the command line arguments provided.
         ......................... domainname.com passed test Intersite

DC1's OS is Windows Server 2016 - Core, DC2 and 3 both have Desktop Experience version of Server 2016. DC1 and 2 are VM's in a Hyper V while DC3 is a physical machine.
everything is fine on DC2 except it is not advertising as time server, since it is PDC, it must advertise as authoritative time server

You need to transfer rest of FSMO roles on DC2 and forcefully / gracefully demote DC1 since its not replicating since December start

After that clean metadata for removed DC from AD and promote server again as domain controller
What's the best way to do each one? Are there easy instructions somewhere I could follow? I just don't want to mess this up.
1st make sure you can locate netlogon and Sysvol shares on all rest of servers and create test GPO and make sure it is able to replicate to all DCs?
If anywhere in DHCP scopes it is defined, remove it, also remove it from static dns entries
After that turn off faulty DC permanently and make sure you are able to logon in domain on workstations and servers

Now:
seize all FSMO roles one by one on DC2

check "netdom query fsmo" output , it should be identical on all servers

Then do metadata clean-up for failed DC from DC2
https://servergurunow.wordpress.com/2017/08/08/metadata-cleanup-of-a-domain-controller-2/

After that bring new server in domain and promote it to domain controller

If you still not clear or not sure, then hire some professional for that.
The SYSVOL folder has been located in each DC: \\DC\c$\Windows\SYSVOL\sysvol. The NETLOGON folder is in \\DC\c$\WINDOWS\SYSVOL\sysvol\domainname.com\SCRIPTS

I'm not understanding what you're asking for in regards to DHCP. Are you saying you want me to remove any scopes that we have? Is this on all 3 DC's or just the faulty DC1? DC1 doesn't have DHCP running. Only DC2 and DC3 do.

You said you want me to turn off DC1 permanently? I read later that you wanted me to turn it back on and re-promote it again unless you meant a fresh OS install?
You are taking it wrongly

1st you need to understand that you cannot seize fsmo roles unless original dc holding them gone offline permanently
I didn't say to turn on dc back after you turned it off
If you need turned off dc back, but obviously you need to fresh install
I am in impression that you know fsmo seizing concept.. Anyway

In terms of dhcp, what i am saying is that if dc being turned off if there in dhcp scope as dns server, remove it
Also remove any static dns entries you may have on member servers pointing to dc being turned off

I hope its clear now
Ok, I can remove the entry for it as a DNS Server in DHCP along with removing it from DNS static entries. I can do that. So I need to do that after turning off DC1 then I go ahead with the role seizing and metadata cleanup? What you're saying is if I want DC1 back, I should just create a new VM for it with a fresh install and then promote it.
Dhcp change and static dns changes should be done in advance
Later on turn off dc and proceed
One quick question, should I demote DC1 before turning it off or does that not matter?
ASKER CERTIFIED SOLUTION
Avatar of Mahesh
Mahesh
Flag of India image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Hey there, your instructions worked last week. After the worked, I rebuilt the server and then installed the DNS Server and AD DS services and was able to re-promote this and assign the OM roles again. The replication returned to normal. Thanks!