Replication issues - DFSR, DNS Active Directory

Drew Ferguson
Drew Ferguson used Ask the Experts™
on
Hi Team
I'm having issues with DC replication, DNS issues etc
PCs are unable to join the domain where DC05 is.  
Please help.

Thanks
dc05.PNG
dc05-a.PNG
dc05-b.PNG
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
AlexSenior Infrastructure Analyst

Commented:
Ok so

Firstly run some DCDIAG's on the domain, see what's actually failing, it's likely to be some sort of DNS setting. Run repadmin too to get a report of your replication status.

Prior to anything else though, have you made any sort of changes? Is this a new DC? Have you demoted another DC out of the estate? etc etc etc

Thanks
Alex

Commented:
first thing check the windows firewall on both sides of machines.. disable it and check ps

Author

Commented:
Hi Alex
Here you go

ll failed). Look in the details tab for error code and description.
         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:04:54
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc05$. The target name used was LDAP/FHEDC05.FloridaHouse.local/FLORIDAH
OUSE. This indicates that the target server failed to decrypt the ticket provide
d by the client. This can occur when the target server principal name (SPN) is r
egistered on an account other than the account the target service is using. Ensu
re that the target SPN is only registered on the account used by the server. Thi
s error can also happen if the target service account password is different than
 what is configured on the Kerberos Key Distribution Center for that target serv
ice. Ensure that the service on the server and the KDC are both configured to us
e the same password. If the server name is not fully qualified, and the target d
omain (FLORIDAHOUSE.LOCAL) is different from the client domain (FLORIDAHOUSE.LOC
AL), check if there are identically named server accounts in these two domains,
or use the fully-qualified name to identify the server.
         An error event occurred.  EventID: 0x000003EE
            Time Generated: 10/22/2019   09:09:30
            Event String:
            The processing of Group Policy failed. Windows could not authenticat
e to the Active Directory service on a domain controller. (LDAP Bind function ca
ll failed). Look in the details tab for error code and description.
         An error event occurred.  EventID: 0x000003EE
            Time Generated: 10/22/2019   09:14:30
            Event String:
            The processing of Group Policy failed. Windows could not authenticat
e to the Active Directory service on a domain controller. (LDAP Bind function ca
ll failed). Look in the details tab for error code and description.
         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:16:33
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc10$. The target name used was E3514235-4B06-11D1-AB04-00C04FC2DCD2/77d
485cc-5ae9-4a8b-ae8c-bd99032cadb0/FloridaHouse.local@FloridaHouse.local. This in
dicates that the target server failed to decrypt the ticket provided by the clie
nt. This can occur when the target server principal name (SPN) is registered on
an account other than the account the target service is using. Ensure that the t
arget SPN is only registered on the account used by the server. This error can a
lso happen if the target service account password is different than what is conf
igured on the Kerberos Key Distribution Center for that target service. Ensure t
hat the service on the server and the KDC are both configured to use the same pa
ssword. If the server name is not fully qualified, and the target domain (FLORID
AHOUSE.LOCAL) is different from the client domain (FLORIDAHOUSE.LOCAL), check if
 there are identically named server accounts in these two domains, or use the fu
lly-qualified name to identify the server.
         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:17:37
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc10$. The target name used was ldap/FHEDC10.FloridaHouse.local. This in
dicates that the target server failed to decrypt the ticket provided by the clie
nt. This can occur when the target server principal name (SPN) is registered on
an account other than the account the target service is using. Ensure that the t
arget SPN is only registered on the account used by the server. This error can a
lso happen if the target service account password is different than what is conf
igured on the Kerberos Key Distribution Center for that target service. Ensure t
hat the service on the server and the KDC are both configured to use the same pa
ssword. If the server name is not fully qualified, and the target domain (FLORID
AHOUSE.LOCAL) is different from the client domain (FLORIDAHOUSE.LOCAL), check if
 there are identically named server accounts in these two domains, or use the fu
lly-qualified name to identify the server.
         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:19:30
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc05$. The target name used was LDAP/FHEDC05.FloridaHouse.local/FloridaH
ouse.local@FLORIDAHOUSE.LOCAL. This indicates that the target server failed to d
ecrypt the ticket provided by the client. This can occur when the target server
principal name (SPN) is registered on an account other than the account the targ
et service is using. Ensure that the target SPN is only registered on the accoun
t used by the server. This error can also happen if the target service account p
assword is different than what is configured on the Kerberos Key Distribution Ce
nter for that target service. Ensure that the service on the server and the KDC
are both configured to use the same password. If the server name is not fully qu
alified, and the target domain (FLORIDAHOUSE.LOCAL) is different from the client
 domain (FLORIDAHOUSE.LOCAL), check if there are identically named server accoun
ts in these two domains, or use the fully-qualified name to identify the server.

         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:19:30
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc05$. The target name used was ldap/FHEDC05.FloridaHouse.local/FloridaH
ouse.local@FLORIDAHOUSE.LOCAL. This indicates that the target server failed to d
ecrypt the ticket provided by the client. This can occur when the target server
principal name (SPN) is registered on an account other than the account the targ
et service is using. Ensure that the target SPN is only registered on the accoun
t used by the server. This error can also happen if the target service account p
assword is different than what is configured on the Kerberos Key Distribution Ce
nter for that target service. Ensure that the service on the server and the KDC
are both configured to use the same password. If the server name is not fully qu
alified, and the target domain (FLORIDAHOUSE.LOCAL) is different from the client
 domain (FLORIDAHOUSE.LOCAL), check if there are identically named server accoun
ts in these two domains, or use the fully-qualified name to identify the server.

         An error event occurred.  EventID: 0x000003EE
            Time Generated: 10/22/2019   09:19:30
            Event String:
            The processing of Group Policy failed. Windows could not authenticat
e to the Active Directory service on a domain controller. (LDAP Bind function ca
ll failed). Look in the details tab for error code and description.
         An error event occurred.  EventID: 0x00009017
            Time Generated: 10/22/2019   09:19:55
            Event String:
            A fatal alert was received from the remote endpoint. The TLS protoco
l defined fatal alert code is 70.
         An error event occurred.  EventID: 0x00009017
            Time Generated: 10/22/2019   09:19:55
            Event String:
            A fatal alert was received from the remote endpoint. The TLS protoco
l defined fatal alert code is 70.
         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:22:53
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc04$. The target name used was E3514235-4B06-11D1-AB04-00C04FC2DCD2/9c9
0420b-0a85-4ca5-9af4-76a03f7b02ee/FloridaHouse.local@FloridaHouse.local. This in
dicates that the target server failed to decrypt the ticket provided by the clie
nt. This can occur when the target server principal name (SPN) is registered on
an account other than the account the target service is using. Ensure that the t
arget SPN is only registered on the account used by the server. This error can a
lso happen if the target service account password is different than what is conf
igured on the Kerberos Key Distribution Center for that target service. Ensure t
hat the service on the server and the KDC are both configured to use the same pa
ssword. If the server name is not fully qualified, and the target domain (FLORID
AHOUSE.LOCAL) is different from the client domain (FLORIDAHOUSE.LOCAL), check if
 there are identically named server accounts in these two domains, or use the fu
lly-qualified name to identify the server.
         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:22:53
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc03$. The target name used was E3514235-4B06-11D1-AB04-00C04FC2DCD2/f69
39a23-bd12-465e-b915-065adfbb83db/FloridaHouse.local@FloridaHouse.local. This in
dicates that the target server failed to decrypt the ticket provided by the clie
nt. This can occur when the target server principal name (SPN) is registered on
an account other than the account the target service is using. Ensure that the t
arget SPN is only registered on the account used by the server. This error can a
lso happen if the target service account password is different than what is conf
igured on the Kerberos Key Distribution Center for that target service. Ensure t
hat the service on the server and the KDC are both configured to use the same pa
ssword. If the server name is not fully qualified, and the target domain (FLORID
AHOUSE.LOCAL) is different from the client domain (FLORIDAHOUSE.LOCAL), check if
 there are identically named server accounts in these two domains, or use the fu
lly-qualified name to identify the server.
         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:22:53
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc01$. The target name used was E3514235-4B06-11D1-AB04-00C04FC2DCD2/cac
35597-404a-4539-9b23-88fc4acb455e/FloridaHouse.local@FloridaHouse.local. This in
dicates that the target server failed to decrypt the ticket provided by the clie
nt. This can occur when the target server principal name (SPN) is registered on
an account other than the account the target service is using. Ensure that the t
arget SPN is only registered on the account used by the server. This error can a
lso happen if the target service account password is different than what is conf
igured on the Kerberos Key Distribution Center for that target service. Ensure t
hat the service on the server and the KDC are both configured to use the same pa
ssword. If the server name is not fully qualified, and the target domain (FLORID
AHOUSE.LOCAL) is different from the client domain (FLORIDAHOUSE.LOCAL), check if
 there are identically named server accounts in these two domains, or use the fu
lly-qualified name to identify the server.
         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:22:54
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc02$. The target name used was E3514235-4B06-11D1-AB04-00C04FC2DCD2/cfc
97ba2-ca8c-438a-b06d-69b6719b2dc3/FloridaHouse.local@FloridaHouse.local. This in
dicates that the target server failed to decrypt the ticket provided by the clie
nt. This can occur when the target server principal name (SPN) is registered on
an account other than the account the target service is using. Ensure that the t
arget SPN is only registered on the account used by the server. This error can a
lso happen if the target service account password is different than what is conf
igured on the Kerberos Key Distribution Center for that target service. Ensure t
hat the service on the server and the KDC are both configured to use the same pa
ssword. If the server name is not fully qualified, and the target domain (FLORID
AHOUSE.LOCAL) is different from the client domain (FLORIDAHOUSE.LOCAL), check if
 there are identically named server accounts in these two domains, or use the fu
lly-qualified name to identify the server.
         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:22:55
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc05$. The target name used was LDAP/FHEDC05. This indicates that the ta
rget server failed to decrypt the ticket provided by the client. This can occur
when the target server principal name (SPN) is registered on an account other th
an the account the target service is using. Ensure that the target SPN is only r
egistered on the account used by the server. This error can also happen if the t
arget service account password is different than what is configured on the Kerbe
ros Key Distribution Center for that target service. Ensure that the service on
the server and the KDC are both configured to use the same password. If the serv
er name is not fully qualified, and the target domain (FLORIDAHOUSE.LOCAL) is di
fferent from the client domain (FLORIDAHOUSE.LOCAL), check if there are identica
lly named server accounts in these two domains, or use the fully-qualified name
to identify the server.
         An error event occurred.  EventID: 0x000003EE
            Time Generated: 10/22/2019   09:24:30
            Event String:
            The processing of Group Policy failed. Windows could not authenticat
e to the Active Directory service on a domain controller. (LDAP Bind function ca
ll failed). Look in the details tab for error code and description.
         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:26:09
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc05$. The target name used was cifs/FHEDC05.FloridaHouse.local. This in
dicates that the target server failed to decrypt the ticket provided by the clie
nt. This can occur when the target server principal name (SPN) is registered on
an account other than the account the target service is using. Ensure that the t
arget SPN is only registered on the account used by the server. This error can a
lso happen if the target service account password is different than what is conf
igured on the Kerberos Key Distribution Center for that target service. Ensure t
hat the service on the server and the KDC are both configured to use the same pa
ssword. If the server name is not fully qualified, and the target domain (FLORID
AHOUSE.LOCAL) is different from the client domain (FLORIDAHOUSE.LOCAL), check if
 there are identically named server accounts in these two domains, or use the fu
lly-qualified name to identify the server.
         An error event occurred.  EventID: 0x000003EE
            Time Generated: 10/22/2019   09:29:31
            Event String:
            The processing of Group Policy failed. Windows could not authenticat
e to the Active Directory service on a domain controller. (LDAP Bind function ca
ll failed). Look in the details tab for error code and description.
         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:33:21
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc10$. The target name used was LDAP/77d485cc-5ae9-4a8b-ae8c-bd99032cadb
0._msdcs.FloridaHouse.local. This indicates that the target server failed to dec
rypt the ticket provided by the client. This can occur when the target server pr
incipal name (SPN) is registered on an account other than the account the target
 service is using. Ensure that the target SPN is only registered on the account
used by the server. This error can also happen if the target service account pas
sword is different than what is configured on the Kerberos Key Distribution Cent
er for that target service. Ensure that the service on the server and the KDC ar
e both configured to use the same password. If the server name is not fully qual
ified, and the target domain (FLORIDAHOUSE.LOCAL) is different from the client d
omain (FLORIDAHOUSE.LOCAL), check if there are identically named server accounts
 in these two domains, or use the fully-qualified name to identify the server.
         An error event occurred.  EventID: 0x40000004
            Time Generated: 10/22/2019   09:33:21
            Event String:
            The Kerberos client received a KRB_AP_ERR_MODIFIED error from the se
rver fhedc10$. The target name used was ldap/fhedc10.FloridaHouse.local. This in
dicates that the target server failed to decrypt the ticket provided by the clie
nt. This can occur when the target server principal name (SPN) is registered on
an account other than the account the target service is using. Ensure that the t
arget SPN is only registered on the account used by the server. This error can a
lso happen if the target service account password is different than what is conf
igured on the Kerberos Key Distribution Center for that target service. Ensure t
hat the service on the server and the KDC are both configured to use the same pa
ssword. If the server name is not fully qualified, and the target domain (FLORID
AHOUSE.LOCAL) is different from the client domain (FLORIDAHOUSE.LOCAL), check if
 there are identically named server accounts in these two domains, or use the fu
lly-qualified name to identify the server.
         ......................... FHEDC05 failed test SystemLog
      Starting test: VerifyReferences
         ......................... FHEDC05 passed test VerifyReferences


   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test
         CrossRefValidation

   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test
         CrossRefValidation

   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

   Running partition tests on : FloridaHouse
      Starting test: CheckSDRefDom
         ......................... FloridaHouse passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... FloridaHouse passed test CrossRefValidation

   Running enterprise tests on : FloridaHouse.local
      Starting test: LocatorCheck
         ......................... FloridaHouse.local passed test LocatorCheck
      Starting test: Intersite
         ......................... FloridaHouse.local passed test Intersite

C:\Users\Administrator.FLORIDAHOUSE>

THis DC was offline for a while and we just realized it
11/26 Forrester Webinar: Savings for Enterprise

How can your organization benefit from savings just by replacing your legacy backup solutions with Acronis' #CyberProtection? Join Forrester's Joe Branca and Ryan Davis from Acronis live as they explain how you can too.

Author

Commented:
Sajid

Windows firewall disabled
AlexSenior Infrastructure Analyst

Commented:
The krbtgt account password is out of sync... Disable and stop the kdc service on all DCs but the PDC emulator. Then reboot them one at a time. This will force all to obtain their TGT from the PDC and will enable them to start replication. After that, enable and start the kdc service.

Author

Commented:
Alex

Should I follow your recommendation or follow the link you gave me
AlexSenior Infrastructure Analyst

Commented:
I'd start off with the way I do it and then progress into the more indepth way Microsoft do. Either would do the job but it's essentially the same thing. The reboot normally does the trick

Regards,
Alex

Author

Commented:
OK
I will do your recommendation and post the dcdiag after

Author

Commented:
Is KDC the Kerberos Key Distribution Center service

Author

Commented:
Should I do the servers one at a time or all at once.
SteveArchitect/Designer

Commented:
Have you renamed/decommissioned/moved any of your DCs recently?
Notice you mention a DC was offline for a while, which may be relevant. how long was it off for, and is it the one that is no longer ale to replicate?

There is a limited lifespan for an offline DC. if its more than the Tombstone period the DC is dead and cannot be recovered (basically decommission and create a new DC)
AlexSenior Infrastructure Analyst

Commented:
One at a time, not all at once, otherwise the TGT won't replicate correctly.

Author

Commented:
Steve

DC05 was tombstoned

Author

Commented:
Alex
I did it and ran dcdiag and seems like I have the same issues
AlexSenior Infrastructure Analyst

Commented:
https://community.spiceworks.com/topic/2146253-windows-server-how-to-fix-a-tombstoned-domain-controller

You'll need to clear your Lingering objects and then force replication.

Realistically i'd probably demote the server out and re-promo it, but that on it's own could cause issues as well.

Author

Commented:
ok I'm going to demote DC05

Author

Commented:
Alex/Steve
I"m going to demote DC05

Author

Commented:
Guys,

I"m getting this error
dc05-demotion-error.PNG
AlexSenior Infrastructure Analyst

Commented:
You're going to have to use NTDSUTIL to remove it from Active Directory.

https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/ad-ds-metadata-cleanup

best Practice here.
SteveArchitect/Designer

Commented:
DC05 was tombstoned
And is this the server you are saying is having an issue?

if yes it is now dead and cannot be brought back to life. Decomission (manually) and create a new DC.
its a bit messy but following this works:
https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/ad-ds-metadata-cleanup

Note: make sure you have a full backup of your AD before doing this. its risky to mess with ADSIedit.
AlexSenior Infrastructure Analyst

Commented:
also, you'll need to connect to a live domain controller, I.e Not dc05

Author

Commented:
Thanks guys

Can I get step by step instructions on how to remove DC05 by force.
Senior Infrastructure Analyst
Commented:
To clean up server metadata by using Ntdsutil
Open a command prompt as an administrator: On the Start menu, right-click Command Prompt, and then click Run as administrator. If the User Account Control dialog box appears, provide credentials of an Enterprise Administrator if required, and then click Continue.

At the command prompt, type the following command, and then press ENTER:

ntdsutil

At the ntdsutil: prompt, type the following command, and then press ENTER:

metadata cleanup

At the metadata cleanup: prompt, type the following command, and then press ENTER:

remove selected server <ServerName>

In Server Remove Configuration Dialog, review the information and warning, and then click Yes to remove the server object and metadata.

At this point, Ntdsutil confirms that the domain controller was removed successfully. If you receive an error message that indicates that the object cannot be found, the domain controller might have been removed earlier.

At the metadata cleanup: and ntdsutil: prompts, type quit, and then press ENTER.

To confirm removal of the domain controller:

Open Active Directory Users and Computers. In the domain of the removed domain controller, click Domain Controllers. In the details pane, an object for the domain controller that you removed should not appear.

Open Active Directory Sites and Services. Navigate to the Servers container and confirm that the server object for the domain controller that you removed does not contain an NTDS Settings object. If no child objects appear below the server object, you can delete the server object. If a child object appears, do not delete the server object because another application is using the object.

Author

Commented:
awesome
thanks

Author

Commented:
Should I run this on DC05 (the bad server) or the server that holds the FSMO roles?
AlexSenior Infrastructure Analyst

Commented:
no, the server that holds the FSMO roles, it needs to communicate with the rest of the domain doesn't it, as such since DC5 can't speak with the domain doing it there is pointless. :-)

Author

Commented:
It does, I'll do it on the primary DC
AlexSenior Infrastructure Analyst

Commented:
Excellent, that'll then replicate out to your other domain controllers and remove 05 from your domain, at which point, you demolish 05 and rebuild it from scratch, DO NOT ADD IT BACK!

Author

Commented:
Can I just change the hostname from DC05 to DC06
SteveArchitect/Designer

Commented:
no. dc05 thinks it is still a dc but isnt.
a wipe is the only safe way to reuse it

Author

Commented:
ok
DC05 is still in the domain controller container in AD
Should I just delete it
SteveArchitect/Designer

Commented:
no. follow the steps provided in links and posts above.

Author

Commented:
I did

Author

Commented:
What should I post
DrDave242Principal Support Engineer

Commented:
You don't have to reinstall the OS on DC05; just force-demote it so that it knows it's no longer a DC. (I typically do that before performing a metadata cleanup, but it'll work either way.) Check the "Force the removal of this domain controller" box during demotion to force-demote it. This will remove AD from the DC without attempting to contact any other DC in the domain.

Once the DC has been force-demoted and its metadata has been removed from AD, you're free to re-promote it or do whatever else you want with it.
SteveArchitect/Designer

Commented:
true, you dont *have* to wipe and create a new server but i recommend it if you intend to re use it as a dc. your AD is critical to yoir systems and it should be protected from unnecessary risk where possible (in my view).
AlexSenior Infrastructure Analyst

Commented:
morning,

So where are you at now? I'm back online so I can help again.

Regards
Alex

Author

Commented:
Alex/Steve/DrDave/ and all
So I decommissioned DC05, changed the hostname of the server to DC06, installed AD and DNS and configured it.
Everything seems to be working now but I'll give it another day to two before closing this question.
AlexSenior Infrastructure Analyst

Commented:
run repadmin to ensure replication is working correctly.

Run DCDIAG to confirm AD is happy.

Pleased the instructions helped :-)

Author

Commented:
i will
SteveArchitect/Designer

Commented:
great news. Glad its looking good :-)

Author

Commented:
Ok Guys
Look like the issue is finally resolved.
I have no issues anymore.  
Thanks you so much

Author

Commented:
Steve and Alex much appreciation
AlexSenior Infrastructure Analyst

Commented:
Excellent,

please mark the posts as answers which helped you get to your end goal for other people whom use the forum

Regards
alex

Author

Commented:
Thanks all

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial