How to sync DC machine password after reset it?

We have an issue here, let me explian all happen
We have 18 branches, 21 domain controllers, the primary DC system  board crashed,  HP dispatched a tech to replace the board, when the server come back online fine, other DCs start to receive Kerberos error ID 4, which indecated the PDC machine password userd to encrpypt the Kerberos service ticket is different than that on the target server. The result of this is branch DC is not sync with this PDC, the replication topology is set up all branch DC sync with this PDC and vice vise.  And now helpdesk start to receiving calls about user changed password at branch , but not able to access portal and Citrix farm at datacenter, unless they use old password.

What I have been done
1) use netdom resetpwd to reset this PDC password against another DC See MS KB
2) these two DC can talk each other , but the reset password fail to sync to all other DCs, since other DC only talks to the PDC which is the one changed machine password.
3) try to create a new replication set between the one can talk to PDC with other branch DCs,  but it not replicate ( error " the naming context is in the process of being removed or is not replicated from the specified server)
4) ReplMon shows all replication are failing , except these two DC ( the PDC and reset password on other DC)

I am stuck here right now.
I have
 good backup of PDC before it went down.

Any advice will be very appreciated.
Who is Participating?
The only caution deals with not stopping KDC on GC server/s
Do you have branches with multiple DC?
The other issue is that the branches are already in a strange state. Try stopping the KDC on two currently not replicating DC and perform the instructions included in the article you posted and see whether that brings both in without adversely affecting the four that already sync at the same time.  If it does, you can try the same with five or six. And then the remainder. Double check your replication topology setup once AD replication is restored.
Did you follow the directions included in the document you found and referenced with your question?
I.e. did you stop the kerberos on all DC's experiencing issues step 2 Note? Then step 3 deleted cached kerberos credentials?
Once you restore replication, you might want to consider the AD DC replication scheme from hub and spoke to all DCs can replicate with one another or if you have a specific VPN setup between location to setup replication between those that are already connected by a VPN.
Kerberos event 4 is not likely caused by the password of the PDC. Instead, there are two reasons that this may be logged:
a duplicated SPN that is logged in the event;
a duplicated IP address(A record) in DNS. i.e. in DNS two computers are having same IP address regisgtered.

So, use the command below to check duplicated SPN:
ldifde -f spn.txt -s ServerName -d "dc=domain,dc=com" -r "(objectclass=computer)" -l serviceprincipalname

Also check DNS to see if another machine is having the same IP with the PDC.

When user changes the password, it is synced firstly with the PDC. When the PDC is not reachable, the change failed and that is why the user can still use the old password. Again, this can be caused by duplicated IP address in DNS.

If the issue persists and you have a good system state backup, try to backup the PDC and check again.

Hope this helps!
Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

prcdiscoverAuthor Commented:
Arnold and Hshao
Thank you,

Arnold, with MS support on line last night, we have made four out of 18 branches DC replicate now with the primary DC, we didnt try to stop all KDC services, ( we should),
What we did
While accessing comgt1 shares, we got target account name is incorrect error
Downloaded and installed kerbtray
Tested kerbtray and it failed to reset the secure channel
Downloaded and installed klist
We stopped KDC service, and disabled it
Ran klist purge to purge the user tickets
Ran the following 2 commands :
netdom resetpwd /server:DC2 /userd:domain\administrator /passwordd:*

net use \\DC2.domain.local\IPC$
Forced the DC to replicate from DC2 by running repadmin /syncall /d /e
Restarted KDC and set it to automatic
Able to access \\PDC
Verified replication monitor

the questions come from me
1) Does one server( in this case a DC) only have one machine password through entire forest? I believe the answer is true,
2) if 1) is yes, then everytime I ran netdom from the PDC against a DC, the machine password for the PDC will changed, and try to sync to all other DC
3) according to the MS KB, KDC need to stop on all DCs, will this affect user login, anyother impacts?
Funny is the MS support doesnt know this KB, I am just showing her ( from India) the KB about all KDC service on all DCs have to stop.
Will keep this updated,
prcdiscoverAuthor Commented:
Forgot to add one more thing,
This morning , all branch user can not access exchange through Outlook, they still can access through webmail.
Portal and other shared folder from HQ site are still accessible.
Dont understand why cant exchange through outlook at the mean time webmail is fine, this happen on all branches, include the four branchs' DC can replicate with PDC
What is the error that the outlook users get?
If you manage outlook settings via GPO, use GPMC to get a Policy Result set to see what is happening i.e. a GPO policy dealing with outlook settings is not applying.
prcdiscoverAuthor Commented:
All the users except from HQ are prompted for the user name and password, when the user type the correct password, it still prompted, and it prompted to different server ( switch between the exchange backend server and primary DC) , the webmail, which is point to the exchange front server is working fine. ( I am worry about this may stop working anytime)
It looks like the exchange backend server kerberbos machine ticket timeout,
Not sure,
Unfortunately, in your situation there could be two possible "correct passwords."
The new and the old.  Try both. Does the user's account get locked.
Which DC  does the exchange Front Server is use to authenticate users?

The issue could have been the result of the replication model you have just for a case where the HUB goes down, each child has their own recent changes they want to push.
The other possibility is that you have a machine password change requirement that kicked in within the same time frame the server malfunctioned. All speculation what led to this.  I think the kerberos is only used to setup the secure channel from DC to DC to exchange data.
At this point you have four or five out of the 21 that resumed replication.
Was the suggestion to look for a duplicate IP helpful in solving this issue?
You might as well go one at a time to the DCs that do not sync and clear their machine ticket and reboot based on the article you found or using the process you did with MS support if it is different.

prcdiscoverAuthor Commented:
I have talked to two users, they havent change password for more than a month.
both frontend and backend server are authenticated by the secondary DC on HQ site, when I type set l, it shows the secondary DC, but how about user, when user authenticated by exchange, how the aithentication topology works, I did mention, when user keep type the user name and password, it actaully prompted different servers, one to backend server, then PDC, then backend server again, then PDC.  so not sure how to change the authentication server when user login through Outlook.

There is no duplicate IP address on the network. I have double checked.

I may want to do stop KDC on all DCs at the same time, then reset it, but I not sure the possible impact,
prcdiscoverAuthor Commented:
This issue has been fixed.
Basically, the KB article need to follow

1)      stop kdc on all DCs have issue
2)      start kerbtray, purge ticket one by one,
3)      reset password  netdom resetpwd /server:localserver /userd: yourdoamin\\administrator /passwordd:*
4)      start \\pdc wait couple of minutes
5)      start kdc  on all DCs wait couple of minutes
6)       run  repadmin /syncall /d /e from PDC

Some of the site still failed with same error “ The target principal name is incorrect”

Keep  re run this on the failed DC, until it fixed.

Thank you all for the help.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.