Why do all accounts get locked out after resetting KRBTGT account?

About a month ago I reset our KRBTGT account using this script from Technet:

https://gallery.technet.microsoft.com/Reset-the-krbtgt-account-581a9e51

Everything verified and succeeded. Every since then like clockwork every Saturday almost our entire AD of User Accounts gets locked out. Computer Accounts are unaffected. I have checked and found a ton of event ID 4771's with the Failure Code 0x18 and 0x12. This is reflected in their meanings documented below:

https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=4771

They claim that pre-auth is failing and then causing the lockout. I've checked and a lot of our Kerberos tickets do expire on Saturday. In addition to the 4771's I also see a lot of 4768 events.

https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=4768

I've tried rebooting all clients, though not servers, to flush the existing tickets but that hasn't helped. Neither has bouncing the DC's or Restarting the Ticket Granting service. I've even played with some powershell scripts to use klist to flush the tickets for all users. Though I haven't gotten them to work remotely.

Still this has been going on for over a month and I can't find anyone else having a similar issue. Resetting the KRBTGT account should NOT cause this to happen from everything I've read. We are at 2012 R2 Functional for our Forest and Domain level. When going up a level in the past we never had a similar issue. It was only when I ran the above script that all heck broke loose.

I'm happy to provide any additional information requested.

Thanks
CCB-TechAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Jose Gabriel Ortega CastroEE Solution Guide - CEO Faru Bonon ITCommented:
The  KRBTGT account is a system account that shouldn't be using for anything.
Here's what is the KRBTGT https://blogs.technet.microsoft.com/janelewis/2006/12/19/the-krbtgt-account-what-is-it/
And At least you are in a disaster recovery scenario you should not be messing with that account.
0
CCB-TechAuthor Commented:
I agree that you should almost never need to reset that account. However, there is a danger of an attacker achieving persistence that can only be negated by changing the password on the KRBTGT account periodically. See here for the dangers of Golden Tickets:

 https://blog.stealthbits.com/complete-domain-compromise-with-golden-tickets/

As of higher functional levels of Active Directory Microsoft recommended changing the password on a regular schedule:

https://technet.microsoft.com/en-us/library/dn745899.aspx?f=255&MSPPError=-2147217396#Anchor_5

So the article you linked is correct, but is not reflective of current best practices. Also, your answer doesn't address the fact that I have already changed the password and it shouldn't have caused the fallout that it has.
0
Jose Gabriel Ortega CastroEE Solution Guide - CEO Faru Bonon ITCommented:
IT's hard trying to "guess" what is going on if you don't provide information about your event viewer or anything related to addressing the issue.
You should have events related to whats going on in your event viewer or in the directory events.
0
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

CCB-TechAuthor Commented:
I supplied the only Event ID evidence I have available in the form of 4771 and 4768 and the Failure Codes they Contain. I'm not asking for a "guess" as all evidence has been provided. Here is a TLDR version as it appears you didn't read the entirety of the question:

  1. Change: KRBTGT Account Reset Using Microsoft Provided Powershell Over A Month Ago
  2. Symptom: Every Saturday almost all AD Accounts are locked as if a bad password was provided. Unlocking them fixes the problem until the next Saturday.
  3. Evidence: Event ID's 4771 and 4768 as listed above reference a pre-auth failure of the Kerberos ticket. This failure leads to the lockouts. No other events tied to this show in any DC as far as I can tell.

Sorry for appearing snarky but I provided what you are talking about from the get-go and I don't understand why you are asking for that information as if I didn't.
0
masnrockCommented:
Have you tried looking into duplicate SPNs, and going down that path as a fix to your issues? Based on what you haven't presenting, are we to assume that it's been random accounts locking out, or just a certain set of them?
0
CCB-TechAuthor Commented:
I have not checked into that yet. Good idea. It appears to be random accounts locking out.

What is really peculiar is that some accounts that should no longer be active are even being locked out. Which re-enforces the random part.
0
CCB-TechAuthor Commented:
Okay good idea but the only duplicate SPNs I identified were tied to a decommissioned server from years ago. So back to square one :(.
0
arnoldCommented:
This account is reset every 30 days if not mistaken through GPO.

in short, you have to rejoin/reset the key set on each system. The key they have through the means you went suggests when the systems came back with the key they have, which is no longer
When used through GPO/Automated process, the old key and the new key are maintained and the system transition from the old key to the new...
the same way if you set a GPO to have each system, (workstation and/or workstation) reset their Machine password...

what is the remedy you are using when all systems are locked out?
0
CCB-TechAuthor Commented:
I've not seen anything hinting at the KRBTGT account being reset every 30 days. I thought that was only for Computer Accounts?

We have actually not had any impact on Computer Accounts, only User Accounts. Something I should have been clearer on in my first post. *Editing now :)* Users are getting locked out same as if they all tried bad passwords. When in reality something is amiss with the Kerberos Ticket as far as I can tell causing a similar lockout.

We have a third party application that makes unlocking all users relatively simple happily.
0
arnoldCommented:
You have to scour the security event logs on the DC or use the aclockout tool with the eventmgmt from MS to pull the data to identify the DC that locks the user account and see whether the same DC consistently locks all users' accounts.

Potentially an unsync DC in the environment gets auth requests on access to resources such as a network resource, file share etc. and it is the only one that is hit by the fileserver and that is how the user gets locked out while on login and other access the user accounts are not affected.

use dcdiag to confirm that all the DCs in the environment are synchronized. potentially one might be grossly out of sync.
0
CCB-TechAuthor Commented:
I haven't yet used the Accout Lockout tool on this problem so I'll give it a shot.

I was concerned about a sync problem as well but once a week I get a health report on my AD from this Powershell script:

https://gallery.technet.microsoft.com/scriptcenter/Active-Directory-Health-709336cd

I just re-ran it and got a clean bill of health on all measured points. Even our RODC looks fine. That was one of the first things I checked. I know an unhealthy AD can cause all kinds of oddities. I'll amend this information to my first Post as well.

I'll look through the logs with the Account Lockout tool. Maybe that'll shed some new light.
0
arnoldCommented:
The account lockout tool will identify the DC that locked the account, there tool includes a event log collection tool that you would need to specify the login/logout events that you want it to extract.

Identifying the source of the lockout i.e. all users who get locked out are attempting to access a specific resource. Point being identifyinig what might be causing this issue.
Have users altered their password since the KRBTG update, i.e. some have saved credentials on share access, devices etc. that have not been updated and these are the items that lock the accounts after X attempts to login.....access .
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
CCB-TechAuthor Commented:
I'm checking into all those details and continuing the chase down the Event Log rabbit hole. I'll post more as I gather information. Thanks for the help so far!
0
CCB-TechAuthor Commented:
@Arnold thanks for the idea! I dug around in the Event Logs this time focusing on the Lockouts themselves vs the Kerberos Logs. This led me to find a Security Scanner that was doing a scan at the exact time the lockouts were happening. Now, mind you it has been doing this for years without issue. So I'm not sure how the KRBTGT password change caused an issue. But regardless I ran the scheduled scan and it locked out my users.

I tweaked the settings to remove the password auditing and it worked without any lockouts! So Saturday morning we will know if this was at the root of the problem. I will post an update on Monday and with luck close out this ticket.

Thanks!
0
arnoldCommented:
not familiar, unless the security scanner has a kerberous dependence.....
0
CCB-TechAuthor Commented:
The comment to refocus on the Account Lockouts instead of the Kerberos logs led me to the actual root of the problem. As I mentioned a security scan was the guilty party. I'm still not sure what caused the Kerberos change to make it act up. But after changing the settings within it we have had our first Saturday without lockouts in almost 2 months!

So I'm closing this question. Thanks @Arnold!
0
arnoldCommented:
glad I could help.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2012

From novice to tech pro — start learning today.