CCB-Tech
asked on
Why do all accounts get locked out after resetting KRBTGT account?
About a month ago I reset our KRBTGT account using this script from Technet:
https://gallery.technet.microsoft.com/Reset-the-krbtgt-account-581a9e51
Everything verified and succeeded. Every since then like clockwork every Saturday almost our entire AD of User Accounts gets locked out. Computer Accounts are unaffected. I have checked and found a ton of event ID 4771's with the Failure Code 0x18 and 0x12. This is reflected in their meanings documented below:
https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=4771
They claim that pre-auth is failing and then causing the lockout. I've checked and a lot of our Kerberos tickets do expire on Saturday. In addition to the 4771's I also see a lot of 4768 events.
https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=4768
I've tried rebooting all clients, though not servers, to flush the existing tickets but that hasn't helped. Neither has bouncing the DC's or Restarting the Ticket Granting service. I've even played with some powershell scripts to use klist to flush the tickets for all users. Though I haven't gotten them to work remotely.
Still this has been going on for over a month and I can't find anyone else having a similar issue. Resetting the KRBTGT account should NOT cause this to happen from everything I've read. We are at 2012 R2 Functional for our Forest and Domain level. When going up a level in the past we never had a similar issue. It was only when I ran the above script that all heck broke loose.
I'm happy to provide any additional information requested.
Thanks
https://gallery.technet.microsoft.com/Reset-the-krbtgt-account-581a9e51
Everything verified and succeeded. Every since then like clockwork every Saturday almost our entire AD of User Accounts gets locked out. Computer Accounts are unaffected. I have checked and found a ton of event ID 4771's with the Failure Code 0x18 and 0x12. This is reflected in their meanings documented below:
https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=4771
They claim that pre-auth is failing and then causing the lockout. I've checked and a lot of our Kerberos tickets do expire on Saturday. In addition to the 4771's I also see a lot of 4768 events.
https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=4768
I've tried rebooting all clients, though not servers, to flush the existing tickets but that hasn't helped. Neither has bouncing the DC's or Restarting the Ticket Granting service. I've even played with some powershell scripts to use klist to flush the tickets for all users. Though I haven't gotten them to work remotely.
Still this has been going on for over a month and I can't find anyone else having a similar issue. Resetting the KRBTGT account should NOT cause this to happen from everything I've read. We are at 2012 R2 Functional for our Forest and Domain level. When going up a level in the past we never had a similar issue. It was only when I ran the above script that all heck broke loose.
I'm happy to provide any additional information requested.
Thanks
ASKER
I agree that you should almost never need to reset that account. However, there is a danger of an attacker achieving persistence that can only be negated by changing the password on the KRBTGT account periodically. See here for the dangers of Golden Tickets:
https://blog.stealthbits.com/complete-domain-compromise-with-golden-tickets/
As of higher functional levels of Active Directory Microsoft recommended changing the password on a regular schedule:
https://technet.microsoft.com/en-us/library/dn745899.aspx?f=255&MSPPError=-2147217396#Anchor_5
So the article you linked is correct, but is not reflective of current best practices. Also, your answer doesn't address the fact that I have already changed the password and it shouldn't have caused the fallout that it has.
https://blog.stealthbits.com/complete-domain-compromise-with-golden-tickets/
As of higher functional levels of Active Directory Microsoft recommended changing the password on a regular schedule:
https://technet.microsoft.com/en-us/library/dn745899.aspx?f=255&MSPPError=-2147217396#Anchor_5
So the article you linked is correct, but is not reflective of current best practices. Also, your answer doesn't address the fact that I have already changed the password and it shouldn't have caused the fallout that it has.
IT's hard trying to "guess" what is going on if you don't provide information about your event viewer or anything related to addressing the issue.
You should have events related to whats going on in your event viewer or in the directory events.
You should have events related to whats going on in your event viewer or in the directory events.
ASKER
I supplied the only Event ID evidence I have available in the form of 4771 and 4768 and the Failure Codes they Contain. I'm not asking for a "guess" as all evidence has been provided. Here is a TLDR version as it appears you didn't read the entirety of the question:
Sorry for appearing snarky but I provided what you are talking about from the get-go and I don't understand why you are asking for that information as if I didn't.
- Change: KRBTGT Account Reset Using Microsoft Provided Powershell Over A Month Ago
- Symptom: Every Saturday almost all AD Accounts are locked as if a bad password was provided. Unlocking them fixes the problem until the next Saturday.
- Evidence: Event ID's 4771 and 4768 as listed above reference a pre-auth failure of the Kerberos ticket. This failure leads to the lockouts. No other events tied to this show in any DC as far as I can tell.
Sorry for appearing snarky but I provided what you are talking about from the get-go and I don't understand why you are asking for that information as if I didn't.
Have you tried looking into duplicate SPNs, and going down that path as a fix to your issues? Based on what you haven't presenting, are we to assume that it's been random accounts locking out, or just a certain set of them?
ASKER
I have not checked into that yet. Good idea. It appears to be random accounts locking out.
What is really peculiar is that some accounts that should no longer be active are even being locked out. Which re-enforces the random part.
What is really peculiar is that some accounts that should no longer be active are even being locked out. Which re-enforces the random part.
ASKER
Okay good idea but the only duplicate SPNs I identified were tied to a decommissioned server from years ago. So back to square one :(.
This account is reset every 30 days if not mistaken through GPO.
in short, you have to rejoin/reset the key set on each system. The key they have through the means you went suggests when the systems came back with the key they have, which is no longer
When used through GPO/Automated process, the old key and the new key are maintained and the system transition from the old key to the new...
the same way if you set a GPO to have each system, (workstation and/or workstation) reset their Machine password...
what is the remedy you are using when all systems are locked out?
in short, you have to rejoin/reset the key set on each system. The key they have through the means you went suggests when the systems came back with the key they have, which is no longer
When used through GPO/Automated process, the old key and the new key are maintained and the system transition from the old key to the new...
the same way if you set a GPO to have each system, (workstation and/or workstation) reset their Machine password...
what is the remedy you are using when all systems are locked out?
ASKER
I've not seen anything hinting at the KRBTGT account being reset every 30 days. I thought that was only for Computer Accounts?
We have actually not had any impact on Computer Accounts, only User Accounts. Something I should have been clearer on in my first post. *Editing now :)* Users are getting locked out same as if they all tried bad passwords. When in reality something is amiss with the Kerberos Ticket as far as I can tell causing a similar lockout.
We have a third party application that makes unlocking all users relatively simple happily.
We have actually not had any impact on Computer Accounts, only User Accounts. Something I should have been clearer on in my first post. *Editing now :)* Users are getting locked out same as if they all tried bad passwords. When in reality something is amiss with the Kerberos Ticket as far as I can tell causing a similar lockout.
We have a third party application that makes unlocking all users relatively simple happily.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I haven't yet used the Accout Lockout tool on this problem so I'll give it a shot.
I was concerned about a sync problem as well but once a week I get a health report on my AD from this Powershell script:
https://gallery.technet.microsoft.com/scriptcenter/Active-Directory-Health-709336cd
I just re-ran it and got a clean bill of health on all measured points. Even our RODC looks fine. That was one of the first things I checked. I know an unhealthy AD can cause all kinds of oddities. I'll amend this information to my first Post as well.
I'll look through the logs with the Account Lockout tool. Maybe that'll shed some new light.
I was concerned about a sync problem as well but once a week I get a health report on my AD from this Powershell script:
https://gallery.technet.microsoft.com/scriptcenter/Active-Directory-Health-709336cd
I just re-ran it and got a clean bill of health on all measured points. Even our RODC looks fine. That was one of the first things I checked. I know an unhealthy AD can cause all kinds of oddities. I'll amend this information to my first Post as well.
I'll look through the logs with the Account Lockout tool. Maybe that'll shed some new light.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I'm checking into all those details and continuing the chase down the Event Log rabbit hole. I'll post more as I gather information. Thanks for the help so far!
ASKER
@Arnold thanks for the idea! I dug around in the Event Logs this time focusing on the Lockouts themselves vs the Kerberos Logs. This led me to find a Security Scanner that was doing a scan at the exact time the lockouts were happening. Now, mind you it has been doing this for years without issue. So I'm not sure how the KRBTGT password change caused an issue. But regardless I ran the scheduled scan and it locked out my users.
I tweaked the settings to remove the password auditing and it worked without any lockouts! So Saturday morning we will know if this was at the root of the problem. I will post an update on Monday and with luck close out this ticket.
Thanks!
I tweaked the settings to remove the password auditing and it worked without any lockouts! So Saturday morning we will know if this was at the root of the problem. I will post an update on Monday and with luck close out this ticket.
Thanks!
not familiar, unless the security scanner has a kerberous dependence.....
ASKER
The comment to refocus on the Account Lockouts instead of the Kerberos logs led me to the actual root of the problem. As I mentioned a security scan was the guilty party. I'm still not sure what caused the Kerberos change to make it act up. But after changing the settings within it we have had our first Saturday without lockouts in almost 2 months!
So I'm closing this question. Thanks @Arnold!
So I'm closing this question. Thanks @Arnold!
glad I could help.
Here's what is the KRBTGT https://blogs.technet.microsoft.com/janelewis/2006/12/19/the-krbtgt-account-what-is-it/
And At least you are in a disaster recovery scenario you should not be messing with that account.