Link to home
Start Free TrialLog in
Avatar of aungelbach
aungelbachFlag for United States of America

asked on

Mystery account lockout at 1:05PM everyday

Problem:

A server [Netwrix] is creating an account lockout at a specific time every day. It is a non-interactive login and only happens once a day at the same time every day.


Scenario:

Active Directory with multiple sites (just for full disclosure)

One site has 5 servers.  One being a Domain Controller another being a Netwrix server.

The Netwrix server authenticates to the DC that is in the same site.


As part of the takeover process we changed the password to a service account used on the Netwrix server, and also changed the account used by Netwrix.

There are no services using named accounts, just NT auth, Local system, etc

There are no scheduled tasks that use the old account.  

None that use any accounts really.


The SQL database did have some instances of maintenance jobs owned by the old account. Changing those fixed a few lingering account lockouts.

We did not see anything else in SQL that could be tied to this, but SQL is not our strong suit.


There do not appear to be any login scripts from GPO.

Backup uses a different account

Nothing stands out in sysinternal autoruns


Open to any suggestions of where to look next.  



Avatar of Jagar Mohammad
Jagar Mohammad
Flag of Canada image

Since you have already ruled out some potential causes of the account lockouts, here are some other areas you may want to investigate:

  1. Scheduled tasks: Even though you mentioned there are no scheduled tasks that use the old account, it is still worth checking if there are any scheduled tasks that may be indirectly causing the lockouts. You can use the Task Scheduler to view all the tasks and their properties.

  2. Services: Although you mentioned there are no services using named accounts, it's still worth checking if there are any services running on the Netwrix server that may be using cached credentials for the old account. You can use the Services console to view all the services and their properties.

  3. Network connections: It's possible that some application or service is using the old credentials to connect to another server or resource on the network. You can use the netstat command to view all active network connections and their associated processes.

  4. Group Policy Objects: There may be some group policies applied to the Netwrix server or the user accounts that are causing the account lockouts. You can use the Group Policy Management Console to review all the applied policies and their settings.

  5. Security logs: Finally, you should review the security logs on the Netwrix server and the domain controller to see if there are any events related to the account lockouts. Look for event IDs 4740 (account lockout) and 4625 (failed login attempts). These logs may provide more information about the source of the lockouts.

can we see a screenshot showing where you see the daily-created account?

if you don't delete it today, will a new account be created tomorrow?

are the newly created user accounts always the same username or randomly?
Avatar of aungelbach

ASKER

@bbao 
The same account is locked out each day at the same time, give or take a 15 seconds.

The account has always existed, but once the password was changed, it began creating failed login events.


@Jagar
Thank you for the tips.  The only logs that show failed login or locked accounts are on the domain controller.   The Netwrix server doesn't show any login failure events at all.


Ironically, Netwrix is an audit reporting program and it does log the failed logins and locked account detected by the domain controller and that the source is the Netwrix server.  We did not find anything in the scheduled tasks, services or GPO using the account.  


Thank you both for the suggestions.





Hello @aungelbach 


In case if you are still facing this issue, I would suggest to take a different route.

To my understanding from your question, Its a single account which is getting locked out on a fixed time.

The investigation needs to have more clarity around the impacted account. if its a single account getting locked out every day at a fixed time, The focus we should keep is to identify from where the bad password attempts are coming up.


In general, the bad password attempts are

   1) From user's computer, where the password which was stored for some reasons was not updated after  a recent password change or password wrongly updated for unknow reasons.

  2) From a different computer, used by the user for some activity, however some program or scheduled jobs running with a wrong password

  3) From a computer, with in the domain - however doesn't have any relation with the target user who is getting impacted

   4) From an internet facing system, which has AD integrated credentials used for authentication 



1 and 2 are due to the fact that user failed to keep the password updated some where. Our job is to identify where the password needs to be udpated :). 

3 and 4 are totally different scenarios. However, user may not have any role in this. To be clear, if one computer on the domain got some malware running and trying to do a dictionary attack on random users every day. These programs are intelligent to stop attempting once account gets locked out. Even, the same program might be attempting for multiple users every day. ie, this could be a larger risk from the security perspective. 4 is also of the same category. The only difference is that since the attempts are made from internet, they dont have any other access.



Now - How should we track the source of lockouts?


This has two steps. First, We need to identify on which domain controller the lockouts are happening. If you have 5 domain controllers, bad password attempt can hit on one of the five domain controller. And once cross the bad password threshed, account will get locked out on that domain controller. And then replicated to other domain controllers. IF I recall correctly, the lock out events on PDC will give you the source domain controller from where account lockout got triggered. This is important to understand because we need to be sure if its happening on a random domaincontrolelr or a fixed domain controller. 


If its a fixed domain controller, the next step is to enable netlogon debug logging. 



https://learn.microsoft.com/en-us/troubleshoot/windows-client/windows-security/enable-debug-logging-netlogon-service


If the lockouts are happening through random domain controller, We may need to get netlogon debug logging on all domain controllers.


The reason why we need debug logging is to trace the requests handedled by netlogon service. This includes the authentication requests. 

Once netlogon debug logging is enabled, Wait for the next lockout event. Look back on the netlogon debug log file and locate the details on the bad password attempts from where its coming up. This will have the information about the calling computer or ip address - which can help to naildown where we need to shift the focus.



The below blog post will give more clarity with screenshots on netlogon debug logging.


https://serverspace.io/support/help/how-to-troubleshoot-account-lockout-issues-in-active-directory/


If its happening from one of the internal computer or ip - Look closer on that machine. If the user is connected with that machine, may be he or she can help us to explain the use cases. If the user is not connected, it could be some kind of malware attempting to crack the password. 


If its happening from an internet facing application server - try blocking those public IPs/ ranges. If its happening again from a new IP, try changing the login name of the user to some random one (I had to do it once). Here is one similar thread.

https://community.spiceworks.com/topic/2231713-disable-domain-account-lockouts-from-failed-owa-login-attempts


What ever be the case, Keep the investigations going on in the right direction until the root cause is identified - especially it the case falls under 3 or 4.

If there is any further assistance required, Please feel free to put your questions as comments.


Good luck.


Shaba

<p>Hi Shaba,&nbsp;</p><p><br></p><p>Thank you for the input.</p><p><br></p><p>We know 100% the lockouts are coming from a specific server: &nbsp;NW-SR01&nbsp;</p><p>The Active Directory site where NW-ST01 is located only has 1 domain controller, so all the lockouts are logged on that DC.</p><p>The account lockout is only happening to one user account: &nbsp;NW-User01</p><p>This is the user account that is used in a number of services running on the NW-SR01 server: Netwrix, SQL, Backup, etc</p><p>The account lockout only happens once per day at 1:05pm, from the NW-SR01 server.</p><p>The system logs show it is not an interactive login, so some script, task or service must be doing it.</p><p>None of the related servers are internet facing.</p><p><br></p><p>We were forced to change the password for the NW-User01 account and this started the account lockout issues.</p><p>We have fixed all but one. &nbsp;</p><p><br></p><p><br></p><p><br></p>
ASKER CERTIFIED SOLUTION
Avatar of Shabarinath TR
Shabarinath TR
Flag of India image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial

Hi Shaba, 


Thank you again for the link.  I did go through all those suggestions and still came up empty.


I'm confident now that the issue is somewhere in SQL, so I will reach out to an SQL friend for help there.


I will mark you suggestion as the solution as it was very informative.


Thank you again.