Link to home
Start Free TrialLog in
Avatar of splitrockit
splitrockit

asked on

Account Lockouts

I am getting the following error on a 2008 R2 Server. Any help would be appreciated.




An account failed to log on.

Subject:
      Security ID:            SYSTEM
      Account Name:            x
      Account Domain:            x
      Logon ID:            0x3e7

Logon Type:                  3

Account For Which Logon Failed:
      Security ID:            NULL SID
      Account Name:            Administrator
      Account Domain:            x

Failure Information:
      Failure Reason:            Unknown user name or bad password.
      Status:                  0xc000006d
      Sub Status:            0xc000006a

Process Information:
      Caller Process ID:      0x298
      Caller Process Name:      C:\Windows\System32\lsass.exe

Network Information:
      Workstation Name:      x
      Source Network Address:      x
      Source Port:            65518

Detailed Authentication Information:
      Logon Process:            Advapi  
      Authentication Package:      MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
      Transited Services:      -
      Package Name (NTLM only):      -
      Key Length:            0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
      - Transited services indicate which intermediate services have participated in this logon request.
      - Package name indicates which sub-protocol was used among the NTLM protocols.
      - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.
Avatar of Rich Weissler
Rich Weissler

Looks like something on computer 'x' is attempting to login to the 'administrator' account, but is using a bad password.

If you actually use the administrator account (and don't have it disabled for example), that might indicate an administrator who lost the password.
Could be malware attempting to brute force the password, in which case, I'd track down computer 'x' and wipe/rebuild.
Could be a service or software with a stored password, and the account password was changed.
Avatar of splitrockit

ASKER

Thanks Rich. I see that the Called Process Name is lsass.exe.  The Logon Process:  Advapi . This is happening on a Windows 2008 R2 server. Any idea how I can find that process and how to find where it is storing the bogus credentials?
The parts of that log entry you obscured, is the source the same machine as the destination, or is the call coming from a different computer?

If I were tracking it down, I'd try the following:
First, because it's easy -- on the source machine, I'd pull up services, and see if any of them are attempting to authenticate as Administrator.
If that isn't fruitful: second, from an elevated command prompt, I'd run 'netstat -aonb >[textfile].txt', and look at that connection on port 65518 (although it may be on a different port by then)... although I don't know if that refers back to lsass.exe on the local machine.  If that does refer to lsass, it should give you the IP address and port of the process on the other end of that connection, which you'd trace out the same way.  (And there's likely a better way to do this... but it's the quick and dirty way I'd start with if I were doing it...)
Agree with Rich. Need to trace to the machine that you masked away. Its authentication via network failed. Most of the time, it is not a user that is attempting to login but a scheduled task service using an admin account that failed. If you see regular timing in the event, that would be one strong symptom of the task configured in the machine.

Just to share some isolation done by others,

In the most severely affected system I have done the following to isolate the issue and after each reverted the change:


Shut down the terminal / remote desktop services server and the generic failed logons did continue.

Disconnected the domain controller server from the network and the generic failed logons did continue.

Rebooted the server into Safe Mode with no networking and the generic failed logons did not continue.

Stopped and disabled all "unnecessary" services (monitoring agent, backup, network filtering integration, TeamViewer, antivirus, etc) and the generic failed logons did continue.

Stopped and disabled Windows Server Essentials services (WseComputerBackupSvc, WseEmailSvc, WseHealthSvc, WseMediaSvc, WseMgmtSvc, and WseNtfSvc) and the generic failed logons did not continue.

Eventually, stopped and disabled the Windows Server Essentials Management Service (WseMgmtSvc) and the generic failed logons did not continue.

I have double-checked that the Windows Server Essentials Management Service (WseMgmtSvc) is responsible for these generic failed logons by disabling it for a few days and there were no generic failed logons and enabling it for a few days and there were thousands of generic failed logons.

I found the following scheduled task:


Name: "Alert Evaluations"
Location: "\Microsoft\Windows\Windows Server Essentials"
Author: "Microsoft Corporation"
Description: "This task periodically evaluates the health of the computer."
Account: "SYSTEM"
Triggers: "At 08:54 on 28/10/2014 - After triggered, repeat every 30 minutes indefinitely"
Actions: "Start a program: C:\Windows\System32\Essentials\RunTask.exe /asm:"C:\Windows\Microsoft.Net\assembly\GAC_MSIL\AlertFramework\v4.0_6.3.0.0__31bf3856ad364e35\AlertFramework.dll" /class:Microsoft.WindowsServerSolutions.NetworkHealth.AlertFramework.HealthScheduledTask /method:EvaluateAlertsTaskAction /task:"Alert Evaluations""
This timeframe almost exactly matches the behaviour above so I disabled it to see if it affects the issue.
For info,

Event ID: 4625. "An account failed to log on".

Logon Type: 3. "Network (i.e. connection to shared folder on this computer from elsewhere on network)".

Security ID: NULL SID. "A valid account was not identified".

Sub Status: 0xC0000064. "User name does not exist".

Caller Process Name: C:\Windows\System32\lsass.exe. Local Security Authority Subsystem Service (LSASS), is a process in Microsoft Windows operating systems that is responsible for enforcing the security policy on the system. It verifies users logging on to a Windows computer or server, handles password changes, and creates access tokens. It also writes to the Windows Security Log.
Have you checked Credentials Manager?  Try after deleting save accounts/passwords in Credential manager.  It could be as simple as a saved account information after mapping a drive.
Yes, you can check:

1. If there is any service still using this AD account.
2. The iTunes saved user credentials
3. mapped network drive using this credential
4. printers mapped using this credential.
Based on the Sub Status of 0xc000006a, it is states user name is correct but the password is wrong.
The logon type is 3 and it occurs due to accessing a computer from elsewhere on the network (i.e Remote Desktop sharing tool), or accessing other resources like Network Share from elsewhere on the network by passing credentials.

One of the most common sources of logon events with Logon type 3 is connections to shared folders or printers.
Need to trace down the machine to check still any scheduled task, services that uses the USER Credentials used for connecting to the shared folder, printer, and even service (appls or tools) using LDAP to authenticate user.

Can try also from the cmd window run:  rundll32 keymgr.dll,KRShowKeyMgr and remove any items that appear in the list of Stored User Names and Passwords. Restart the computer.
My apologies to all those who contributed. I have been away for a few days. I appreciate all your contributions. I will look at them all today. Thank you all for your help.
I am using Netwrix. It shows the attempts are coming form fe80::cd7f:8a42:63ec:ea70%10

Any idea how I can translate that?

Thanks again.
Looks like a IPv6 address for fe80::cd7f:8a42:63ec:ea70%10
Link-Local addresses are designed to be used for addressing on a
   single link for purposes such as automatic address configuration, neighbor discovery, or when no routers are present.

   Routers must not forward any packets with Link-Local source or destination addresses to other links.

https://tools.ietf.org/html/rfc4291#section-2.5.6

The link-local address as the name implies, is specific only to that local network. In other words, the routers can have the same link-local address and still the directly connected network can communicate with each other without any conflict. Wondering if there is some misconfigured routers..
It is coming from IPV6 and my guess is that it could be an IP on your DHCP server.  Have you looked on your DHCP server and see what device has that IP.
The current network configuration is using IPv4 DHCP.
Have you tried disabling the IPV6 address on the computer which is getting locked?
The Source Network Address is the Link-local IPv6 address. IPv6 is installed on the machine and it is set to obtain an address automatically but there is no IPv6 DHCP server on the network.

Maybe good to trace that source machine. You should be able to see its hostname too in the event log.

Once you have identified the source, try running process monitor on the machine and trace to the actual time when error occur in the event log. From there, roughly you know the actual process running and likely be the one causing the issues. I have seen such tools that is running to check for who is login in that source machine.  It may be due to that application trying to map shares or create pipe etc on an IPV6 link local address that does not exist.. Just stop that process or application to see if error still keeping coming..
This is what I get in Netwrix.
It seems to be coming from the server itself. There are no scheduled tasks running. Not sure where else to look

User generated image
Here is what I am seeing in Microsoft Message Analyzer:


Name      Value      Bit Offset      Bit Length      Type      
EventID      4625                  Int32      
Keywords      -9218868437227405312                  Int64      
Level      0                  Byte      
LevelDisplayName      Information                  String      
Channel      Security                  String      
Computer      Server2.domain.local                  String      
Opcode      0                  Int16      
OpcodeDisplayName      Info                  String      
ProcessId      616                  Int32      
EventData      map{SubjectUserSid=S-1-5-18,SubjectUserName=SERVER2$,SubjectDomainName=UN,SubjectLogonId=999,TargetUserSid=S-1-0-0,TargetUserName=Administrator,TargetDomainName=Domain,Status=-1073741715,FailureReason=%%2313,SubStatus=-1073741718,LogonType=3,LogonProcessName=Advapi  ,AuthenticationPackageName=MICROSOFT_AUTHENTICATION_PACKAGE_V1_0,WorkstationName=SERVER2,TransmittedServices=-,LmPackageName=-,KeyLength=0,ProcessId=616,ProcessName=C:\Windows\System32\lsass.exe,IpAddress=192.168.1.1,IpPort=63417}                  MapValue`2      
SubjectUserSid      S-1-5-18                  String      
SubjectUserName      SERVER2$                  String      
SubjectDomainName                        String      
SubjectLogonId      999                  UInt64      
TargetUserSid      S-1-0-0                  String      
TargetUserName      Administrator                  String      
TargetDomainName      Domain            String      
Status      -1073741715                  Int32      
FailureReason      %%2313                  String      
SubStatus      -1073741718                  Int32      
LogonType      3                  UInt32      
LogonProcessName      Advapi                    String      
AuthenticationPackageName      MICROSOFT_AUTHENTICATION_PACKAGE_V1_0                  String      
WorkstationName      SERVER2                  String      
TransmittedServices      -                  String      
LmPackageName      -                  String      
KeyLength      0                  UInt32      
ProcessId      616                  UInt64      
ProcessName      C:\Windows\System32\lsass.exe                  String      
IpAddress      192.168.1.1                  String      
IpPort      63417                  String      
ProviderId      54849625-5478-4994-a5ba-3e3b0328c30d                  Guid      
ProviderName      Microsoft-Windows-Security-Auditing                  String      
EventRecordID      422482572                  Int64      
Task      12544                  Int32      
TaskDisplayName      Logon                  String      
ThreadId      10860                  Int32      
TimeCreated      2016-11-28T08:47:58.1391268                  DateTime      
Version      0                  Byte      
Message      An account failed to log on.

Subject:
      Security ID:            S-1-5-18
      Account Name:            SERVER2$
      Account Domain:            Domain
      Logon ID:            0x3e7

Logon Type:                  3

Account For Which Logon Failed:
      Security ID:            S-1-0-0
      Account Name:            Administrator
      Account Domain:            

Failure Information:
      Failure Reason:            Unknown user name or bad password.
      Status:                  0xc000006d
      Sub Status:            0xc000006a

Process Information:
      Caller Process ID:      0x268
      Caller Process Name:      C:\Windows\System32\lsass.exe

Network Information:
      Workstation Name:      SERVER2
      Source Network Address:      192.168.1.1
      Source Port:            63417

Detailed Authentication Information:
      Logon Process:            Advapi  
      Authentication Package:      MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
      Transited Services:      -
      Package Name (NTLM only):      -
      Key Length:            0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which request...                  String      
UserId                        String
unlikely it is IPv6 related if the log shared is the one of concern - strange why your initial sharing is Ipv6 on the source Ip address which differs.
    Workstation Name:      SERVER2
    Source Network Address:      192.168.1.1
    Source Port:            63417

For below, SID: S-1-0-0 means Nobody or NULL SID, in short no security principal.

     Account For Which Logon Failed:
      Security ID:            S-1-0-0
      Account Name:            Administrator

I am thinking on possibilities below

e.g.  Windows Workgroup logons, printer and file sharing. All PC's connected to the "Home network" will try and logon to each other for their respective WorkGroup. In some cases they will need accounts and passwords setup on each machine.

Perhaps the only way you would get this sorted is to do a network trace of the data and see exactly what it is. There is a free downloadable network trace program called Network Monitor from Microsoft. Go to the server and check out the process and connectivity as shared
I didn't realise this server was facing the public Internet. That being the case, what services actually need to be exposed to the Internet? I'm hoping you'll come back and say just HTTP and FTP, as the rest of the services - the file server, SQL Server and NetBIOS services in particular, really shouldn't be facing the Internet at all.

Once you've listed which services need to be exposed, I'd edit the rules for the remaining enabled rules, go to the scope tab and enter the internal network ranges into the "Remote IP Addresses" list. That way you're ensuring that traffic originating from any other network is unconditionally dropped. That means the Internet requests won't even get as far as being processed by LSASS which will make the server that much safer.
https://social.technet.microsoft.com/Forums/office/en-US/ebb828ed-885b-463a-b962-3675c9c3d0a8/security-threat-in-event-id-4625-unable-to-determine-method-ntlm-logon-type3-through?forum=winservergen
Here is what I am getting from Network Monitor. It appears as though the request is coming from outside the network. Anything in bold I changed or added.

Any help would be appreciated.

 Frame: Number = 6890, Captured Frame Length = 119, MediaType = ETHERNET
- Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[84-2B-2B-4C-E5-0D],SourceAddress:[C0-EA-E4-01-8F-18]
- DestinationAddress: 842B2B 4CE50D [84-2B-2B-4C-E5-0D]
Rsv: (100001..)
UL: (......0.) Universally Administered Address
IG: (.......0) Individual address (unicast)
- SourceAddress: C0EAE4 018F18 [C0-EA-E4-01-8F-18]
Rsv: (110000..)
UL: (......0.) Universally Administered Address
IG: (.......0) Individual address (unicast)
EthernetType: Internet IP (IPv4), 2048(0x800)
- Ipv4: Src = 192.168.1.1, Dest = 192.168.1.10, Next Protocol = TCP, Packet ID = 21307, Total IP Length = 105
- Versions: IPv4, Internet Protocol; Header Length = 20
Version: (0100....) IPv4, Internet Protocol
HeaderLength: (....0101) 20 bytes (0x5)
- DifferentiatedServicesField: DSCP: 0, ECN: 0
DSCP: (000000..) Differentiated services codepoint 0
ECT: (......0.) ECN-Capable Transport not set
CE: (.......0) ECN-CE not set
TotalLength: 105 (0x69)
Identification: 21307 (0x533B)
- FragmentFlags: 16384 (0x4000)
Reserved: (0...............)
DF: (.1..............) Do not fragment
MF: (..0.............) This is the last fragment
Offset: (...0000000000000) 0
TimeToLive: 64 (0x40)
NextProtocol: TCP, 6(0x6)
Checksum: 25592 (0x63F8)
SourceAddress: 192.168.1.1    Router
DestinationAddress: 192.168.1.10 Server
- Tcp: Flags=...AP..., SrcPort=49705, DstPort=LDAP(389), PayloadLen=65, Seq=3799759036 - 3799759101, Ack=1884227581, Win=65535 (scale factor 0x0) = 65535
SrcPort: 49705
DstPort: LDAP(389)
SequenceNumber: 3799759036 (0xE27BB8BC)
AcknowledgementNumber: 1884227581 (0x704F07FD)
- DataOffset: 80 (0x50)
DataOffset: (0101....) 20 bytes
Reserved: (....000.)
NS: (.......0) Nonce Sum not significant
- Flags: ...AP...
CWR: (0.......) CWR not significant
ECE: (.0......) ECN-Echo not significant
Urgent: (..0.....) Not Urgent Data
Ack: (...1....) Acknowledgement field significant
Push: (....1...) Push Function
Reset: (.....0..) No Reset
Syn: (......0.) Not Synchronize sequence numbers
Fin: (.......0) Not End of data
Window: 65535 (scale factor 0x0) = 65535
Checksum: 0xE0A9, Good
UrgentPointer: 0 (0x0)
TCPPayload: SourcePort = 49705, DestinationPort = 389
Ldap: Bind Request, MessageID: 2, Version: 3
- LDAPMessage: Bind Request, MessageID: 2
- ParserHeader:
- AsnId: Sequence and SequenceOf types (Universal 16)
+ LowTag:
- AsnLen: Length = 63, LengthOfLength = 0
Length: 63 bytes, LengthOfLength = 0
- MessageID: 2
+ AsnIntegerHeader:
AsnInt: 2 (0x2)
- OperationHeader: Bind Request, 0(0)
- AsnId: Application Constructed Tag (0)
+ LowTag:
- AsnLen: Length = 58, LengthOfLength = 0
Length: 58 bytes, LengthOfLength = 0
- BindRequest: Version:3, Name:cn=administrator,cn=users,dc=domain,dc=local, UserName: Password Authentication type = simple
+ Version: 3
- Name: cn=administrator,cn=users,dc=un,dc=local
+ AsnOctetStringHeader:
OctetStream: cn=administrator,cn=users,dc=un,dc=local
- Authentication: UserName: Password, Authentication type = simple
- AuthenticationTypeHeader: Authentication type = simple
- AsnId: Context Specific Primitive Tag (0)
+ LowTag:
- AsnLen: Length = 11, LengthOfLength = 0
Length: 11 bytes, LengthOfLength = 0
SimpleAuthentication: Password
This is just showing a LDAP authentication. Nothing canbbe derived from it but I noted you stated router instead of server for the source. So is the source a router or server? This capture is from the source server?
In the event it lists the name of the server which is server2 but then lists the address as 192.168.1.1 which is the router.
Router is making a LDAP call to Server2. Would the router be able to support such AAA LDAP to server? If it does support then is the AAA field forbadmin set up correctly?

Maybe can check on router log
LDAP support on IOS is limited to VPN authentication and unfortunately, cannot be used for Admin (exec) authentication.

CSCug65194    Document LDAP nonsupport for login authentication

AAA does not support using an LDAP method for interactive login authentication. Customers may configure "aaa authentication login default group ldap", but when an interactive (terminal) session tries to authenticate using LDAP, the

following message is syslogged:

"LDAP: LDAP doesn't support [sic] interactive login"
https://supportforums.cisco.com/discussion/11841531/ldap-authentication-router-vty-login
This issue is causing another by with the Sam database issues:
Error      12/1/2016 12:27:10 PM      Directory-Services-SAM      12294      None
The SAM database was unable to lockout the account of Administrator due to a resource error, such as a hard disk write failure (the specific error code is in the error data) . Accounts are locked after a certain number of bad passwords are provided so please consider resetting the password of the account mentioned above.

I have locked the account and I am still getting these error messages.
ASKER CERTIFIED SOLUTION
Avatar of btan
btan

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
The issue was coming from the router. It was trying to sync to AD using expired credentials. Thanks for all your help.
Thanks for sharing