LSASS.EXE consuming excessive CPU on 2003 server

Hi,

I have an issue with one of my client site.

They have 2 DC's (2003 server). The first DC having some issues with LSASS.EXE consuming excessive CPU and the server is locking up. To bring the server, we need to unplug the n/w cable or reboot the server. It will be ok for few minutes and happening again.
So far, i have followed the steps;

1) Checked for virus, installed MS malicious removal tool but couldn't find any infection

2) Tried to install this hostfix  http://support.microsoft.com/kb/939268 but it says, already running with latest SP, so couldn't install.

3) Installed this hotfix and rebooted the server http://support.microsoft.com/kb/902058, unfortunately after the reboot, the CPU usage became 100%.

I couldn't find any errors in event logs. This is really driving me crazy.

Any advise would be much appriaciated.
LVL 24
Radhakrishnan RSenior Technical LeadAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

JaihuntCommented:
I believe you are running windows 2003 SP2. Please check all recent patches and updates are updated.

Some Lsass.exe related recent hot fixes which will update the files to recent version

http://support.microsoft.com/?id=2581130
http://support.microsoft.com/?id=979159
http://support.microsoft.com/?id=976947
http://support.microsoft.com/?id=976947

Reference

http://blogs.technet.com/b/yongrhee/archive/2013/07/31/list-of-domain-controllers-dc-s-related-hotfixes-post-sp2-for-windows-server-2003-sp2-or-windows-server-2003-r2-sp2.aspx

Also check what version Lsass.exe there in the server and check and update it to latest version.
0
Radhakrishnan RSenior Technical LeadAuthor Commented:
Hi,

Thanks, i'll have a look and get back to you.
0
SandeshdubeySenior Server EngineerCommented:
Troubleshooting High LSASS CPU Utilization on a Domain Controller (Part 1 of 2)
http://blogs.technet.com/b/askds/archive/2007/08/20/troubleshooting-high-lsass-cpu-utilization-on-a-domain-controller-part-1-of-2.aspx
http://support.microsoft.com/kb/2550044

Troubleshooting High CPU Utilization issues using Tracelog.exe
http://blogs.technet.com/b/askperf/archive/2012/01/20/troubleshooting-high-cpu-utilization-issues-using-tracelog-exe.aspx

In addition I would also recommend to disable Antivirus and security application if any installed on DC and reboot the server and check also ensure that latest hotfix and SP is installed on the server .

Process explore:http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
 
Hope this helps
0
Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

Radhakrishnan RSenior Technical LeadAuthor Commented:
Hi,

I had tried to install the hotfixes but 2 of them couldn't install as it said already SP2 installed, hence it's not required. Rebooted the server, as soon as i logged in, the CPU utilization went 100%.

Already i had installed process explore and wireshark, from this only i had found lsass.exe comsuming 100% CPU but couldn't find the reason why?

I have advised my client to shut down all of their computers, once the machines were shutdown, the CPU utilization came to 9 - 10% (wondering).

Started one batch of (20 machines) machines, immediately the CPU utilization reached 100%. This should have happened even if i reboot another batch of machines.

I suspect that there is something causing from the client side. Could be a virus but MS malicious removal or Malware bite didn't catch any infections.

This is bit challenging and critical issue for me now, as my client stopped one of my payment and asked to resolve the issue first.

Any more ideas?
0
SandeshdubeySenior Server EngineerCommented:
I will recommend to scan the computer with latest AV and also installed hotfix too.This could be due to virus issue.Virus like conflicker can cause banswidth choke,etc.Take first couple of computer out of network install hofix and scan and perform the test and check how does it work.

Also ensure that there is no addition app,thirdy party software/application installed on DC.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Radhakrishnan RSenior Technical LeadAuthor Commented:
Hi Sandeshdubey,

Thanks for your prompt reply.

There are plenty of 3rd party applications running on the DC. Already i had advised them to remove the 3rd party applications from the DC but they said that it's required for them and it's running from past few years.

However, as a testing purpose, i'll uninstall and let you know :)
0
Radhakrishnan RSenior Technical LeadAuthor Commented:
Hi,

Further update:-

Uninstalled most of the 3rd party application on the server but the issue still persists.

Also, performed sfc /scannow which finished without any issues but still the lsass.exe consuming high cpu.

Re-registered all the dll's in system32 folder, rebooted the server, it was ok for few minutes, the CPU usage went 100% as soon as the users started log in.

It seems to be some kind relation between user authentication (kerberos) but couldn't figure it out.

Any help would be much appriciated.
0
SandeshdubeySenior Server EngineerCommented:
Have you check event log on DC are you getting any errors and warming message?

As you have two Dcs, if you shut this problematic DC.Does the issue occur on other DC,do client face any authentication issue?
0
SandeshdubeySenior Server EngineerCommented:
In addition if the other DC is good and clients are not facing issue.Take the backup of problematic server related data,app,etc as per requirement.Clean the OS/format and reload the OS and add the server back as DC.

Also dont install the application/thirdy pary software on DC it should be on member server.
0
Radhakrishnan RSenior Technical LeadAuthor Commented:
Hi,

Finally, i have fixed the issue. It was due to a 3rd party application called "Impero" (some monitoring software).

I performed a netstat -ano and tracked down the PID for lsass.exe and checked which computers are getting more connections.

Disabled the non MS services one by one and found that the CPU usage went down as soon as i disabled the Impero services.

It has been disabled via GPO for globally and everything seems to be fine at the moment.

Thanks for your help in this matter.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2003

From novice to tech pro — start learning today.