sadian
asked on
inetinfo.exe 100% CPU and exchange server 2000
I am running Windows server 2000 & MS exchange 2000 with sp3 and all other relavent updates. Just yesturday around 2:30pm Inetinfo.exe started caping out my CPU at 100% utilization and it is also using anywhere between 250 and 300MB of memory.
I have attempted to troubleshoot this issue but many of the guides to resolve the inetinfo.exe CPU utilization issue involve a bug that was fixed with one of the previous exchange service packs which I had installed well over a year ago.
The first thing I did was set all of the exchange related services to manual and reboot the machine. This seems to stop the problem, but then of course I don't have a running exchange server. If I try to kill INETINFO.exe with task manager, it locks up taskmgr.exe and I was forced to reboot to stop taskmgr.exe.
I found that I can successfully kill the INETINFO.exe with sysinternals process explorer, however - the system auto-restarts INETINFO.exe, but it doesn't restart w3svc or pop3 (which I then have to manually start) and the problem starts happening again within a minute once I do. When I kill INETINFO and it restarts itself, it doesn't go wacko on the memory & CPU until I restart the other exchange & IIS related services.
Secondly, I used sysinternals process explorer to nail down which thread inside INETINFO.exe is causing the problem and it appears to be caused by pretty much explusively* by PHATQ.DLL (the advanced transfer queue). The PHATQ.DLL in this circumstance has a delta between 200-300 and uses all of the CPU time. If I kill that thread using the process explorer it appears to stop the high CPU usage but then my emails don't get delivered and INETINFO.exe is still there using 250+MB of ram. *I say pretty much exclusively because in one only case over 7 hours of monitoring it was the LHATQ.DLL thread causing the problem instead.
Finally, I read somewhere that high utilization can be caused by worms/viruses. We are currently running a completely updated version of trendmicro's officescan (which includes an exchange email scanner/filter). To test this case I disconnected the network from the server and this did not appear to make any changes in the way that INETINFO.exe or phatq.dll were behaving.
I spent from 2:30PM to about 9:30PM yesturday troubleshooting this problem. 2:30 to 5:00pm are high traffic times and there was no change in INETINFO.exe or phatq.dll's performance during the time I spent troubleshooting. Finally, I also spent time monitoring the traffic on my ports on my firewall and after-hours there wasn't much of anything targeting my server according to tcpdump on my firewall.
Any thoughts on this issue are greatly appreciated, if I could give this question 10000 points I alsolutely would because my company if heavily reliant on our exchange server and a lot of the customer traffic to our web site (which is located on a second server machine on the same lan) results in emails which are processed by this exhange server.
Thanks in advance,
Sadian
I have attempted to troubleshoot this issue but many of the guides to resolve the inetinfo.exe CPU utilization issue involve a bug that was fixed with one of the previous exchange service packs which I had installed well over a year ago.
The first thing I did was set all of the exchange related services to manual and reboot the machine. This seems to stop the problem, but then of course I don't have a running exchange server. If I try to kill INETINFO.exe with task manager, it locks up taskmgr.exe and I was forced to reboot to stop taskmgr.exe.
I found that I can successfully kill the INETINFO.exe with sysinternals process explorer, however - the system auto-restarts INETINFO.exe, but it doesn't restart w3svc or pop3 (which I then have to manually start) and the problem starts happening again within a minute once I do. When I kill INETINFO and it restarts itself, it doesn't go wacko on the memory & CPU until I restart the other exchange & IIS related services.
Secondly, I used sysinternals process explorer to nail down which thread inside INETINFO.exe is causing the problem and it appears to be caused by pretty much explusively* by PHATQ.DLL (the advanced transfer queue). The PHATQ.DLL in this circumstance has a delta between 200-300 and uses all of the CPU time. If I kill that thread using the process explorer it appears to stop the high CPU usage but then my emails don't get delivered and INETINFO.exe is still there using 250+MB of ram. *I say pretty much exclusively because in one only case over 7 hours of monitoring it was the LHATQ.DLL thread causing the problem instead.
Finally, I read somewhere that high utilization can be caused by worms/viruses. We are currently running a completely updated version of trendmicro's officescan (which includes an exchange email scanner/filter). To test this case I disconnected the network from the server and this did not appear to make any changes in the way that INETINFO.exe or phatq.dll were behaving.
I spent from 2:30PM to about 9:30PM yesturday troubleshooting this problem. 2:30 to 5:00pm are high traffic times and there was no change in INETINFO.exe or phatq.dll's performance during the time I spent troubleshooting. Finally, I also spent time monitoring the traffic on my ports on my firewall and after-hours there wasn't much of anything targeting my server according to tcpdump on my firewall.
Any thoughts on this issue are greatly appreciated, if I could give this question 10000 points I alsolutely would because my company if heavily reliant on our exchange server and a lot of the customer traffic to our web site (which is located on a second server machine on the same lan) results in emails which are processed by this exhange server.
Thanks in advance,
Sadian
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
The biggest shame of all of this is that I have a brand new server sitting on my desk that is waiting for me to install all of its software (the new server was off during all of this hoopla) and if the existing server could behave for a week more - I could have it our of production. *ARGH!* :-)
Sadian
Sadian
You can view the queues via Exchange system manager.
ESM, Servers, <your server>, Protocols, SMTP, Default SMTP VS, Queues.
SQL doesn't use inetinfo so that would continue to operate normally.
Simon.
ESM, Servers, <your server>, Protocols, SMTP, Default SMTP VS, Queues.
SQL doesn't use inetinfo so that would continue to operate normally.
Simon.
Questions:
1.) Leave IIS services running and all exchange services running. The only service I want you to keep stopped is SMTP. Does the problem still occur?
2.) Dismount all mailbox stores on the server and start Exchange services and IIS services. Does inetinfo.exe process still hang?
Let me know the answer to these questions and I will help you out. It's hard to know root cause unless you capture a hang dump of inetinfo.exe process during the hang. Their are several possibilities of what the actual problem could be. I would agree with Sembee to uninstall AV first to see if problem still occurs. If it does, reply to my above questions.
Thx,
Russ Maxwell
1.) Leave IIS services running and all exchange services running. The only service I want you to keep stopped is SMTP. Does the problem still occur?
2.) Dismount all mailbox stores on the server and start Exchange services and IIS services. Does inetinfo.exe process still hang?
Let me know the answer to these questions and I will help you out. It's hard to know root cause unless you capture a hang dump of inetinfo.exe process during the hang. Their are several possibilities of what the actual problem could be. I would agree with Sembee to uninstall AV first to see if problem still occurs. If it does, reply to my above questions.
Thx,
Russ Maxwell
ASKER
Russ:
1.) Leave IIS & exchange services on - turn SMTP off:
When I try to stop SMTP - it doesn't respond. So I set it to manual in services.msc and rebooted. Afterward INETINFO.exe is at nominal memory and cpu useage. (of course this stops the email)
2.) dismount all mailbox stores on the server & start exchange and IIS services:
After I restarted with SMTP disabled, I dismounted the mailbox store (only had one) and the public folders store. After they were dismounted, I enabled SMTP. Afterward INETINFO.exe is at nominal memory and cpu useage. (of course, again, this stops the email)
1.) Leave IIS & exchange services on - turn SMTP off:
When I try to stop SMTP - it doesn't respond. So I set it to manual in services.msc and rebooted. Afterward INETINFO.exe is at nominal memory and cpu useage. (of course this stops the email)
2.) dismount all mailbox stores on the server & start exchange and IIS services:
After I restarted with SMTP disabled, I dismounted the mailbox store (only had one) and the public folders store. After they were dismounted, I enabled SMTP. Afterward INETINFO.exe is at nominal memory and cpu useage. (of course, again, this stops the email)
ASKER
I'd like to add to that -
after I disabled the stores. I reenabled the public store and that didn't seem to cause a problem. but the second I enabled the mailbox store - it went poof until I reset the INETINFO.exe (which also reset SMTP to off by default)
after I disabled the stores. I reenabled the public store and that didn't seem to cause a problem. but the second I enabled the mailbox store - it went poof until I reset the INETINFO.exe (which also reset SMTP to off by default)
ASKER
Must have been something with the scanmail program that hooks into SMTP. I reinstalled scanmail on the server and now it seems to be working like a champ.
Cool, it probably was a 3rd party transport event sink that was jacking stuff up in the transport engine.
Thx,
Russ Maxwell
Thx,
Russ Maxwell
ASKER
Its our main application server. It serves files, is our DNS server, and runs our MSSQL 2k standard database (both applications appear to be unaffected when INNETINFO.exe is behaving) I should also include that our AV seems to be behaving normally even while the problems are occuring or also if IIS/Exchange services are completely shut down. (our email scan monitor detects and scans new emails normally in both cases.)
What are the queues like:
I assume you're refering to the email queues - we serve about 50 users and don't have what I would consider to be a lot of email traffic. Especially this time of year - its our slow season. Where could I find the email queues?
I have to look futher into the antivirus in regard to this issue. Today I plan to call in the local MS experts to see what their perspecive is.
If anyone has any more input - I would sure appreciate any/all of it!
Thank you for your quick response,
Sadian