TANGLAD
asked on
w3wp.exe 100% cpu
Have an ERP server on LAN and a IIS in DMZ
Both servers are Dual Core 1.6 GHz, 1 GB memory, no RAID
Bandwith is 10 down / 1 up
Enabled ICMP and ping the wan interface with 39 ms replys
I'm responsible for the servers OS and firewall and network. The IIS is hosting an ASP.net solution. the IIS queries the backend ERP (SQL 2005). I haven't anything to do with the Website and don't have any knowledge about it.
Yesterday we had serious problems since the w3wp.exe process took 100% cpu. There were 60 users accessing the web site. I killed the process but it went to 100% again in few seconds. Restarted the machine but it went to 100% again.
Then I changed the settings for application pools so the w3wp.exe is stopped when it reach 50% cpu. It seemed to help keeping the site stable.
Question 1: I would like to know if this application pool change has any impact for the users accessing the site? Are they experiencing any drop-outs, missing transactions or anything?
Question 2: Why did the w3wp.exe process go to 100%? (Is the hardware to small? or is it bad code in the application? or stupid configured queries against the SQL server?
Question 3: I think 39 ms reply on ping is slow (but not sure). can a slow reply affect the w3wp.exe process?
Please help
Both servers are Dual Core 1.6 GHz, 1 GB memory, no RAID
Bandwith is 10 down / 1 up
Enabled ICMP and ping the wan interface with 39 ms replys
I'm responsible for the servers OS and firewall and network. The IIS is hosting an ASP.net solution. the IIS queries the backend ERP (SQL 2005). I haven't anything to do with the Website and don't have any knowledge about it.
Yesterday we had serious problems since the w3wp.exe process took 100% cpu. There were 60 users accessing the web site. I killed the process but it went to 100% again in few seconds. Restarted the machine but it went to 100% again.
Then I changed the settings for application pools so the w3wp.exe is stopped when it reach 50% cpu. It seemed to help keeping the site stable.
Question 1: I would like to know if this application pool change has any impact for the users accessing the site? Are they experiencing any drop-outs, missing transactions or anything?
Question 2: Why did the w3wp.exe process go to 100%? (Is the hardware to small? or is it bad code in the application? or stupid configured queries against the SQL server?
Question 3: I think 39 ms reply on ping is slow (but not sure). can a slow reply affect the w3wp.exe process?
Please help
ASKER
Thanks. Your answer to question 2: the Microsoft article talks about Exchange but that is not on the servers.
You should check the scripts you edited the last days before the w3wp.exe began to hang up! There seems to be a script that is doing an endless loop for example. Or maybe you are not closing and/or releasing an object correctly? Check your scripts in detail at first!!
Could be also an third party component (activex dll) you are using! Maybe you implemented such a component?
If you limit the cpu time, IIS could handle fewer request than before when reaching the limit.
You should think about expanding the RAM cause windows is using a big part of it for its services and things.
SQL 2005 is running on a different server I think. If not, you definitly need more RAM or should transfer SQL Server to another server!
Could be also an third party component (activex dll) you are using! Maybe you implemented such a component?
If you limit the cpu time, IIS could handle fewer request than before when reaching the limit.
You should think about expanding the RAM cause windows is using a big part of it for its services and things.
SQL 2005 is running on a different server I think. If not, you definitly need more RAM or should transfer SQL Server to another server!
ASKER
Im also wondering why only 60 users at the samt time can kill the cpu?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
I recently hung up one application and iis 6.0 with classic asp with a third party script,cause the author did not destroy all objects (recordsets and connections)! And this application has only up to 10 concurrent sessions ;)
ASKER
OK thanks. I don't know if there are any endless loop. How can i find out. The site seems to work ok when only a few users are on. What actually happens when a user user calls a script with an endless loop? Will the user experiense anything? any errors to look for in logfiles?
ASKER
Can we see something out of this part of a logfile?
12/02/2007 09:58:10 L-8 Critical - Error Global.Application_Error 0 Unhanded exception
Root-1-128410494018906250,
Message: Exception of type System.Web.HttpException was thrown.
Type : System.Web.HttpException
Source : System.Web
Stack : at System.Web.HttpAsyncResult .End()
at System.Web.UI.Page.AspComp atEndProce ssRequest( IAsyncResu lt result)
at ASP.main_aspx.EndProcessRe quest(IAsy ncResult ar) in c:\WINDOWS\Microsoft.NET\F ramework\v 1.1.4322\T emporary ASP.NET Files\root\159ca395\f2cf4e 5e\haubxu5 j.0.cs:lin e 84
at System.Web.CallHandlerExec utionStep. OnAsyncHan dlerComple tion(IAsyn cResult ar)
Target : System.Object End()
------- Inner Exception ------
Message: Exception of type System.Web.HttpUnhandledEx ception was thrown.
Type : System.Web.HttpUnhandledEx ception
Source : System.Web
Stack : at System.Web.UI.Page.HandleE rror(Excep tion e)
at System.Web.UI.Page.Process RequestMai n()
at System.Web.UI.Page.Process Request()
at System.Web.Util.AspCompatA pplication Step.Syste m.Web.Http Applicatio n+IExecuti onStep.Exe cute()
at System.Web.HttpApplication .ExecuteSt ep(IExecut ionStep step, Boolean& completedSynchronously)
Target : Boolean HandleError(System.Excepti on)
------- Inner Exception ------
Message: Object reference not set to an instance of an object.
Type : System.NullReferenceExcept ion
Source : Zpider.eShop.Web.Business
Stack : at Zpider.eShop.Web.Business. BusinessSe rver._GetA greedPrice (String articleNo, Double quantity, Boolean bonusValue, Int32 customerNo, String zipCode)
at Zpider.eShop.Web.Business. BusinessSe rver.GetAg reedPrice( String articleNo, Double quantity, Boolean bonusValue)
at Zpider.eShop.Web.Business. BusinessAr ticleDataC lass._Fill PriceDataT able(DataR ow dsRow)
at Zpider.eShop.Web.Business. BusinessAr ticleDataC lass._Fill ResultData ()
at Zpider.eShop.Web.Business. BusinessBa seDataClas s.bdFetchN ext()
at Zpider.eShop.Web.Base.Zpid erScriptPr ovider._Ta gForeachCl ose(Closin gTagItem item, IDictionary hAttributes, ScriptRunContext runContext)
at Zpider.eShop.Web.Base.Zpid erScriptPr ovider.Pro cessClosin gTag(Closi ngTagItem item, IDictionary hAttributes, ScriptRunContext runContext)
at Zpider.Sys.Lib.Template.Sc riptProces sor._DoExe cuteScript (ParsedScr ipt script, ScriptRunContext runctx)
at Zpider.Sys.Lib.Template.Sc riptProces sor.Execut eScript(Pa rsedScript script, String& outText, IDictionary hProperties)
at Zpider.eShop.Web.Base.Main .ProcessSc ript(Modul e module, IDictionary props)
at Zpider.eShop.Web.Base.Main .ProcessSc ript(Strin g page)
at Zpider.eShop.Web.eShop.Mai nForm.Page _Load(Obje ct sender, EventArgs e)
at System.Web.UI.Control.OnLo ad(EventAr gs e)
at System.Web.UI.Control.Load Recursive( )
at System.Web.UI.Page.Process RequestMai n()
Target : Zpider.eShop.Lib.ArticlePr ice _GetAgreedPrice(System.Str ing, Double, Boolean, Int32, System.String)
12/02/2007 09:58:10 L-8 Critical - Error Global.Application_Error 0 Unhanded exception
Root-1-128410494018906250,
Message: Exception of type System.Web.HttpException was thrown.
Type : System.Web.HttpException
Source : System.Web
Stack : at System.Web.HttpAsyncResult
at System.Web.UI.Page.AspComp
at ASP.main_aspx.EndProcessRe
at System.Web.CallHandlerExec
Target : System.Object End()
------- Inner Exception ------
Message: Exception of type System.Web.HttpUnhandledEx
Type : System.Web.HttpUnhandledEx
Source : System.Web
Stack : at System.Web.UI.Page.HandleE
at System.Web.UI.Page.Process
at System.Web.UI.Page.Process
at System.Web.Util.AspCompatA
at System.Web.HttpApplication
Target : Boolean HandleError(System.Excepti
------- Inner Exception ------
Message: Object reference not set to an instance of an object.
Type : System.NullReferenceExcept
Source : Zpider.eShop.Web.Business
Stack : at Zpider.eShop.Web.Business.
at Zpider.eShop.Web.Business.
at Zpider.eShop.Web.Business.
at Zpider.eShop.Web.Business.
at Zpider.eShop.Web.Business.
at Zpider.eShop.Web.Base.Zpid
at Zpider.eShop.Web.Base.Zpid
at Zpider.Sys.Lib.Template.Sc
at Zpider.Sys.Lib.Template.Sc
at Zpider.eShop.Web.Base.Main
at Zpider.eShop.Web.Base.Main
at Zpider.eShop.Web.eShop.Mai
at System.Web.UI.Control.OnLo
at System.Web.UI.Control.Load
at System.Web.UI.Page.Process
Target : Zpider.eShop.Lib.ArticlePr
That is the problem! You can not detect any endless loop or non closed object within any log! You have to look at your code manually! Its a annoying to check the whole code, but in most cases it is the only way to fix this.
What happens is that IIS can not handle more than a number of requests at the same time. And if you are calling an endless loop the script "hangs" a least for the time you specified in script timeout. While this happens IIS can not server other requests. Or if the number for max. requests at a time is calling an endless loop.
You could raise the number of concurrent requests, but it would not help because this will consume mor ressources and after a while maybe these extra requests are running on the same error.
I would try to check my code and see if there is anything I can fix. Look for all "do...while", "do...until", "for...next" and check if your objects are closed and destroyed. Do not rely on garbage collector in ASP(.NET)! ;)
What happens is that IIS can not handle more than a number of requests at the same time. And if you are calling an endless loop the script "hangs" a least for the time you specified in script timeout. While this happens IIS can not server other requests. Or if the number for max. requests at a time is calling an endless loop.
You could raise the number of concurrent requests, but it would not help because this will consume mor ressources and after a while maybe these extra requests are running on the same error.
I would try to check my code and see if there is anything I can fix. Look for all "do...while", "do...until", "for...next" and check if your objects are closed and destroyed. Do not rely on garbage collector in ASP(.NET)! ;)
Ouch.. got out of my meeting and Sonic already covered the thing here :D
"Object reference not set to an instance of an object."
Seems like you are calling a method or property of an object that is not created or allready destroyed (think it's Zpider.eShop.Web.Business) .
But this error should not result in 100% cpu of w3wp.exe! It's worth a try to fix this error first and see what happens. Maybe there is more happening in background related to that error.
Seems like you are calling a method or property of an object that is not created or allready destroyed (think it's Zpider.eShop.Web.Business)
But this error should not result in 100% cpu of w3wp.exe! It's worth a try to fix this error first and see what happens. Maybe there is more happening in background related to that error.
ASKER
I have looked in the IIS log and can count the users that way. But when the CPU went 100% and a lot of users couldn't load the page, would the connections still be logged in the IIS log or do we have no chance to tell how many users was trying?
Good question. Normally the requests are recognized, but not processed. If there is a log entry I never looked for.
I do not think there will be a log entry on every request, cause the log entires will report success or failure. If the requests are timing out there will be a log entry. If the requests will result to a 403 or whatever error status, they wil be logged, but if they are not processed correctly there will be no entry in log.
But this can be wrong! I do not know exactly.
I do not think there will be a log entry on every request, cause the log entires will report success or failure. If the requests are timing out there will be a log entry. If the requests will result to a 403 or whatever error status, they wil be logged, but if they are not processed correctly there will be no entry in log.
But this can be wrong! I do not know exactly.
ASKER
Would it help adding more hardware?
Both servers are IBM System x3400 7975 - Dual Core Xeon 5110 1.6 GHz - 1 GB Memory - Single HDD SATA 300 7.200 rpm
Would another processor avoid the 100% cpu usage?
Both servers are IBM System x3400 7975 - Dual Core Xeon 5110 1.6 GHz - 1 GB Memory - Single HDD SATA 300 7.200 rpm
Would another processor avoid the 100% cpu usage?
NO! As I said, please check your scripts and components! There is an error within them I think and there is no way arround sighting all scripts!
With better hardware you will only delay the 100% cpu effect! I am sure it will happen again. I am administrating IIS since Windows 2000 release for arround 7 to 8 years now, and in most if not alll cases a bad code was the source of these types of problems.
With better hardware you will only delay the 100% cpu effect! I am sure it will happen again. I am administrating IIS since Windows 2000 release for arround 7 to 8 years now, and in most if not alll cases a bad code was the source of these types of problems.
Unfortunatly there is no way to detect the one script causing the error automatically. There is no debug tool or log that can help you.
Question 2: Check this article out http://support.microsoft.com/kb/900243