Link to home
Start Free TrialLog in
Avatar of TANGLAD
TANGLAD

asked on

w3wp.exe 100% cpu

Have an ERP server on LAN and a IIS in DMZ

Both servers are Dual Core 1.6 GHz, 1 GB memory, no RAID

Bandwith is 10 down / 1 up

Enabled ICMP and ping the wan interface with 39 ms replys

I'm responsible for the servers OS and firewall and network. The IIS is hosting an ASP.net solution. the IIS queries the backend ERP (SQL 2005). I haven't anything to do with the Website and don't have any knowledge about it.

Yesterday we had serious problems since the w3wp.exe process took 100% cpu. There were 60 users accessing the web site. I killed the process but it went to 100% again in few seconds. Restarted the machine but it went to 100% again.

Then I changed the settings for application pools so the w3wp.exe is stopped when it reach 50% cpu. It seemed to help keeping the site stable.

Question 1: I would like to know if this application pool change has any impact for the users accessing the site? Are they experiencing any drop-outs, missing transactions or anything?

Question 2: Why did the w3wp.exe process go to 100%? (Is the hardware to small? or is it bad code in the application? or stupid configured queries against the SQL server?

Question 3: I think 39 ms reply on ping is slow (but not sure). can a slow reply affect the w3wp.exe process?


Please help
Avatar of bungHoc
bungHoc
Flag of Viet Nam image

Question 1: I guess yes, especially if you want to keep overheads as low as possible and have a bunch of sites under application pool.

Question 2: Check this article out http://support.microsoft.com/kb/900243
Avatar of TANGLAD
TANGLAD

ASKER

Thanks. Your answer to question 2: the Microsoft article talks about Exchange but that is not on the servers.
You should check the scripts you edited the last days before the w3wp.exe began to hang up! There seems to be a script that is doing an endless loop for example. Or maybe you are not closing and/or releasing an object correctly? Check your scripts in detail at first!!

Could be also an third party component (activex dll) you are using! Maybe you implemented such a component?

If you limit the cpu time, IIS could handle fewer request than before when reaching the limit.

You should think about expanding the RAM cause windows is using a big part of it for its services and things.

SQL 2005 is running on a different server I think. If not, you definitly need more RAM or should transfer SQL Server to another server!
Avatar of TANGLAD

ASKER

Im also wondering why only 60 users at the samt time can kill the cpu?
ASKER CERTIFIED SOLUTION
Avatar of Sven
Sven
Flag of Germany image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
I recently hung up one application and iis 6.0 with classic asp with a third party script,cause the author did not destroy all objects (recordsets and connections)! And this application has only up to 10 concurrent sessions ;)
Avatar of TANGLAD

ASKER

OK thanks. I don't know if there are any endless loop. How can i find out. The site seems to work ok when only a few users are on. What actually happens when a user user calls a script with an endless loop? Will the user experiense anything? any errors to look for in logfiles?
Avatar of TANGLAD

ASKER

Can we see something out of this part of a logfile?

12/02/2007 09:58:10 L-8 Critical - Error Global.Application_Error 0 Unhanded exception
Root-1-128410494018906250,
 Message: Exception of type System.Web.HttpException was thrown.
Type : System.Web.HttpException
Source : System.Web
Stack : at System.Web.HttpAsyncResult.End()
at System.Web.UI.Page.AspCompatEndProcessRequest(IAsyncResult result)
at ASP.main_aspx.EndProcessRequest(IAsyncResult ar) in c:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files\root\159ca395\f2cf4e5e\haubxu5j.0.cs:line 84
at System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)
Target : System.Object End()
------- Inner Exception ------
   Message: Exception of type System.Web.HttpUnhandledException was thrown.
   Type : System.Web.HttpUnhandledException
   Source : System.Web
   Stack : at System.Web.UI.Page.HandleError(Exception e)
at System.Web.UI.Page.ProcessRequestMain()
at System.Web.UI.Page.ProcessRequest()
at System.Web.Util.AspCompatApplicationStep.System.Web.HttpApplication+IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   Target : Boolean HandleError(System.Exception)
------- Inner Exception ------
      Message: Object reference not set to an instance of an object.
      Type : System.NullReferenceException
      Source : Zpider.eShop.Web.Business
      Stack : at Zpider.eShop.Web.Business.BusinessServer._GetAgreedPrice(String articleNo, Double quantity, Boolean bonusValue, Int32 customerNo, String zipCode)
at Zpider.eShop.Web.Business.BusinessServer.GetAgreedPrice(String articleNo, Double quantity, Boolean bonusValue)
at Zpider.eShop.Web.Business.BusinessArticleDataClass._FillPriceDataTable(DataRow dsRow)
at Zpider.eShop.Web.Business.BusinessArticleDataClass._FillResultData()
at Zpider.eShop.Web.Business.BusinessBaseDataClass.bdFetchNext()
at Zpider.eShop.Web.Base.ZpiderScriptProvider._TagForeachClose(ClosingTagItem item, IDictionary hAttributes, ScriptRunContext runContext)
at Zpider.eShop.Web.Base.ZpiderScriptProvider.ProcessClosingTag(ClosingTagItem item, IDictionary hAttributes, ScriptRunContext runContext)
at Zpider.Sys.Lib.Template.ScriptProcessor._DoExecuteScript(ParsedScript script, ScriptRunContext runctx)
at Zpider.Sys.Lib.Template.ScriptProcessor.ExecuteScript(ParsedScript script, String& outText, IDictionary hProperties)
at Zpider.eShop.Web.Base.Main.ProcessScript(Module module, IDictionary props)
at Zpider.eShop.Web.Base.Main.ProcessScript(String page)
at Zpider.eShop.Web.eShop.MainForm.Page_Load(Object sender, EventArgs e)
at System.Web.UI.Control.OnLoad(EventArgs e)
at System.Web.UI.Control.LoadRecursive()
at System.Web.UI.Page.ProcessRequestMain()
      Target : Zpider.eShop.Lib.ArticlePrice _GetAgreedPrice(System.String, Double, Boolean, Int32, System.String)
 
That is the problem! You can not detect any endless loop or non closed object within any log! You have to look at your code manually! Its a annoying to check the whole code, but in most cases it is the only way to fix this.

What happens is that IIS can not handle more than a number of requests at the same time. And if you are calling an endless loop the script "hangs" a least for the time you specified in script timeout. While this happens IIS can not server other requests. Or if the number for max. requests at a time is calling an endless loop.

You could raise the number of concurrent requests, but it would not help because this will consume mor ressources and after a while maybe these extra requests are running on the same error.

I would try to check my code and see if there is anything I can fix. Look for all "do...while", "do...until", "for...next" and check if your objects are closed and destroyed. Do not rely on garbage collector in ASP(.NET)! ;)

Ouch.. got out of my meeting and Sonic already covered the thing here  :D
"Object reference not set to an instance of an object."

Seems like you are calling a method or property of an object that is not created or allready destroyed (think it's Zpider.eShop.Web.Business).

But this error should not result in 100% cpu of w3wp.exe! It's worth a try to fix this error first and see what happens. Maybe there is more happening in background related to that error.
Avatar of TANGLAD

ASKER

I have looked in the IIS log and can count the users that way. But when the CPU went 100% and a lot of users couldn't load the page, would the connections still be logged in the IIS log or do we have no chance to tell how many users was trying?
Good question. Normally the requests are recognized, but not processed. If there is a log entry I never looked for.

I do not think there will be a log entry on every request, cause the log entires will report success or failure. If the requests are timing out there will be a log entry. If the requests will result to a 403 or whatever error status, they wil be logged, but if they are not processed correctly there will be no entry in log.

But this can be wrong! I do not know exactly.
Avatar of TANGLAD

ASKER

Would it help adding more hardware?

Both servers are IBM System x3400 7975 - Dual Core Xeon 5110 1.6 GHz - 1 GB Memory - Single HDD SATA 300 7.200 rpm

Would another processor avoid the 100% cpu usage?
NO! As I said, please check your scripts and components! There is an error within them I think and there is no way arround sighting all scripts!

With better hardware you will only delay the 100% cpu effect! I am sure it will happen again. I am administrating IIS since Windows 2000 release for arround 7 to 8 years now, and in most if not alll cases a bad code was the source of these types of problems.
Unfortunatly there is no way to detect the one script causing the error automatically. There is no debug tool or log that can help you.