Solved

w3wp.exe 100% cpu

Posted on 2007-12-03
16
1,075 Views
Last Modified: 2010-08-05
Have an ERP server on LAN and a IIS in DMZ

Both servers are Dual Core 1.6 GHz, 1 GB memory, no RAID

Bandwith is 10 down / 1 up

Enabled ICMP and ping the wan interface with 39 ms replys

I'm responsible for the servers OS and firewall and network. The IIS is hosting an ASP.net solution. the IIS queries the backend ERP (SQL 2005). I haven't anything to do with the Website and don't have any knowledge about it.

Yesterday we had serious problems since the w3wp.exe process took 100% cpu. There were 60 users accessing the web site. I killed the process but it went to 100% again in few seconds. Restarted the machine but it went to 100% again.

Then I changed the settings for application pools so the w3wp.exe is stopped when it reach 50% cpu. It seemed to help keeping the site stable.

Question 1: I would like to know if this application pool change has any impact for the users accessing the site? Are they experiencing any drop-outs, missing transactions or anything?

Question 2: Why did the w3wp.exe process go to 100%? (Is the hardware to small? or is it bad code in the application? or stupid configured queries against the SQL server?

Question 3: I think 39 ms reply on ping is slow (but not sure). can a slow reply affect the w3wp.exe process?


Please help
0
Comment
Question by:TANGLAD
  • 8
  • 6
  • 2
16 Comments
 
LVL 7

Expert Comment

by:bungHoc
ID: 20394393
Question 1: I guess yes, especially if you want to keep overheads as low as possible and have a bunch of sites under application pool.

Question 2: Check this article out http://support.microsoft.com/kb/900243
0
 
LVL 1

Author Comment

by:TANGLAD
ID: 20394412
Thanks. Your answer to question 2: the Microsoft article talks about Exchange but that is not on the servers.
0
 
LVL 11

Expert Comment

by:Sven
ID: 20394441
You should check the scripts you edited the last days before the w3wp.exe began to hang up! There seems to be a script that is doing an endless loop for example. Or maybe you are not closing and/or releasing an object correctly? Check your scripts in detail at first!!

Could be also an third party component (activex dll) you are using! Maybe you implemented such a component?

If you limit the cpu time, IIS could handle fewer request than before when reaching the limit.

You should think about expanding the RAM cause windows is using a big part of it for its services and things.

SQL 2005 is running on a different server I think. If not, you definitly need more RAM or should transfer SQL Server to another server!
0
 
LVL 1

Author Comment

by:TANGLAD
ID: 20394481
Im also wondering why only 60 users at the samt time can kill the cpu?
0
 
LVL 11

Accepted Solution

by:
Sven earned 500 total points
ID: 20394491
@TANGLAD:

You can hang up the iis process with only one user calling a endless loop, at least for a while. If there are more users calling a script with an endless loop the process could hang up as in your case.

IIS is very sensitive for bad written code or non closed connections, objects and so on.
0
 
LVL 11

Expert Comment

by:Sven
ID: 20394509
I recently hung up one application and iis 6.0 with classic asp with a third party script,cause the author did not destroy all objects (recordsets and connections)! And this application has only up to 10 concurrent sessions ;)
0
 
LVL 1

Author Comment

by:TANGLAD
ID: 20394512
OK thanks. I don't know if there are any endless loop. How can i find out. The site seems to work ok when only a few users are on. What actually happens when a user user calls a script with an endless loop? Will the user experiense anything? any errors to look for in logfiles?
0
 
LVL 1

Author Comment

by:TANGLAD
ID: 20394528
Can we see something out of this part of a logfile?

12/02/2007 09:58:10 L-8 Critical - Error Global.Application_Error 0 Unhanded exception
Root-1-128410494018906250,
 Message: Exception of type System.Web.HttpException was thrown.
Type : System.Web.HttpException
Source : System.Web
Stack : at System.Web.HttpAsyncResult.End()
at System.Web.UI.Page.AspCompatEndProcessRequest(IAsyncResult result)
at ASP.main_aspx.EndProcessRequest(IAsyncResult ar) in c:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files\root\159ca395\f2cf4e5e\haubxu5j.0.cs:line 84
at System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)
Target : System.Object End()
------- Inner Exception ------
   Message: Exception of type System.Web.HttpUnhandledException was thrown.
   Type : System.Web.HttpUnhandledException
   Source : System.Web
   Stack : at System.Web.UI.Page.HandleError(Exception e)
at System.Web.UI.Page.ProcessRequestMain()
at System.Web.UI.Page.ProcessRequest()
at System.Web.Util.AspCompatApplicationStep.System.Web.HttpApplication+IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   Target : Boolean HandleError(System.Exception)
------- Inner Exception ------
      Message: Object reference not set to an instance of an object.
      Type : System.NullReferenceException
      Source : Zpider.eShop.Web.Business
      Stack : at Zpider.eShop.Web.Business.BusinessServer._GetAgreedPrice(String articleNo, Double quantity, Boolean bonusValue, Int32 customerNo, String zipCode)
at Zpider.eShop.Web.Business.BusinessServer.GetAgreedPrice(String articleNo, Double quantity, Boolean bonusValue)
at Zpider.eShop.Web.Business.BusinessArticleDataClass._FillPriceDataTable(DataRow dsRow)
at Zpider.eShop.Web.Business.BusinessArticleDataClass._FillResultData()
at Zpider.eShop.Web.Business.BusinessBaseDataClass.bdFetchNext()
at Zpider.eShop.Web.Base.ZpiderScriptProvider._TagForeachClose(ClosingTagItem item, IDictionary hAttributes, ScriptRunContext runContext)
at Zpider.eShop.Web.Base.ZpiderScriptProvider.ProcessClosingTag(ClosingTagItem item, IDictionary hAttributes, ScriptRunContext runContext)
at Zpider.Sys.Lib.Template.ScriptProcessor._DoExecuteScript(ParsedScript script, ScriptRunContext runctx)
at Zpider.Sys.Lib.Template.ScriptProcessor.ExecuteScript(ParsedScript script, String& outText, IDictionary hProperties)
at Zpider.eShop.Web.Base.Main.ProcessScript(Module module, IDictionary props)
at Zpider.eShop.Web.Base.Main.ProcessScript(String page)
at Zpider.eShop.Web.eShop.MainForm.Page_Load(Object sender, EventArgs e)
at System.Web.UI.Control.OnLoad(EventArgs e)
at System.Web.UI.Control.LoadRecursive()
at System.Web.UI.Page.ProcessRequestMain()
      Target : Zpider.eShop.Lib.ArticlePrice _GetAgreedPrice(System.String, Double, Boolean, Int32, System.String)
 
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 
LVL 11

Expert Comment

by:Sven
ID: 20394531
That is the problem! You can not detect any endless loop or non closed object within any log! You have to look at your code manually! Its a annoying to check the whole code, but in most cases it is the only way to fix this.

What happens is that IIS can not handle more than a number of requests at the same time. And if you are calling an endless loop the script "hangs" a least for the time you specified in script timeout. While this happens IIS can not server other requests. Or if the number for max. requests at a time is calling an endless loop.

You could raise the number of concurrent requests, but it would not help because this will consume mor ressources and after a while maybe these extra requests are running on the same error.

I would try to check my code and see if there is anything I can fix. Look for all "do...while", "do...until", "for...next" and check if your objects are closed and destroyed. Do not rely on garbage collector in ASP(.NET)! ;)

0
 
LVL 7

Expert Comment

by:bungHoc
ID: 20394539
Ouch.. got out of my meeting and Sonic already covered the thing here  :D
0
 
LVL 11

Expert Comment

by:Sven
ID: 20394557
"Object reference not set to an instance of an object."

Seems like you are calling a method or property of an object that is not created or allready destroyed (think it's Zpider.eShop.Web.Business).

But this error should not result in 100% cpu of w3wp.exe! It's worth a try to fix this error first and see what happens. Maybe there is more happening in background related to that error.
0
 
LVL 1

Author Comment

by:TANGLAD
ID: 20409977
I have looked in the IIS log and can count the users that way. But when the CPU went 100% and a lot of users couldn't load the page, would the connections still be logged in the IIS log or do we have no chance to tell how many users was trying?
0
 
LVL 11

Expert Comment

by:Sven
ID: 20410001
Good question. Normally the requests are recognized, but not processed. If there is a log entry I never looked for.

I do not think there will be a log entry on every request, cause the log entires will report success or failure. If the requests are timing out there will be a log entry. If the requests will result to a 403 or whatever error status, they wil be logged, but if they are not processed correctly there will be no entry in log.

But this can be wrong! I do not know exactly.
0
 
LVL 1

Author Comment

by:TANGLAD
ID: 20417990
Would it help adding more hardware?

Both servers are IBM System x3400 7975 - Dual Core Xeon 5110 1.6 GHz - 1 GB Memory - Single HDD SATA 300 7.200 rpm

Would another processor avoid the 100% cpu usage?
0
 
LVL 11

Expert Comment

by:Sven
ID: 20418029
NO! As I said, please check your scripts and components! There is an error within them I think and there is no way arround sighting all scripts!

With better hardware you will only delay the 100% cpu effect! I am sure it will happen again. I am administrating IIS since Windows 2000 release for arround 7 to 8 years now, and in most if not alll cases a bad code was the source of these types of problems.
0
 
LVL 11

Expert Comment

by:Sven
ID: 20418036
Unfortunatly there is no way to detect the one script causing the error automatically. There is no debug tool or log that can help you.
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

Foreword (May 2015) This web page has appeared at Google.  It's definitely worth considering! https://www.google.com/about/careers/students/guide-to-technical-development.html How to Know You are Making a Difference at EE In August, 2013, one …
If you don't have the right permissions set for your WordPress location in IIS, you won't be able to perform automatic updates. Here's how to fix the problem.
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to selectively show certain fields based on user input using rules to gather relevant information and data from your forms. The rules feature provides you with an opportunity…
Learn how to set-up custom confirmation messages to users who complete your Wufoo form. Include inputs from fields in your form, webpage redirects, and more with Wufoo’s confirmation options.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

24 Experts available now in Live!

Get 1:1 Help Now