IIS7 Performance Issue

Recently migrated from windows2003 / iis6 to 2008 (32bit) iis7.  Went from a single server to two servers in a webfarm.  Much bigger/faster database server.  Prior to the upgrade, the single server environment was bottlenecked by database capacity.  We'd hit a certain transaction threshold and the sites would bog down, with the datbase was pegged at 95%.   So that was the issue...so I thought.  Moving to the new server environment, the db is crusing along at 20% under the same load.  However, I still get occasionally bogged down on the site performance (page loads over 30 seconds, etc) for pages that normaly load in 1 second or better.  Under this condition, the web server is barely using the cpu  (avg 15%, spikes to 40 or 50%), using only 2gig of 4 avilable on memory.   Now the load I'm talking about is roughly 100k page views (asp.net database driven pages with lots fo images) a day...yes a day.  Traffic is spread faily evenly with no major peak hour, but slow hours in the early am hours.

We're runing asp.net 2.0 apps that have legacy dependency on 1.1 modules, hence we need to run our app pools in classic mode (6.0) equiv.  Using perfmon, I can see that when active requests hit above 30 or so, then they begin to to really rise rapidly and the request queue count goes from 0 to 40 or 50.  DUring this time, the pages are being served very slowing and of course, some time out.

I can consistently recreate the situation by taking one of the servers out of hte load balancer and throwing all requests a one server. Within a couple of minutes the active requests count goes way up and the site performs badly.  Again, no bottleneck in network, database or memory.  It looks like a thread pool issue.  I've done some minor tweeks to the app pool configuration.  But for the most part, no difference.

 My question is, how can get IIS to utilize the cpu resources available (the web server barely breaks a sweat)?   Also, what performance counters would most likely reveal where the bottle neck is?  I am having failed request tracking installed on the servers to see at that reveals anything.  The pages that end up sitting there in the worker process queue are the same pages that normally get served up in 1 second.

Thanks for any help.
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

First off what apps cause this issue?
Any apps? Have you ispolated your app into seperate app pools?

IIS will take any resources that are available by default.

What are your app poll settings? Anything in the http.sys error logs?

Also if you really know to know what is happening look at Debug Diag. I suspect the app is doing something it shouldn't.
And this thread might help:

Also I'm suggesting that this post be in the asp.net zone also.
Acronis True Image 2019 just released!

Create a reliable backup. Make sure you always have dependable copies of your data so you can restore your entire system or individual files.

MarkAtPowerchordAuthor Commented:
Our apps are separated into app pools by logical groups (apps replated to a client; apps running in 1.1 framework, etc).
The app that causes the problem is a B2C application with particularly high volume.  When it does get into the state where it is accumulating active requests (into the 100's)  all the apps regardless of app pool become effected in terms of performance.  

I know have the Failed Request tracing service available and will make use of that to gather more info.

Ummh try disabling teh TCP/IP chimney stack offloading if it is enabled

I believe IIS7, win 7, IE7, and Win 2008 can all be configured for a 64 bit as well as for a 32 bit operating environment. Uprading to a 64 bit operating environment might increase your performance considerably.   See this link for a FAQ  on the 64 bit vs 32 bit issue.  

Some questions,

1. Are you storing the images in the DB as VarBinary or in the File system

2. Have you tried GZIPing the response ? [If no, then try to GZIP it]

3. Do you use much of javascript and CSS ? [If no, try minifying the JS and CSS]

4. Use a tool like fiddler and try to analyze the HTTP calls made per request

MarkAtPowerchordAuthor Commented:
So far, nothing helpful.  This is clearly a thread blocking issue.  What is not clear is what IS blocking the thread.  I'll be doing some additional dianostics today.
YOU mihght have to get heavy with DebugDiag & procmon and look for any causes there.
MarkAtPowerchordAuthor Commented:
Found solution, application was using System.Drawing namespace to open images in order to determine there height and width.  GDI functions in System.Drawing are not supported in ASP.net.  This was causing the file locking.   Refactoring application.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft IIS Web Server

From novice to tech pro — start learning today.