Solved

Slow Response Times after move to New Server IIS6 to IIS7.5

Posted on 2013-06-23
22
2,115 Views
Last Modified: 2014-06-12
Hi,

I recently moved to a new server with Windows 2008, IIS7.5 and SQL Server 2008, the new server also has 24 CPUs against the previous 8 and 32GB Ram against 24. Since the move the site has seen response times drop from 3 to 5 seconds on average and in some cases on a page reading info from a stored procudre just to display it from 3 to 8 seconds.

I have run a response time checker it shows an increased WAIT time in a waterfall chart from 1 to 5.1 seconds.

I have tried as much as I can find, including changing pipelines, 32/64 bits app pool etc.

Any ideas welcome?

Cheers
Karl
0
Comment
Question by:Kbba04527
  • 14
  • 8
22 Comments
 
LVL 8

Expert Comment

by:didnthaveaname
ID: 39269732
Do you have experience with sql profiler traces? I would recommend running a profile trace to see what the sql server response times are (to see if the delay is on iis or sql server's side).  Has anything else changed ? (ie. physical to vm, more roles on the server, etc)
0
 

Author Comment

by:Kbba04527
ID: 39269757
Hi,

I don't have experience with SQL profiler, but this seems the perfect time to learn. I'll have a read up on that tomorrow.

I looked at the roles and the old server had the added roles of DNS Server and Application Server, do you think this could be an issue and if I added them could it be problematic to a live server?

Cheers
Karl
0
 
LVL 8

Expert Comment

by:didnthaveaname
ID: 39269812
I personally wouldn't put DNS on a web server because of the increase in the attack surface area. Especially with DoS if it's authoritative for your domain. In terms of performance, I don't think it would be a huge issue. My concern is mostly with sql, as it's a bit of a memory hog and could starve your other roles.  I would also check how much memory sql server is currently consuming. If it's a lot, you may want to consider capping it (capping it probably isn't a bad safeguard regardless).

Capping memory: http://msdn.microsoft.com/en-us/library/ms178067.aspx

SQL server profiler: http://msdn.microsoft.com/en-us/library/ms181091.aspx

As an aside, I should have previously mentioned that profiler traces can have severe impacts on sql performance. The more you capture, the more of an impact. And try to be as restrictive as possible with your filters.
0
 

Author Comment

by:Kbba04527
ID: 39270546
Hi,

I profiled both servers for the same query and the profiler response times to run the query were

NEW : 00:00:00.0565171
OLD : 00:00:00.0289281

Would this be enough to cause 3 seconds to go to 7/8 seconds?

Cheers
Karl
0
 

Author Comment

by:Kbba04527
ID: 39270920
I think I have made a bit of a breakthrough.

On the page that's running slow I increased it to include more records, so that response time was hitting 10 seconds then took out the code on the page that writes to the screen, this reduced the load time to half a second. It still loops the records, but doesn't write to screen.

Any ideas on why it would be so slow?

Cheers
Karl
0
 
LVL 8

Expert Comment

by:didnthaveaname
ID: 39270958
What is the current resource use looking like on ram/CPU side, out of curiosity? I imagine you have plenty of wiggle room, but just want to make sure.
0
 

Author Comment

by:Kbba04527
ID: 39270968
The CPUs aren't really moving off 1%, max 15%. I'm using 5.5GB of 32GB Ram.

I have just installed Dynamic Compression and no change.
0
 

Author Comment

by:Kbba04527
ID: 39271991
Hi,

I installed DataWizard SQL Profiler and it's showing me highest queries by duration to run and this query sits at number 1 with an average run time of 4.2 seconds, so it is SQL that's causing the delay

I'm a bit stuck now, whats the best way to work out why?

Cheers
Karl
0
 
LVL 8

Expert Comment

by:didnthaveaname
ID: 39272038
Try looking at the execution plan of the SQL query when you execute it.  Is it an ad hoc query or a stored procedure?
0
 

Author Comment

by:Kbba04527
ID: 39272091
Stored Procedure. I'll paste in info shortly
0
 

Author Comment

by:Kbba04527
ID: 39272100
"select ID,orderDate,MTYOrders,MTYPostage,FFOrders,FFPostage,FMOrders,FMPostage, MTYOrders+MTYPostage+FFOrders+FFPostage+FMOrders+FMPostage AS TOTAL,[UPDATE],[Note] from viewReporting
where orderdate >= @date_from
and orderdate <= @date_to
order by ID desc"		

Open in new window

0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 

Author Comment

by:Kbba04527
ID: 39272167
On a side note, my site response time as a whole has dropped between 1-2 seconds as all pages feature some sql.
0
 
LVL 8

Expert Comment

by:didnthaveaname
ID: 39272306
Is your old server still up and running, by chance?  I was hoping that you might be able to take a look at the execution plans in both environments to see if there is anything different in the new vs. old. And how did you migrate the database to the new server?
0
 

Author Comment

by:Kbba04527
ID: 39272409
The old server is still up and running. To migrate I did a backup and restore. I'll look over the execution plan now.
0
 

Author Comment

by:Kbba04527
ID: 39272542
Old Server
Old Server
New Server
New Server
The old one is the faster one that's moaning.
0
 
LVL 8

Expert Comment

by:didnthaveaname
ID: 39272597
That's really confounding...  Especially with the missing index on the old server...  You may want to support the query with an index (i'd start with a nonclustered index on orderdate and then you can look at including some of the other columns - unfortunately, indexing is a bit of an art and I am not as practiced as I'd like to be before I start definitively spouting out indexes) to try and avoid the table scan, however the execution plan is showing that you're comparing apples to apples, so...  The only thing I can think of is either: you're bottlenecked on the hardware in some way or there is a lot more data in the new server's viewReport table.  A table scan will touch every row of a table regardless of whether or not it qualifies.
0
 

Author Comment

by:Kbba04527
ID: 39272627
I have added an Index to the new one on ID and Order Date to no avail, the funny thing is if you run the query against the stored procedure and say for example 01/01/2012 to 01/05/2013 it takes about 10 seconds. If you go into the code and remove just the part that writes to the screen and leave the loop running through it, it runs in about .5 seconds.

The codes seems slower site wide, but only hugely on pages which write a lot to the screen.
0
 
LVL 8

Expert Comment

by:didnthaveaname
ID: 39281573
Ugh.  I was honestly hoping it was going to be a SQL issue, but I think it may be something with IIS, which I am freely going to admit I am not as strong on.  I'll poke around in hopes of finding something, but hopefully someone else may have some new ideas.  Is this new server a virtual server or no?
0
 

Author Comment

by:Kbba04527
ID: 39281598
Hi,

Cheers for all your help.

This is not a Virtual Server.

Cheers
Karl
0
 
LVL 8

Accepted Solution

by:
didnthaveaname earned 500 total points
ID: 39281712
You may want to look into this: http://www.iislogs.com/steveschofield/troubleshooting-iis-7-network-performance-issues-and-tcp-chimney-offload-receive-side-scaling-and-network-direct-memory-access.  I've seen a lot of articles talking about this feature causing a lot of issues, and the default behavior changed to be enabled in 2008 and it would have an impact network traffic too.

Edit: Also, and I hate using a competitor forum to answer an answer here, but it is another thing to look at (as he seemed to be having the same obscure issue you are with the exact same circumstances): http://stackoverflow.com/questions/12474880/iis-7-5-on-windows-server-2008-r2-slow (he mentions in a followup that it was ISAPI filters)
0
 

Author Comment

by:Kbba04527
ID: 39281767
Hi,

That last link, I tested the code in asp.net vb and without sessions it was 2 seconds, with it 12 seconds - hopefully a breakthrough.

I now need to look at ISAPI Filters. Will keep you posted.

Cheers
Karl
0
 

Author Comment

by:Kbba04527
ID: 39281843
Hi,

I removed all the ISAPI filters to no avail and then found this - http://serverfault.com/questions/249262/why-do-web-sites-in-iis7-7-5-have-asp-net-isapi-filters-enabled-by-default - so I popped them back on.

But it gives me a good starting point to look at the speed issue with the session test.

Cheers
Karl
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Sometimes drives fill up and we don't know why.  If you don't understand the best way to use the tools available, you may end up being stumped as to why your drive says it's not full when you have no space left!  Here's how you can find out...
New Windows 7 Installations take days for Windows-Updates to show up and install. This can easily be fixed. I have finally decided to write an article because this seems to get asked several times a day lately. This Article and the Links apply to…
To efficiently enable the rotation of USB drives for backups, storage pools need to be created. This way no matter which USB drive is installed, the backups will successfully write without any administrative intervention. Multiple USB devices need t…
This tutorial will show how to configure a single USB drive with a separate folder for each day of the week. This will allow each of the backups to be kept separate preventing the previous day’s backup from being overwritten. The USB drive must be s…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now