komatt
asked on
Classic ASP website/app hosted on IIS 8 Server 2012 R2 has begun randomly hanging
Hi EE! :)
Our website has been working flawlessly until recently - we noticed that some pages began to intermittently load slowly, hanging for somewhere between 10-30 seconds, sometimes loading successfully after a considerable wait, but in many cases leading the application in the app pool to hang for a long time, and sometimes requiring a manual restart of IIS. Even restarting IIS takes a long time when it's hanging.
The website is several years old and coded in classic ASP/VB - the pages seemingly causing the bother are fairly heavy ones in regard to sever side code with lots of recordsets etc, but have NEVER caused any problems until the last couple of weeks or so.
The website app runs in 32bit compatibility mode set in the IIS apppool (We use 32bit mode as we're connecting to a 32bit mysql database on another server via win-mysql 32bit ODBC connector)
Where things stand currently:
What I have done so far without success:
Despite the actions taken above I can still recreate the problem by simply making multiple requests to one of the dynamic pages. It may take a couple of minutes but I can force its hand soon enough.
Things I've also tried:
Please can someone help. I've been on this problem for 72 hours straight and have come to the end of the line for my knowledge, and I don't know what else to do apart from ask for some help ;)
This is a live server and I need to resolve this quickly (if I don't want to lose my job lol) - I'd even be willing to compensate an IIS engineer to look at this in more detail to help me find a resolution.
Cheers,
Matt
Hardware/Software - Windows Server 2012 R2 with all updates. IIS 8 + SQL SERVER
Our website has been working flawlessly until recently - we noticed that some pages began to intermittently load slowly, hanging for somewhere between 10-30 seconds, sometimes loading successfully after a considerable wait, but in many cases leading the application in the app pool to hang for a long time, and sometimes requiring a manual restart of IIS. Even restarting IIS takes a long time when it's hanging.
The website is several years old and coded in classic ASP/VB - the pages seemingly causing the bother are fairly heavy ones in regard to sever side code with lots of recordsets etc, but have NEVER caused any problems until the last couple of weeks or so.
The website app runs in 32bit compatibility mode set in the IIS apppool (We use 32bit mode as we're connecting to a 32bit mysql database on another server via win-mysql 32bit ODBC connector)
Where things stand currently:
- No changes have been made to our classic ASP application source code.
- No applications were installed recently on the server.
- No changes made to the website conf/app in IIS
- Disk space looks fine. Disk check OK. CPU looks OK.
What I have done so far without success:
- Deleted website & app in IIS and recreated.
- Isolated website in IIS - turned all other websites and apps off.
- Uninstalled windows updates < 80 days (I have since reinstalled after discovering it made no difference)
- Performed a system restore of server state/config < 75 days
- Replaced hard disks and rebuilt array
- Uninstalled & reinstalled classic ASP component via server management console.
Despite the actions taken above I can still recreate the problem by simply making multiple requests to one of the dynamic pages. It may take a couple of minutes but I can force its hand soon enough.
Things I've also tried:
- Setup a failed trace request in IIS for the app and set the trigger to 20 seconds. I have examined some of the large xml files outputted in verbose but can't see any actual 'errors' or 'warnings'. Just some repeated references to certain parts of the source code. Just to reiterate. - this source code has been working perfectly for some years.
- Tried older, known working versions of the source code just to make sure we hadn't accidentally changed anything.
- Reinstalled 32bit mysql 5.1 driver ODBC (longshot)
Please can someone help. I've been on this problem for 72 hours straight and have come to the end of the line for my knowledge, and I don't know what else to do apart from ask for some help ;)
This is a live server and I need to resolve this quickly (if I don't want to lose my job lol) - I'd even be willing to compensate an IIS engineer to look at this in more detail to help me find a resolution.
Cheers,
Matt
Hardware/Software - Windows Server 2012 R2 with all updates. IIS 8 + SQL SERVER
ASKER
Hi,
Thanks for your response.
I have checked the database and all looks good. I am 90% convinced it's not the MySQL database. Glad we've virtually eliminated this.
Remote database server
What I have done so far on the remote database server:
The problem appears to be related to the worker process in IIS. Here are some more details:
Cheers,
Thanks for your response.
I have checked the database and all looks good. I am 90% convinced it's not the MySQL database. Glad we've virtually eliminated this.
Remote database server
- CentOS 6.9 64bit / Linux 2.6.32-696.18.7.el6.x86_64
)
- Mysql 5.1.73
What I have done so far on the remote database server:
- Rebooted Server
- Replaced disks
- Restarted MySQL database
- Repaired/optimised database & tables
- Tried 2 year old version of the same database.
- Checked my.cnf - all looks good.
The problem appears to be related to the worker process in IIS. Here are some more details:
- The delay/hang appears to be only occurring on one page, but intermittently.
- When the hang happens, I can open another browser e.g. firefox - and view the page, that is until it hangs in that browser too. So it's the sessions themselves sometimes hanging forever or timing out.
- I've isolated the website so no outside traffic/bots etc are hitting it and consuming resources while i test
- There's some fairly heavy code in terms of the number of recordsets in the failing page - but it's previously been bulletproof. *source code hasn't changed.
Cheers,
> The delay/hang appears to be only occurring on one page,
> There's some fairly heavy code in terms of the number of recordsets
The next thing I would do is break apart the page. Keep outputting some data that you are expecting and then do a response.end until you can pin point the culprit.
I would want to make sure my indexing is good on the database and you said you have done that. The next thing I want to do is make sure I am not sending more than 30 to 50 records to the browser at a time. Use getrows instead of looping through recordsets is the final thing.
You can post the code to your one page and we can take a look for anything. But if you can try and break it out as I suggested you may find the issue.
> There's some fairly heavy code in terms of the number of recordsets
The next thing I would do is break apart the page. Keep outputting some data that you are expecting and then do a response.end until you can pin point the culprit.
<%
' lots of code
response.write some_variable_that_is_from_a_recordset
response.end
' lots of code
' lots of code
%>I would want to make sure my indexing is good on the database and you said you have done that. The next thing I want to do is make sure I am not sending more than 30 to 50 records to the browser at a time. Use getrows instead of looping through recordsets is the final thing.
You can post the code to your one page and we can take a look for anything. But if you can try and break it out as I suggested you may find the issue.
This question needs an answer!
Become an EE member today
7 DAY FREE TRIALMembers can start a 7-Day Free trial then enjoy unlimited access to the platform.
View membership options
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
My guess is the amount of records you have has grown and especially if your dynamic page is accessing tables that are related, the amount of data you are outputting could be too great or the database needs to be indexed properly.
The first thing I would test is limiting the recordset's to just 10 rows of data and see if that makes a difference. Maybe even replace the database with a test database that has limited amount of records and that can give you a good clue.
Looping recordsets when there is a lot of data can also slow things. Using getrows for instance can give you back a lot of time. You said you checked disk space and cpu. But did you check read/write activity and memory?