Web connections getting reset IIS 8.0 on Windows 2012 - App Pool needed resetting to resolve this
Running IIS 8.0 on Windows 2012 - every now and again, we seem to be unable to login to a hosted web site - the fix is to re-cycle the App Pool for the web site. I have a monitor (SNMP ) looking at the nuber of TCP connections and in the last occurance it was at 2K connections so do not think it is TCP conenctions on the server.
Has this been seen or any configuration settings possible ? I have looked at chaning the number of TCP conenctions but this is not exceeding the current value. This may run for 8 hours without any issues then, any new connections/logins do not work until we re-cylce the app pool.
Anu suggestions welcome.
Has this been seen or any configuration settings possible ? I have looked at chaning the number of TCP conenctions but this is not exceeding the current value. This may run for 8 hours without any issues then, any new connections/logins do not work until we re-cylce the app pool.
Anu suggestions welcome.
ASKER
Advanced settings for the pool :
General
.Net Framework v4.0, Enable 32Bit Apps, Managed Pipeline mode : Enabled, Queue length :5000
STart Auto : True, Start mode : Always Running
CPU
Limit (1.100 of%) :0, Limit Action:0 , Limit Interval :5 , processor affinity Enabled : False
Processor affinity Mask : 4294967295 , Same for Processort Affinity Mask (64 bit)
Process Orphaning:
Enabled :False
Rapid-Fail Protection
"Service Unavailable" response : HttpLevel
Enabled :False, Failure Interval : 5, Max Failures :5
Recycling
Disable Overlapping recycle : False, Disable recycling for Config: False
general Recycle Event Log entr
Private memory : 0
Regular Time Interval :1740
Entry in the event log :-
System.Web.HttpUnhandledEx
I am pretty sure that all looked OK but I wouldnt swear on it. Will look next time this happens. I think that probably is Ok as we have other web sites that seem OK as the time ...
Hope the above helps in any way....
General
.Net Framework v4.0, Enable 32Bit Apps, Managed Pipeline mode : Enabled, Queue length :5000
STart Auto : True, Start mode : Always Running
CPU
Limit (1.100 of%) :0, Limit Action:0 , Limit Interval :5 , processor affinity Enabled : False
Processor affinity Mask : 4294967295 , Same for Processort Affinity Mask (64 bit)
Process Orphaning:
Enabled :False
Rapid-Fail Protection
"Service Unavailable" response : HttpLevel
Enabled :False, Failure Interval : 5, Max Failures :5
Recycling
Disable Overlapping recycle : False, Disable recycling for Config: False
general Recycle Event Log entr
Private memory : 0
Regular Time Interval :1740
Entry in the event log :-
System.Web.HttpUnhandledEx
I am pretty sure that all looked OK but I wouldnt swear on it. Will look next time this happens. I think that probably is Ok as we have other web sites that seem OK as the time ...
Hope the above helps in any way....
It appears that you ran out of ephemeral ports due to something making connections and not letting the connection go.
Link: https://blogs.msdn.microsoft.com/sql_protocols/2009/03/09/understanding-the-error-an-operation-on-a-socket-could-not-be-performed-because-the-system-lacked-sufficient-buffer-space-or-because-a-queue-was-full/
Since I'd venture a guess that your running Server 2012, the OS is 64bit and the /3BG switch does not matter... You need to make more ephemeral ports available.
Link: http://support.microsoft.com/kb/196271
After you do this, I would try to track down how a web app is exhausting TCP ports.
Getting a look into your http logs would be helpful.
Dan
Link: https://blogs.msdn.microsoft.com/sql_protocols/2009/03/09/understanding-the-error-an-operation-on-a-socket-could-not-be-performed-because-the-system-lacked-sufficient-buffer-space-or-because-a-queue-was-full/
Since I'd venture a guess that your running Server 2012, the OS is 64bit and the /3BG switch does not matter... You need to make more ephemeral ports available.
Link: http://support.microsoft.com/kb/196271
After you do this, I would try to track down how a web app is exhausting TCP ports.
Getting a look into your http logs would be helpful.
Dan
ASKER
I have an SNMP monitoring the OID (1.3.6.1.2.1.6.9) and this just showed 2.5k TCP connections at the time .
I have TCP ports set as below but I am pretty sure it is not a TCP ports issue on the IIS server
C:\Users\supps>netsh int ipv4 show dynamicport tcp
Protocol tcp Dynamic Port Range
--------------------------
Start Port : 49152
Number of Ports : 16384
We have Logging turned on yes but I have been through it and cannot see anything around the time tha app pool was re-cycled .....
I have TCP ports set as below but I am pretty sure it is not a TCP ports issue on the IIS server
C:\Users\supps>netsh int ipv4 show dynamicport tcp
Protocol tcp Dynamic Port Range
--------------------------
Start Port : 49152
Number of Ports : 16384
We have Logging turned on yes but I have been through it and cannot see anything around the time tha app pool was re-cycled .....
What your SNMP monitor is not seeing is that there are ephemeral ports that have been used but have not been released. You should increase the number of ports to 32K.
The error message that is being reported is quiet specific about the issue occurring.
I have seen this many times before with both web services as well as some high frequency ticker facilities.
Also, can you post the logs. A second pair of eyes in the logs may see something that you do not.
Dan
The error message that is being reported is quiet specific about the issue occurring.
I have seen this many times before with both web services as well as some high frequency ticker facilities.
Also, can you post the logs. A second pair of eyes in the logs may see something that you do not.
Dan
ASKER
The SNMP monitor shows a complete list of TCP ports apparently reports on "The number of TCP connections for which the current state is either ESTABLISHED or CLOSE- WAIT." Label: iso.org.dod.internet.mgmt.
I understand, I have read about ephemeral ports taking up to 4000 MAX and can take up to 4 mins to release. I am not seeing more than 2000 TCP ports from my SNMP monitor ..so not sure this is the issue .
http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html
I understand, I have read about ephemeral ports taking up to 4000 MAX and can take up to 4 mins to release. I am not seeing more than 2000 TCP ports from my SNMP monitor ..so not sure this is the issue .
http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
We have introduced - well a developer changed some code to close connections properly I can now see from my OID that there connections are getting broken down more effectivly.
Tool looks good might use it to see..
Thanks for your help Dan
Tool looks good might use it to see..
Thanks for your help Dan
Also, are there any errors or warnings in the Application Event Log? Do you have http logging enabled? If so, can you post a section of the logs from around the time the website seemed to hang?
When the site is in an apparently hung state, what does the RAM utilization on the server look like? Is there a process that is consuming a large percentage of the overall RAM? I'd look for a process named "w3wp.exe" which is a worker process for IIS.
Dan