Solved

Random "socket write error" in ColdFusion

Posted on 2013-10-29
6
890 Views
Last Modified: 2013-12-02
We have a production web application running on multiple servers using IIS 6 (Windows 2003 standard edition). We have web pages using coldFusion version 9.01 Standard Edition , and we started to see random errors in the coldfusion-event-xxx.log pages. the error is something like this:

10/28 13:32:26 error Error while reading header HTTPREFERER
java.net.SocketException: Software caused connection abort: socket write error
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

NOTE: the header can be anything, but the stack always starts like this.

most times you see a handful of such errors per day, and there is no side effect, but occasionally a server will show a stream of such errors, and it will become largely unresponsive, with queued requests.

a reset of ColdFusion service along with IISRESET fixes the server.

any insight how to go about fixing it?

the CFM pages are making a lot to CFHTTP calls, usually to pages on the local server written in .NET. not sure if this the underlying reason for this problem, but we have increased the number of such calls dramatically i the last year as more and more pages are written in .NET and are accessed from the legacy CFM pages.
0
Comment
Question by:structuredweb
  • 4
  • 2
6 Comments
 
LVL 26

Expert Comment

by:dpearson
Comment Utility
I don't know what's causing the specific issue here between IIS and ColdFusion, but that socket write error in Java almost always occurs because the socket has already been closed by the time you go to write to it.

So I'd look at possible causes for the socket to be terminated quickly.  That could be due to network issues or it can be other problems like hitting a limit on the maximum number of open sockets (so you go to create a new socket, it fails and then you get this write error when the code tries to use the socket).

Sorry I can't be more help but I hope that at least helps fill in some of the picture.

Doug
0
 
LVL 1

Author Comment

by:structuredweb
Comment Utility
dpearson,

thanks for your reply.

How can I see, and change if needed, the maximum number of open sockets? is there anywhere i would see that this limit has been reached?

I do not think this is a net work issue because:

Its a very good and fast network that is tested regularly.

Other pages, non java or CFM, work fine as this is happening.

The one variable which has changed significantly over the last year is the number of CFHTTP calls we make from the CFM pages.
0
 
LVL 26

Expert Comment

by:dpearson
Comment Utility
I know for Linux the max number of open sockets is controlled by the max number of file handles allowed on the system - which in turn is set in:
/etc/security/limits.conf

I don't know how Windows 2003 server handles the limits, but this article suggests some possible approaches to raising the file handle limit:
http://social.technet.microsoft.com/Forums/windowsserver/en-US/1e8bd3c7-0355-411e-ae40-60e270d73f91/file-handlers-on-windows-server-2003

This is the sort of thing that you can run into when under heavy load - since both the number of open files and the number of open sockets will both tend to climb in that case and perhaps hit a limit.

We've certainly routinely had to raise this on Linux servers.

Doug
0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 1

Author Comment

by:structuredweb
Comment Utility
Doug, apparently in Windows Server 2003 there is a limit on the # of open sockets ,and it is 5000

http://smallvoid.com/article/winnt-tcpip-max-limit.html

http://technet.microsoft.com/en-us/library/cc739819(v=ws.10).aspx

these articles basically saying you can increase the number of allowed open sockets, and also decrease the timeout before sockets are finally closed.

what i'm doing now is i have a schedule which looks for these errors every 10 minutes and run a netstat command as well ,so next time i get this situation hopefully i will have data to show me how many open sockets existed in the system at the time of the errors. I'd rather not touch these registry settings before i know for a fact this is the issue.
0
 
LVL 1

Accepted Solution

by:
structuredweb earned 0 total points
Comment Utility
OK, i managed to fix the problem, not at the TCP level, I still get these phantom errors, but by increasing CF concurrency I eliminated these bottlenecks where requests were stuck in thre queue and CF was not responding. this is what I did

1) go to <CF-install-root>\runtime\lib\wsconfig\1
2) Make a copy of jrun_iis6_wildcard.ini
3) Edit jrun_iis6_wildcard.ini, uncomment "maxworkerthreads" and set it to 50
4) go to the admin interface
5) click on Request Tuning
6) Change "Maximum number of simultaneous Template requests" to 40
0
 
LVL 1

Author Closing Comment

by:structuredweb
Comment Utility
well, it worked. still don't know why im seeing the random socket errors, but CF is working just fine now
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Hi, Even though I have created this Tutorial on My personal Blog, Some people might not able to find my website, So here i am posting it again Today, from the topic it is very clear that i will be showing you here the very basic usage of how we …
Lync server 2013 Backup Service Error ID 4049 – After File Share Migration
Viewers learn about the third conditional statement “else if” and use it in an example program. Then additional information about conditional statements is provided, covering the topic thoroughly. Viewers learn about the third conditional statement …
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now