Solved

Random "socket write error" in ColdFusion

Posted on 2013-10-29
6
933 Views
Last Modified: 2013-12-02
We have a production web application running on multiple servers using IIS 6 (Windows 2003 standard edition). We have web pages using coldFusion version 9.01 Standard Edition , and we started to see random errors in the coldfusion-event-xxx.log pages. the error is something like this:

10/28 13:32:26 error Error while reading header HTTPREFERER
java.net.SocketException: Software caused connection abort: socket write error
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

NOTE: the header can be anything, but the stack always starts like this.

most times you see a handful of such errors per day, and there is no side effect, but occasionally a server will show a stream of such errors, and it will become largely unresponsive, with queued requests.

a reset of ColdFusion service along with IISRESET fixes the server.

any insight how to go about fixing it?

the CFM pages are making a lot to CFHTTP calls, usually to pages on the local server written in .NET. not sure if this the underlying reason for this problem, but we have increased the number of such calls dramatically i the last year as more and more pages are written in .NET and are accessed from the legacy CFM pages.
0
Comment
Question by:structuredweb
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 2
6 Comments
 
LVL 27

Expert Comment

by:dpearson
ID: 39610618
I don't know what's causing the specific issue here between IIS and ColdFusion, but that socket write error in Java almost always occurs because the socket has already been closed by the time you go to write to it.

So I'd look at possible causes for the socket to be terminated quickly.  That could be due to network issues or it can be other problems like hitting a limit on the maximum number of open sockets (so you go to create a new socket, it fails and then you get this write error when the code tries to use the socket).

Sorry I can't be more help but I hope that at least helps fill in some of the picture.

Doug
0
 
LVL 1

Author Comment

by:structuredweb
ID: 39611550
dpearson,

thanks for your reply.

How can I see, and change if needed, the maximum number of open sockets? is there anywhere i would see that this limit has been reached?

I do not think this is a net work issue because:

Its a very good and fast network that is tested regularly.

Other pages, non java or CFM, work fine as this is happening.

The one variable which has changed significantly over the last year is the number of CFHTTP calls we make from the CFM pages.
0
 
LVL 27

Expert Comment

by:dpearson
ID: 39613115
I know for Linux the max number of open sockets is controlled by the max number of file handles allowed on the system - which in turn is set in:
/etc/security/limits.conf

I don't know how Windows 2003 server handles the limits, but this article suggests some possible approaches to raising the file handle limit:
http://social.technet.microsoft.com/Forums/windowsserver/en-US/1e8bd3c7-0355-411e-ae40-60e270d73f91/file-handlers-on-windows-server-2003

This is the sort of thing that you can run into when under heavy load - since both the number of open files and the number of open sockets will both tend to climb in that case and perhaps hit a limit.

We've certainly routinely had to raise this on Linux servers.

Doug
0
Three Reasons Why Backup is Strategic

Backup is strategic to your business because your data is strategic to your business. Without backup, your business will fail. This white paper explains why it is vital for you to design and immediately execute a backup strategy to protect 100 percent of your data.

 
LVL 1

Author Comment

by:structuredweb
ID: 39614615
Doug, apparently in Windows Server 2003 there is a limit on the # of open sockets ,and it is 5000

http://smallvoid.com/article/winnt-tcpip-max-limit.html

http://technet.microsoft.com/en-us/library/cc739819(v=ws.10).aspx

these articles basically saying you can increase the number of allowed open sockets, and also decrease the timeout before sockets are finally closed.

what i'm doing now is i have a schedule which looks for these errors every 10 minutes and run a netstat command as well ,so next time i get this situation hopefully i will have data to show me how many open sockets existed in the system at the time of the errors. I'd rather not touch these registry settings before i know for a fact this is the issue.
0
 
LVL 1

Accepted Solution

by:
structuredweb earned 0 total points
ID: 39681583
OK, i managed to fix the problem, not at the TCP level, I still get these phantom errors, but by increasing CF concurrency I eliminated these bottlenecks where requests were stuck in thre queue and CF was not responding. this is what I did

1) go to <CF-install-root>\runtime\lib\wsconfig\1
2) Make a copy of jrun_iis6_wildcard.ini
3) Edit jrun_iis6_wildcard.ini, uncomment "maxworkerthreads" and set it to 50
4) go to the admin interface
5) click on Request Tuning
6) Change "Maximum number of simultaneous Template requests" to 40
0
 
LVL 1

Author Closing Comment

by:structuredweb
ID: 39689447
well, it worked. still don't know why im seeing the random socket errors, but CF is working just fine now
0

Featured Post

DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
In this post we will learn different types of Android Layout and some basics of an Android App.
Viewers will learn about the different types of variables in Java and how to declare them. Decide the type of variable desired: Put the keyword corresponding to the type of variable in front of the variable name: Use the equal sign to assign a v…
Viewers will learn one way to get user input in Java. Introduce the Scanner object: Declare the variable that stores the user input: An example prompting the user for input: Methods you need to invoke in order to properly get  user input:
Suggested Courses

732 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question