• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1544
  • Last Modified:

Network error IOException


Hello there,

I have an Java application deployed on windows 20083 R2 machine connecting to SQL server on the same server.
 
once a week, we start getting this error.
 
Caused by: java.sql.SQLException: I/O Error: Connection reset
Caused by: java.net.SocketException: Connection reset
java.sql.SQLException: Invalid state, the Connection object is closed.

then i have restart the SQL server instance and then all works fine for the next couple of days,weeks. then the same issue. what could it be and how to solve this issue for good.
0
zolf
Asked:
zolf
  • 11
  • 5
  • 3
  • +3
6 Solutions
 
for_yanCommented:
Perhpas soemwhere your application closes connection and then tries to use it again
0
 
Gurvinder Pal SinghCommented:
Most probably you are using the connection object somewhere after closing it.

please post the stack trace and check on which line error has occurred, and see if you have closed the connection before using it.
0
 
zolfAuthor Commented:

>>Perhpas soemwhere your application closes connection and then tries to use it again
i doubt this is the casue since,this does not happen on one particular frame or so. by the way i developed the application using Java Swing,hibernate
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
stachenovCommented:
Perhaps your application doesn't close old connections before opening new ones, so you're hitting some sort of maximum connections limit when you're trying to open yet another connection, and the SQL server just closes it. That would explain why restarting SQL server helps. You could try checking current connections with nestat command and see if their number increase.

If this turns out to be the reason, you need to check every place where you don't need some connection anymore, and don't forget to put the close() call into a finally block so it gets called even in case of exception.
0
 
zolfAuthor Commented:

you see,once a user gets this connection error,then none of my users can login to the application,untill i restart the mssql instance or restart the server.i have around 50 users who are using this application.some via RDP and some local.
0
 
stachenovCommented:
>>you see,once a user gets this connection error,then none of my users can login to the application

That's exactly why I thought about connection limit. You know, even if the connection object goes out of scope, there is no guarantee that it will be garbage collected and finalized anytime soon. This means that "garbage" connections are probably still active from the MSSQL point of view. This is a kind of resource leak which garbage collection mechanism doesn't solve. And if your number of "garbage" connections hits the server limit, it won't be able to open new connections, so new users can't login. Restarting appilcation on any side closes all connections, so you're able to start connecting again. That's the only explanation that I could think right away, but I think it makes sense.
0
 
zolfAuthor Commented:

please see the netstat file of my server
12.txt
0
 
zolfAuthor Commented:

>>stachenov:
Connection management is done via hibernate. It closes the connections automatically.
0
 
zolfAuthor Commented:

We use hibernate which uses own connection pool. Hibernate handles all the connection operation. We only open and close sessions in the app. When user closes app all the DB connections should be terminated also
0
 
Gurvinder Pal SinghCommented:
You need to show the stack trace where the error has occurred.

Also please show the code and point to line where the error has occurred
0
 
stachenovCommented:
You should check its dynamics. Right now there's about 50 MSSQL connections. That could be a lot or just a few depending on the specifics of your application and the numer of users. However, if you see that the number of such connections is constantly increasing although the number of user sessions doesn't increase that much (or, ideally, stays the same or incresing), then you've got a problem.

For example, you could run this:
netstat | findstr ms-sql-s > netstat1.txt

Then, after some users have finished their sessions, run it again:
netstat | findstr ms-sql-s > netstat2.txt

Then compare two files, for example using fc:
fc netstat1.txt netstat2.txt

If you see that netstat2.txt contains some new connections, but none of the connections are closed (are only in netstat1.txt or have changed state), then it's a problem.
0
 
stachenovCommented:
Well, I have never used Hibernate, but there still should be a way to terminate connections, albeit implicitly. You say "open and close sessions in the app" - are you sure that sessions are properly closed?
0
 
zolfAuthor Commented:

>>If you see that netstat2.txt contains some new connections, but none of the connections are closed (are only in netstat1.txt or have changed state), then it's a problem.

can you please explain more,i did not understand exactly what you meant
0
 
zolfAuthor Commented:
Caused by: java.sql.SQLException: Network error IOException: No buffer space available (maximum connections reached?): connect
      at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:410)
      at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)
      at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:184)
      at java.sql.DriverManager.getConnection(DriverManager.java:582)
      at java.sql.DriverManager.getConnection(DriverManager.java:154)
      at org.hibernate.connection.DriverManagerConnectionProvider.getConnection(DriverManagerConnectionProvider.java:133)
      at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
      ... 46 more
Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:529)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at net.sourceforge.jtds.jdbc.SharedSocket.createSocketForJDBC3(SharedSocket.java:307)
      at net.sourceforge.jtds.jdbc.SharedSocket.<init>(SharedSocket.java:257)
      at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:311)
      ... 52 more
0
 
zolfAuthor Commented:

i found this on goole which is similar issue to mine.but in my case i dont have antivirus on my server.please help

http://social.technet.microsoft.com/Forums/en-US/winserverPN/thread/157d8f12-66dc-4fb7-9014-c47e65019d2d/
0
 
CEHJCommented:
I would start looking at this firstly from a SQL Server point of view. Check stuff like

a. the maximum connections configured in the db server
b. wait timeouts in the same

Hibernate will have little influence over that and its connection pool will have to start from what's laid down at the db server
0
 
stachenovCommented:
>>can you please explain more,i did not understand exactly what you meant
Well, I don't have a MSSQL here, but here is an example using http:

1. I open a browser and go to some website.
2. I run:
netstat | findstr http > netstat1.txt

The file netstat1.txt now contains:

  TCP    92.36.85.203:51304     bw-in-f18:https        ESTABLISHED
  TCP    92.36.85.203:51307     www-level3:http        TIME_WAIT
  TCP    92.36.85.203:51308     www-level3:http        TIME_WAIT
  TCP    92.36.85.203:51312     2.22.241.55:http       TIME_WAIT
  TCP    92.36.85.203:51315     www-13-05-prn1:http    TIME_WAIT
  TCP    92.36.85.203:51316     www-13-05-prn1:http    TIME_WAIT
  TCP    92.36.85.203:51318     www-level3:http        TIME_WAIT
  TCP    92.36.85.203:51320     www-level3:http        TIME_WAIT
  TCP    92.36.85.203:51330     images:http            TIME_WAIT
  TCP    92.36.85.203:51331     images:http            TIME_WAIT
  TCP    92.36.85.203:51332     images:http            TIME_WAIT
  TCP    92.36.85.203:51334     images:http            TIME_WAIT
  TCP    92.36.85.203:51336     images:http            TIME_WAIT
  TCP    92.36.85.203:51340     2.22.241.55:http       TIME_WAIT
  TCP    92.36.85.203:51342     bw-in-f113:https       TIME_WAIT
  TCP    92.36.85.203:51344     bw-in-f113:https       TIME_WAIT
  TCP    92.36.85.203:51347     bw-in-f102:https       TIME_WAIT
  TCP    92.36.85.203:51348     bw-in-f102:https       TIME_WAIT
  TCP    92.36.85.203:51351     bw-in-f102:https       TIME_WAIT
  TCP    92.36.85.203:51352     bw-in-f102:https       TIME_WAIT
  TCP    92.36.85.203:51354     static:http            ESTABLISHED
  TCP    92.36.85.203:51356     static:http            ESTABLISHED

Only three connections are actually in the ESTABLISHED state, others are being closed.

3. After some time I run netstat again and put results in another file:
netstat | findstr http > netstat2.txt

4. Now I compare two files:
fc netstat1.txt netstat2.txt

The result is:

***** netstat1.txt
  TCP    92.36.85.203:51304     bw-in-f18:https        ESTABLISHED
  TCP    92.36.85.203:51307     www-level3:http        TIME_WAIT
  TCP    92.36.85.203:51308     www-level3:http        TIME_WAIT
  TCP    92.36.85.203:51312     2.22.241.55:http       TIME_WAIT
  TCP    92.36.85.203:51315     www-13-05-prn1:http    TIME_WAIT
  TCP    92.36.85.203:51316     www-13-05-prn1:http    TIME_WAIT
  TCP    92.36.85.203:51318     www-level3:http        TIME_WAIT
  TCP    92.36.85.203:51320     www-level3:http        TIME_WAIT
  TCP    92.36.85.203:51330     images:http            TIME_WAIT
  TCP    92.36.85.203:51331     images:http            TIME_WAIT
  TCP    92.36.85.203:51332     images:http            TIME_WAIT
  TCP    92.36.85.203:51334     images:http            TIME_WAIT
  TCP    92.36.85.203:51336     images:http            TIME_WAIT
  TCP    92.36.85.203:51340     2.22.241.55:http       TIME_WAIT
  TCP    92.36.85.203:51342     bw-in-f113:https       TIME_WAIT
  TCP    92.36.85.203:51344     bw-in-f113:https       TIME_WAIT
  TCP    92.36.85.203:51347     bw-in-f102:https       TIME_WAIT
***** NETSTAT2.TXT
  TCP    92.36.85.203:51304     bw-in-f18:https        TIME_WAIT
  TCP    92.36.85.203:51315     www-13-05-prn1:http    TIME_WAIT
  TCP    92.36.85.203:51342     bw-in-f113:https       TIME_WAIT
  TCP    92.36.85.203:51347     bw-in-f102:https       TIME_WAIT
*****

***** netstat1.txt
  TCP    92.36.85.203:51348     bw-in-f102:https       TIME_WAIT
  TCP    92.36.85.203:51351     bw-in-f102:https       TIME_WAIT
  TCP    92.36.85.203:51352     bw-in-f102:https       TIME_WAIT
  TCP    92.36.85.203:51354     static:http            ESTABLISHED
  TCP    92.36.85.203:51356     static:http            ESTABLISHED
***** NETSTAT2.TXT
  TCP    92.36.85.203:51348     bw-in-f102:https       TIME_WAIT
  TCP    92.36.85.203:51356     static:http            TIME_WAIT
  TCP    92.36.85.203:51359     bw-in-f100:http        TIME_WAIT
  TCP    92.36.85.203:51360     bw-in-f100:http        ESTABLISHED
  TCP    92.36.85.203:51363     bw-in-f18:https        ESTABLISHED
  TCP    92.36.85.203:51365     bw-in-f18:https        TIME_WAIT
*****

The lines marked with "***** netstat1.txt" are those contained only in the "netstat1.txt" file, but not in "nestat2.txt". Those marked with "***** NETSTAT2.TXT" are only in netstat2.txt. From this, I can see that connections from ports 51304 and 51356 are now being closed and the connection from port 51354 has disappeared completely. However, new connections from ports 51360 and 51363 have been established. This is all normal. If I saw instead that new (marked with *** netstat2.txt) lines with ESTABLISHED state appear, but no ESTABLISHED lines appear in the "*** netstat1.txt" parts, that would mean that connections are opened, but not closed, and that would be a bug in the browser.

By repeating this process several times, you can see connection dynamics on the network level.

To simpify the process, you could just count the number of established connections:

netstat | findstr ESTABLISHED | find /c ":ms-sql-s "

This would count the total number of established MSSQL connections (multiplied by two since netstat lists both sides for local connections). If this number constantly grows, then you're in trouble. But this method doesn't show you which connections are open and which are closed, so when the number stays the same you don't know if the same connections are in place or if some closed and some opened.

If netstat is slow, you may wish to turn off DNS lookups and specify the port instead of the service name:

netstat -n | findstr ESTABLISHED | find /c ":1433 "
0
 
CEHJCommented:
I don't use MS SQL either but the following looks useful:

http://www.informit.com/articles/article.aspx?p=1072372&seqNum=2
0
 
zolfAuthor Commented:

stachenov:

thanks for your feedback.



a. the maximum connections configured in the db server
0 =  unlimited

b. wait timeouts in the same

600 seconds
0
 
CEHJCommented:
You might also try changing the driver from jtids to the SQL Server one
0
 
zolfAuthor Commented:

ok
0
 
gordon_vt02Commented:
Check the connection pool settings for your application as well.  Yes, Hibernate should manage the open/close for you internally but the pool has to be configured.  Check to see what type of connection pool you are using and look at the options it provides.  At a minimum you should be able to set properties for the maximum number of connections, minimum number of idle connections, and a timeout delay that controls how long requesters will wait for a connection to open (when the pool is full) before giving up.  

There should also be an option to check connections before returning them and the ability to specify a validation query.  If you turn those options on, and specify a simple validation query -- something like 'SELECT 1 FROM DUAL' (Oracle-specific, not sure what MSSQL equiv would be) -- the connection pool will run that query before returning a connection object and, if the connection has timed out or otherwise been closed, it will open a new Connection to return to the client (Hibernate in your case).

Especially with your server configured for unlimited connections, this error is probably caused by stale connections in your pool and setting the validation up should help correct that.
0
 
zolfAuthor Commented:

>>Especially with your server configured for unlimited connections, this error is probably caused by stale connections in your pool and setting the validation up should help correct that.
 
should i remove the unlimited connection and set it to say 50??
0
 
gordon_vt02Commented:
That won't necessarily solve your problem, and might actually make it worse depending on how your connection pool is configured.  The problem is most likely with the configuration of the connection pool that hibernate is using.  Try setting up the connection validation and see if that fixes the issue.
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

  • 11
  • 5
  • 3
  • +3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now