Socket Error 10057 - Socket not connected when running within a thread

A database server request thread is inherited from TThread. Each one creates a socket component that it uses to communicate with the database.

This has worked well in various other threaded applications I've written, it's just now I'm running into problems. This may be because the general activity of this application is much greater (although socket activity per thread is much less), though I'm not sure.

I would have thought if this was some kind of synchronisation issue, it wouldn't have worked at all.

What I have found is that after X (random amount but usually more than 10), the socket state changes from wsConnected to wsConnecting. It appears that because the database requests are serviced so quickly Winsock is somehow becoming flooded(?) and the WSAGetLastError returns WSAEWOULDBLOCK. This is then manifesting itself at the time of SenddStr (since the state is obviously not wsConnected).

At the time of SendStr, if the state is wsConnecting I've tried waiting around a bit (calling the ProcessMessages of the Socket object) to only proceed if it becomes connected.
The above results in the application flying through the first few batches of requests and then freezing for ages and then proceeding very slowly.

Really not sure I've fixed it. Is there a better solution to the one that I've come up with?

Thanks

Steve

>> I would have thought if this was some kind of synchronisation issue, it wouldn't have worked at all.

that is not true, as synchronization problkems appear only when 2 or more threads try to simpultaneusly access one recourse. in you case, the chances of that could be 1% or 5% or something like that and this will look like randomness.

>> Really not sure I've fixed it. Is there a better solution to the one that I've come up with?

I'll need to understand a little the flow. is my understanding correct in that you are creating threads very fast and the connections are finishing very quickly?
if so, then why no use a thread pool and a conection pool? that way you can keep the connections open and re-use them (you will need a small amount of synchronization here twhen getting an open connection.
would that "fly" with your application?

I have a managing object that has the threads running in a private list. It has a separate list of pending requests and as a thread terminates, a pending request is taken from the queue and a thread created to execute it.
All this happens very quickly. Between thread creation and disposal is there maybe 1 second max.  But this process is being repeated potentially 10s thousands of times.

I had considered maintaining  a set of constantly open connections but, the Socket component is a derived version of the ICS one. Default Socketty-type behaviour hasn't been modified, but there is quite a complex initialisation process which the socket undergoes when first created which places the database in to some kind of default state which is not changed easily. Altering of the Delphi code isn't the issue, it's the database side of things which I don't even want to go near for risk of inducing error into our other applications which do not need to keep resetting the database. (we're using a MUMPS database, not SQL etc, so database handling is unique to say the least)

Regarding the threads, they absolutely do not access any outside resource. There's obviously no  interface, a simple notification synchronisation method (which does access a COM object, but I can't see this as being an issue - it is within the synchronisation stage anyway). The threads simply issue a command on the database, wait for a result and place this in a list and then terminate.

I know from experience that creating and destroying things fast and many times is bad to say the least.

what you have there is a manager, not a pool. you need a pool. basically, you need to modify the thread os that i gets to do oprations an dnot just one operation.
you have 2 options here:
- the manager adds jobs to the thread, which keeps a list of jobs and if no jobs present sleeps for 500 ms or 1 second or something like that.
- you keep the queue as it is and every thread troes to get one job from the queue and again, if the queue is empty, it sleeps for a while.

this will ease up the load on windows (because creating and destroying a thread also takes time).

next, connection pool. once the thread is create and teh socket is created, you only need to use it. so issue a command, get results, place results wherever you place them and start over over again.
there is one aspect which is not clear to me:
- the DB is beeing put in that default state every single time a soket is created and then taken out of that state every time the socket closes?
- if the DB is placed into that state only once and then for a looong while (or foreveer) it remains in that state, why is it a problem in creating the connection pool?

the only porblem you will need to take into account is while the thread runs, what are the chances of the DB getting out of that default state in which it was placed by the socket creation? if there are chances, then you will need to check before every job if the db is in the right state and place it in if not. but stil, it is a lot better (optimal) than recreating teh socket every time. that also takes time and windows resources.
I don't think you want to depend on the performance of windows layers for your application. if windows cannot create but 100 sockets per minute and that is not enough you're in deep trouble.

so lets break up the problem into smaller problems:
- thread pooling is easy. you create 30 threads at once and then all of them work. this is the simplest way. you can also create the threads only as needed, but that can be a little complicated, so I would leave it to when you have more free time
- connection pooling. this could be tricky since it depends on the DB. but it si possible.

so, is soemthinglike the attached pseud-code feasable for you?

mythread=class(tthread)
fconnection:tdbconnection;
protected
  procedure execute; override;
public
  constructor createl
  destructor destroy; override;
end;
 
procedure execute;
begin
  while not temrinated do
  try
    job:=getjob;// getjob gets a job from the queue 9shynchronized, of course)
    if job=nil then sleep(1000)
    else
      process(job);
  except
    / notify logger of error and log it to file
  end;
end;
 
constructor create;
begin
  create fdbconnection;
  inherited create(false);
end;
 
destructor destroy;
begin
  destroy db connection
end'
 
processjob will use dbconnection to send the command and retrieve the results.

                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:

Thanks for this.

In the other threaded applications, the lifetime of a thread was much longer.

In the case of this application , it does make sense to re-use the thread objects, but this is not going to be five minutes work.

I'll get back to you to let you know if it improves things...

Thanks

Steve

take your time

Hi Ciuly

Finished the implementation of the connection pool, but unfortunately it's made no difference. I'm still getting bucket loads of 10057 errors.

The sockets connect OK but at some stage, the WSAGetLastError indicates that the socket becomes blocked and the state changes to wsConnecting. Therefore when the socket tries to write, 10057 is the result. Is this indicative of Winsock not completing the last operation and therefore blocking the non-blocking socket?

To make matters worse, I'm also occasionally getting (not nearly as often as the 10057) the error "Socket already in-use". I've only ever come across this before when writing socket based service applications and have learnt to live with the error has being a result of running within a service, but are the two issues related?


Thanks

Steve

well, now that you are using thread pooling, you should not get the socket already in use error since one socket is only created once and used multiple times. if you do get it, then you probably missed some initialization code somewhere.

now, since you are using pooling, one thing should be clear:
- if the socket gets disconnected, all jobs on that thread will throw 10057.

are you seeing this behaviour? if so, do a search in your code and check if you somewhere disconnect the socket yourself. if you can't find anything like that, then try to follow the flow of the execution of a job and isolate the places where the socket could get disconnected: for example the server (DB) could disconnect the socket in some circumstances. if this is true, then before each job you will have to chheck if ythe socket is connected or not, and connect it if it's not.
(this sould also be a workaround, since if I am not wrong, you will only do one connection per job, so if the current job disconnects the socket, it will be reconnected when the next job is processed)

if you don't see that behaviour, then there is something you're not telling me or you didn't do as I instructed. because in a thread-connection setup as I explained a few days earlier, once the socket gets disconnected it will stay disconnected andal requests will throw errors.

From what I can see, the 10057 is nothing to do with the socket disconnecting - it hasn't, only its STATE changed (it doesn't fire the OnSessionDisconnect or close events). These are the lines taken from the components (OverbyteICSWsocket),. at the end of the Connect method

        iStatus := WSocket_Synchronized_connect(FHSocket, TSockAddr(sin), sizeof(sin));
        if iStatus = 0 then
            ChangeState(wsConnecting)
        else begin
            iStatus := WSocket_Synchronized_WSAGetLastError;
            if iStatus = WSAEWOULDBLOCK then
                ChangeState(wsConnecting)

iStatus is coming through as WSAEWOULDBLOCK , therefore changing the state from wsConnected to wsConnecting. At the "iStatus=....." line, the socket's state is wsConnected.

There is only one socket per thread, so closing a socket in one thread - or getting the 10057 error - shouldn't affect any others. However, once one thread has the error, the others quickly report it afterwards. I think this is a symptomatic of an underlying cause, and not a result of inter-thread behaviour.

BTW, The sockets always check that they are connected before sending anything and connect if necessary..

To start with the application works fine for a while - reusing the threads as I expect. Only after a minute or so do the first errors pop-up.

Don't know what else to try.

Regards

Steve

well, I usually do this first, but better late than never. so I checked MSDN:
"WSAEWOULDBLOCK
10035
Resource temporarily unavailable.

This error is returned from operations on nonblocking sockets that cannot be completed immediately, for example recv when no data is queued to be read from the socket. It is a nonfatal error, and the operation should be retried later. It is normal for WSAEWOULDBLOCK to be reported as the result from calling connect on a nonblocking SOCK_STREAM socket, since some time must elapse for the connection to be established."

but I am looking at what you are showing to me and it makes no sense. because I said thta you create the socket once. so you should have as many sockets as threads and not one more or less. and if they are all created and they are all connected, how can you get WSAEWOULDBLOCK since you are not connecting anymore?

did you do as I said and reuse the connection? if so, what am I missing?

I read that article as well before making the post.

There is only every one socket per thread. The socket is only told (by me ) to connect the once.

What appears to be happening is that during the socket's lifetime, its connected status changes from wsConnected to wsConnecting, perhaps multiple times. From other articles I've read the status change is a a result of Winsock being asked to do something, yet is busy and as  a result of the socket being a non-blocking  socket, indicates to the socket...

 "if you were a blocking socket, I would have blocked you, but since you're not, I'm returning this error instead."

The general advice, is to wait around before trying again.

This is easier said than done - any ideas?


P,S. Everything you advised me to do I did. Only X amount of threads are  created, each being re-used as necessary.

Only one socket per thread, each being re-initialised when re-used (the socket is not being explicitly closed - it is opened once- the first time it is used and found it not to be connected).

I think it's a speed issue - using a couple of threads and the problem takes longer to appear. There is no synchronisation issues, since I've removed all UI update so all the threads are doing are socket commands - the outside world doesn't know of their progress.

>> This is easier said than done - any ideas?

it's not that hard actually.

repat
  do something (sendstr maybe)
  if wsagetlasterror=WSAEWOULDBLOCK then
    sleep(10)
  else
    break;
until false;

I am guessing that this is a limitation/bug in the windows socket layer, though I couldn't find anything to back it up (well, there are a ton of issues with winsock, but nothing related to many socket operations happenig in  a shortwhile (or at least I couldn't find any))

I would suggest turning to blocking sockets (since you don't need the asynchroneus socket, you beeing in a thread already).

Sorry, hasn't worked

I did exactly as you suggest, but still get the error (but I don;t believe quite as often as before).

Anyway, just to comment on your example

repat
  do something (sendstr maybe)
  if wsagetlasterror=WSAEWOULDBLOCK then
    sleep(10)
  else
    break;
until false;


For the above, there's no check for the initial SendStr - if WSAEWOULDBLOCK is returned(which is what is happening) then the Sleep part is pointless.

I tried reversing this so that  the WSA... was checked for first and only outside the Repeat loop was the sendstr executed.

This did not work at all - WSAEWOULDBLOCK was always being returned so the socket just sat there...

Thanks for your help, things seem to be sorted as best as they can be.

I'm still having problems (Socket already open) re-establishing a socket connection should one become disconnected, but this is unrelated to the initial question.

Thanks again.

Steve

P.S. You need "break" just before the except in your last example...