We help IT Professionals succeed at work.

We've partnered with Certified Experts, Carl Webster and Richard Faulkner, to bring you two Citrix podcasts. Learn about 2020 trends and get answers to your biggest Citrix questions!Listen Now

x

socket send buffer returns EWOULDBLOCK

ambuli
ambuli asked
on
Medium Priority
9,101 Views
Last Modified: 2012-06-21
Hi Experts,

I have a client-server application using socket to communicate.  At times, the server disconnects the client because the write to the client returns EWOULDBLOCK. I initially thought that the client is not reading fast enough thus the TCP receive buffer is filled up.  I then added code to peek at the client's TCP buffer to see if it is getting filled up.   It never gets above 3K while the buffer size is 8K.  I used the FIONREAD to peek at the rec buffer.

My question is, is there a way to peek at the send buffer?  Or what would be the right handling for the server to do if it detects EWOULDBLOCK.  Right now, I try to re-write few times and if that fails I disconnect the client.  

The buffer I am referring to is the TCP socket's send and rec buffer.  I tried setting it to a larger value (16K) from the default(8K), but that does not fix the problem.  

Any help would be highly appreciated.


#define API_SocketIOControl( socket, nFlag, lValue ) \
                                ioctl( socket, nFlag, (int)&lValue )
 
 
   if ( SOCKET_ERROR == API_SocketIOControl( fd, FIONREAD, lBytesAvailable ) )
            {              
                printf("Could not peak at the receive buffer");
            }
#define API_SocketIOControl( socket, nFlag, lValue ) \
                                ioctl( socket, nFlag, (int)&lValue )
 
 
   if ( SOCKET_ERROR == API_SocketIOControl( fd, FIONREAD, lBytesAvailable ) )
            {              
                printf("Could not peak at the receive buffer");
            }

Open in new window

Comment
Watch Question

Duncan RoeSoftware Developer
CERTIFIED EXPERT

Commented:
You should use select() to tell you whether data is available for reading and / or when it is possible to write something. Always read before you try to write. This applies to any TCP application, be it server or client. Rather than loop, use select() to wait until a wanted condition arises. There should be no need to peek at buffers.

Author

Commented:
Thanks.  Agree that I should use select().  But, even if I use select( ) It would block until the socket is writable.  So, I need to estimate how long I should wait.  

I wanted to peek at the send buffer as It would give me a hint about how soon the buffer gets filled in.  
Duncan RoeSoftware Developer
CERTIFIED EXPERT
Commented:
There are basically 3 common ways to use select():

1. Infinite timeout - only wake up when there is something to do,
2. Zero timeout - see what we can do now,
3. finite timeout - do some tasks if no i/o can be done for a while.

Generally I use 1. above. Your writes are unlikely to block if you have serviced all your reads - in fact I have never had a problem since I started doing it that way. Unix / Linux select() can be applied to arbitrary file descriptors - Microsoft select() restricts you to sockets only - a fairly common practice 15 or 20 years ago.
With multiple threads, you can have other threads do the periodic stuff. Even without them, fork()'d children can write a byte to a pipe when there is a message in a message queue for example.

As for peeking at the buffer - yes the real-time arrival speed of data is generally much less than CPU speed so you *could* monitor arriving data. But you don't really know what the operating system is doing vis a vis buffering, and you don't know in a multiprogramming environment when some other process has been given the CPU. Are you needs really that specialised that you want to monitor the arrival of incoming incomplete data?

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts
Duncan RoeSoftware Developer
CERTIFIED EXPERT

Commented:
Also bear in mind that TCP is a streaming protocol, so the only end of message conditions are imposed by the application software
>>>> Or what would be the right handling for the server to do if it detects EWOULDBLOCK.

To add to above information:

You also can handle a non-blocking socket not using select. Or better said, the main purpose of select is for blocking sockets not for non-blocking sockets.

So generally you should do either

(A)
while (true)
{
     rc = select(0, readFds, 0, 0, timeout);
     if (rc > 0)
     {
            nread = recv(sock, buf, sizeof(buf), 0);
            ...
     }
}

or

(B) (non-blocking socket)

bool wouldblock = false;
while (true)
{
       nread = recv(sock, buf, sizeof(buf), 0);
       if (nread < 0 && errno == EWOULDBLOCK)
       {
             wouldblock = true;
       }
       else if (nread < 0)
       {
             log_error();
             break;      
       }
       else //  if (nread >= 0)
       {
             // received a buffer
       }
        // here you could do send jobs (if any)

       // wait a little while in case of wouldblock
       if (wouldblock)  sleep(sleepval);
}

Note, if running an non-blocking socket, it almost ever will return with socket error and EWOULDBLOCK.

Note a sending operation nomally wouldn't return with EWOULDBLOCK beside you've not made the socket generally non-blocking. On some systems you also might get EWOULDBLOCK if a read operation was not fully done. So you should follow the advice of duncan_row to firstly do the reads and then the writes if using one socket for both reading and writing.


Author

Commented:
Hi thanks for the info.  
"Always read before you try to write" :  Is that means, for example the server is trying to write to the socket it should also try to read? (Yes, I am using one socket for both reading and writing between the client and server and it is non-blocking socket)

Here is my experiment result.  I am generating lot of traffic from the server and send to the client.  One thread reads the messages and put them into a queue.  Another thread reads from the queue(readQThread).  My problem originates when the readQThread is slow in reading the messages off the queue.  When the Queue gets filled in I am seeing the EWOULDBLOCK on the server.  I was able to reproduce it fairly quickly.  
>>>> Is that means, for example the server is trying to write to the socket it should also try to read?

I assume at the server side you have one thread for each client connection. If not, you would need to synchronize cause sockets are not really thread-safe. If you have only one thread for both, this thread should run an infinite loop (like in my sample B above) and should first read and second write to the socket. Though normally you could do both on a (TCP/IP) socket in arbitrary order you nevertheless can get problems if either do that by different threads and have *no* synchronisation or - in case of a single tread - if you try to write while the read not yet is complete. You can get different errors in both cases but for a non-blocking socket the EWOULDBLOCK is the most reasonable error return cause for any (possible) blocking the operation immediately was canceled and EWOULDBLOCK returned.
>>>> Agree that I should use select().  But, even if I use select( ) It would block until the socket is writable.  

Indeed, using select with timeout is the easier concept before making the socket non-blocking and handling a cillion of EWOULDBLOCK "errors" for nothing.

Though a non-blocking socket has some charme (like driving a car without brakes?) it is the less recommendable concept.  

Author

Commented:
There are two threads on the server side for reading from and writing to the socket. There are synchronized.  I began to suspect that when the write thread is busy writing messages, there is something to read from the socket as well. Could that be a reason for my EWOULDBLOCK error.  I am not the author of the original code.  I am trying to minimize the changes while solving this issue( as usually the case:-))


>>>> I began to suspect that when the write thread is busy writing messages, there is something to read from the socket as well.

I never used two threads for a non-blocking socket. I think that isn't efficient cause you need to poll anyhow, hence you easily could do sending and reading with one thread. I think the send will return with EWOULDBLOCK in case there currently is a read operation not completed. You might have a workaround by simply wait a little while and try again. But of course you would need to unlock the synchronization after each failure to give the read thread a chance to complete the job. Anyway, using one thread for both reading and writing is much easier. Use a select with little timeout to check for incoming data. After timeout, check the write queue of your thread whether there is something to send. If done, check the stopflag your main thread might have checked, if not do the next loop cycle. That all is straight-forward and well-proofed.
Duncan RoeSoftware Developer
CERTIFIED EXPERT

Commented:
You don't need multiple threads for reading and writing. select() can check for both. That is the classic way of using it. Multithreading was rare indeed when the Socket interface was designed. By using select() to wait for everything (read, write & exceptions) you guarantee coordination between reading and writing. ISelect() (and the non-BSD equibalent, poll()) are the only system calls I know of which can wait for multiple events - they're central to many Unix programs.
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.