socket send buffer returns EWOULDBLOCK
Hi Experts,
I have a client-server application using socket to communicate. At times, the server disconnects the client because the write to the client returns EWOULDBLOCK. I initially thought that the client is not reading fast enough thus the TCP receive buffer is filled up. I then added code to peek at the client's TCP buffer to see if it is getting filled up. It never gets above 3K while the buffer size is 8K. I used the FIONREAD to peek at the rec buffer.
My question is, is there a way to peek at the send buffer? Or what would be the right handling for the server to do if it detects EWOULDBLOCK. Right now, I try to re-write few times and if that fails I disconnect the client.
The buffer I am referring to is the TCP socket's send and rec buffer. I tried setting it to a larger value (16K) from the default(8K), but that does not fix the problem.
Any help would be highly appreciated.
#define API_SocketIOControl( socket, nFlag, lValue ) \
ioctl( socket, nFlag, (int)&lValue )
if ( SOCKET_ERROR == API_SocketIOControl( fd, FIONREAD, lBytesAvailable ) )
{
printf("Could not peak at the receive buffer");
}
I have a client-server application using socket to communicate. At times, the server disconnects the client because the write to the client returns EWOULDBLOCK. I initially thought that the client is not reading fast enough thus the TCP receive buffer is filled up. I then added code to peek at the client's TCP buffer to see if it is getting filled up. It never gets above 3K while the buffer size is 8K. I used the FIONREAD to peek at the rec buffer.
My question is, is there a way to peek at the send buffer? Or what would be the right handling for the server to do if it detects EWOULDBLOCK. Right now, I try to re-write few times and if that fails I disconnect the client.
The buffer I am referring to is the TCP socket's send and rec buffer. I tried setting it to a larger value (16K) from the default(8K), but that does not fix the problem.
Any help would be highly appreciated.
#define API_SocketIOControl( socket, nFlag, lValue ) \
ioctl( socket, nFlag, (int)&lValue )
if ( SOCKET_ERROR == API_SocketIOControl( fd, FIONREAD, lBytesAvailable ) )
{
printf("Could not peak at the receive buffer");
}
#define API_SocketIOControl( socket, nFlag, lValue ) \
ioctl( socket, nFlag, (int)&lValue )
if ( SOCKET_ERROR == API_SocketIOControl( fd, FIONREAD, lBytesAvailable ) )
{
printf("Could not peak at the receive buffer");
}
Duncan Roe
You should use select() to tell you whether data is available for reading and / or when it is possible to write something. Always read before you try to write. This applies to any TCP application, be it server or client. Rather than loop, use select() to wait until a wanted condition arises. There should be no need to peek at buffers.
ASKER
Thanks. Agree that I should use select(). But, even if I use select( ) It would block until the socket is writable. So, I need to estimate how long I should wait.
I wanted to peek at the send buffer as It would give me a hint about how soon the buffer gets filled in.
I wanted to peek at the send buffer as It would give me a hint about how soon the buffer gets filled in.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Also bear in mind that TCP is a streaming protocol, so the only end of message conditions are imposed by the application software
>>>> Or what would be the right handling for the server to do if it detects EWOULDBLOCK.
To add to above information:
You also can handle a non-blocking socket not using select. Or better said, the main purpose of select is for blocking sockets not for non-blocking sockets.
So generally you should do either
(A)
while (true)
{
rc = select(0, readFds, 0, 0, timeout);
if (rc > 0)
{
nread = recv(sock, buf, sizeof(buf), 0);
...
}
}
or
(B) (non-blocking socket)
bool wouldblock = false;
while (true)
{
nread = recv(sock, buf, sizeof(buf), 0);
if (nread < 0 && errno == EWOULDBLOCK)
{
wouldblock = true;
}
else if (nread < 0)
{
log_error();
break;
}
else // if (nread >= 0)
{
// received a buffer
}
// here you could do send jobs (if any)
// wait a little while in case of wouldblock
if (wouldblock) sleep(sleepval);
}
Note, if running an non-blocking socket, it almost ever will return with socket error and EWOULDBLOCK.
Note a sending operation nomally wouldn't return with EWOULDBLOCK beside you've not made the socket generally non-blocking. On some systems you also might get EWOULDBLOCK if a read operation was not fully done. So you should follow the advice of duncan_row to firstly do the reads and then the writes if using one socket for both reading and writing.
To add to above information:
You also can handle a non-blocking socket not using select. Or better said, the main purpose of select is for blocking sockets not for non-blocking sockets.
So generally you should do either
(A)
while (true)
{
rc = select(0, readFds, 0, 0, timeout);
if (rc > 0)
{
nread = recv(sock, buf, sizeof(buf), 0);
...
}
}
or
(B) (non-blocking socket)
bool wouldblock = false;
while (true)
{
nread = recv(sock, buf, sizeof(buf), 0);
if (nread < 0 && errno == EWOULDBLOCK)
{
wouldblock = true;
}
else if (nread < 0)
{
log_error();
break;
}
else // if (nread >= 0)
{
// received a buffer
}
// here you could do send jobs (if any)
// wait a little while in case of wouldblock
if (wouldblock) sleep(sleepval);
}
Note, if running an non-blocking socket, it almost ever will return with socket error and EWOULDBLOCK.
Note a sending operation nomally wouldn't return with EWOULDBLOCK beside you've not made the socket generally non-blocking. On some systems you also might get EWOULDBLOCK if a read operation was not fully done. So you should follow the advice of duncan_row to firstly do the reads and then the writes if using one socket for both reading and writing.
ASKER
Hi thanks for the info.
"Always read before you try to write" : Is that means, for example the server is trying to write to the socket it should also try to read? (Yes, I am using one socket for both reading and writing between the client and server and it is non-blocking socket)
Here is my experiment result. I am generating lot of traffic from the server and send to the client. One thread reads the messages and put them into a queue. Another thread reads from the queue(readQThread). My problem originates when the readQThread is slow in reading the messages off the queue. When the Queue gets filled in I am seeing the EWOULDBLOCK on the server. I was able to reproduce it fairly quickly.
"Always read before you try to write" : Is that means, for example the server is trying to write to the socket it should also try to read? (Yes, I am using one socket for both reading and writing between the client and server and it is non-blocking socket)
Here is my experiment result. I am generating lot of traffic from the server and send to the client. One thread reads the messages and put them into a queue. Another thread reads from the queue(readQThread). My problem originates when the readQThread is slow in reading the messages off the queue. When the Queue gets filled in I am seeing the EWOULDBLOCK on the server. I was able to reproduce it fairly quickly.
>>>> Is that means, for example the server is trying to write to the socket it should also try to read?
I assume at the server side you have one thread for each client connection. If not, you would need to synchronize cause sockets are not really thread-safe. If you have only one thread for both, this thread should run an infinite loop (like in my sample B above) and should first read and second write to the socket. Though normally you could do both on a (TCP/IP) socket in arbitrary order you nevertheless can get problems if either do that by different threads and have *no* synchronisation or - in case of a single tread - if you try to write while the read not yet is complete. You can get different errors in both cases but for a non-blocking socket the EWOULDBLOCK is the most reasonable error return cause for any (possible) blocking the operation immediately was canceled and EWOULDBLOCK returned.
I assume at the server side you have one thread for each client connection. If not, you would need to synchronize cause sockets are not really thread-safe. If you have only one thread for both, this thread should run an infinite loop (like in my sample B above) and should first read and second write to the socket. Though normally you could do both on a (TCP/IP) socket in arbitrary order you nevertheless can get problems if either do that by different threads and have *no* synchronisation or - in case of a single tread - if you try to write while the read not yet is complete. You can get different errors in both cases but for a non-blocking socket the EWOULDBLOCK is the most reasonable error return cause for any (possible) blocking the operation immediately was canceled and EWOULDBLOCK returned.
>>>> Agree that I should use select(). But, even if I use select( ) It would block until the socket is writable.
Indeed, using select with timeout is the easier concept before making the socket non-blocking and handling a cillion of EWOULDBLOCK "errors" for nothing.
Though a non-blocking socket has some charme (like driving a car without brakes?) it is the less recommendable concept.
Indeed, using select with timeout is the easier concept before making the socket non-blocking and handling a cillion of EWOULDBLOCK "errors" for nothing.
Though a non-blocking socket has some charme (like driving a car without brakes?) it is the less recommendable concept.
ASKER
There are two threads on the server side for reading from and writing to the socket. There are synchronized. I began to suspect that when the write thread is busy writing messages, there is something to read from the socket as well. Could that be a reason for my EWOULDBLOCK error. I am not the author of the original code. I am trying to minimize the changes while solving this issue( as usually the case:-))
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
You don't need multiple threads for reading and writing. select() can check for both. That is the classic way of using it. Multithreading was rare indeed when the Socket interface was designed. By using select() to wait for everything (read, write & exceptions) you guarantee coordination between reading and writing. ISelect() (and the non-BSD equibalent, poll()) are the only system calls I know of which can wait for multiple events - they're central to many Unix programs.