How to mix text and binary data over TClient/ServerSockets

Posted on 2004-07-30
Last Modified: 2010-05-18
Question previously asked in Win Networking forum.  No response in two days.

I am programming in Delphi 6 using standard TClient/ServerSockets windows socket connections.  I am writing both the Client and the Server programs, using non-blocking connections.

How do I mix normal text messages with binary (eg image) data?

The basic problem is that I don't know how long each message is.

For solid text, I can put an "End of Transaction" marker in between each message.  No problem.

But for binary data, the data may naturally contain the EOT marker.  Also the binary files may be "large" (over 64K bytes).

As I see it, the alternatives are:

1. The start of each message does contain a "type" flag, like "TXT" or "BIN", then...

2. Use a VERY unusual, multi-byte EOT marker, like hex "00FF00FF" or something.  Not bullet-proof, but close.

3. Send the binary data as counted byte streams, that is: preceeded by the byte count.  I hesitate to do that with "all" data, since most of the data is text and a lost byte is usually no big deal.

4. On the Server side, wait until the Client's receive buffer is empty before sending BIN data.  Then wait again after sending.  That way, for BIN data, the Client can always read until the buffer is empty.

The question is:  What are other / better alternatives?


Phil Henningsen
Question by:PHenningsen

Accepted Solution

danielluyo earned 168 total points
ID: 11680564
The easiest way is to include data type and data size information:


the protocol means:
Data type TXT= text
Data size 0005 = 5 bytes
next 5 bytes are data

Data type BIN= binary
Data size 0009 = 9 bytes
next 9 bytes are data

Author Comment

ID: 11680751
Yesterday I implemented the "counted byte stream" approach, a fancier version of what you suggested.  Each message is preceeded with a text string that looks like:
in this case followed by (supposedly) 46545 bytes of a GIF image.  I'm using the GifImage component by Anders Melander.

Text works fine.  I'm having big problems with the size of the GIF image (which may be getting "off-topic")

The "sending code" looks like:
//=----------------------------------------- SendPictToClient ------------------
// currently all PICTs are GIFs
// for example, an entire transaction may look like:
// Hdr=031,Xmit=Pict,Length=46554,GIF87......
procedure TfmMain.SendPictToClient(SocketHandle: TSocket; anImage: TGifImage);
  iLth, iX: integer;
  sHdr: string;
  MemStream: TMemoryStream;
  MemStream := TMemoryStream.Create;
    iLth := MemStream.Size;
    sHdr := ',Xmit=Pict,Length='+IntToStr(iLth)+',';
    sHdr := 'Hdr='+PadLeft0(IntToStr(Length(sHdr)+7),3)+sHdr;
    MemStream.Position := 0;
    CommListBox1.Items.Insert(0,'Log=Sending> ,'+sHdr+'Remainder_Is=Omitted');
    with ServerSocket1 do
      for iX := 0 to Socket.ActiveConnections-1 do
        if (SocketHandle = 0) or          // =0 means broadcast to all client sockets
           (SocketHandle = Socket.Connections[iX].SocketHandle) then
            on E:ESocketError do
              if (SocketHandle = 0) then
                { nothing, ignore broadcast errors }
                raise Eg3Exception.Create('SendPictToClient, '+E.ClassName+': '+E.Message);
            on E:Exception do
              raise Eg3Exception.Create('SendPictToClient, '+E.ClassName+': '+E.Message);
          end;{if, try..except}
    //MemStream.Free;   // is Freed by the Windows socket object

In testing, the image is loaded from a GIF file on disk.  Win Explorer (and other methods) say that the file is 46554 bytes long.  "MemStream.Size", above, says that it is 46545 bytes, 9 bytes shorter.  MemStream.Position says the same.

LVL 34

Assisted Solution

Slick812 earned 166 total points
ID: 11682873
hello PHenningsen, The relative size of the Gif Image will depend on the settings that you have for the GIF image, so the Size of the mem Stream may not be the same size as the original gif file , I do not beleive that a GIF image will save the original "File" bytes and then write them to the memory stream, instead it generates a NEW gif file and saves it to the mem stream, As far as I have seen, if it does not through an exception. then the file to stream is Good. . . you can test this by doing a save to file instead of a save to stream, and see if the new file is 46545 bytes,, , ,   OR you can do a test like this -

Gif1 := TGifImage.Create;
 MemStream := TMemoryStream.Create;

Gif1 := TGifImage.Create;
MemStream.Position := 0;
// show the Gif1 to see if it is OK

 - - - - -  - - and just a suggestion
you seem to use "Text"  ID and information in your  data segments, I think I would use "Binary" data for binary values, if your size is an integer, I would not use IntToStr text, but use an Integer

Assisted Solution

gmayo earned 166 total points
ID: 11693439
Another option is to encode the binary data into text form. For example, use the hex characters for each byte. This results in transfers exactly twice as big as the original file. Or try MIME encoding, as per binary attachments in emails.

Either way you can use appropriate terminators in data. I use ETX and STX.

I find that TClientSocket and TServerSocket lose data when a large amount is transmitted together, especially if the Internet is involved. The only way to solve that is to use a more advanced component (maybe Indy if that supports buffering) or do the buffering yourself. Also note that a block of X bytes sent in at one end results in multiple blocks A+B+C at the other - in other words, the OnClientRead event is called every time a trickle of information arrives, not necessarily when the whole lot arrives. It does, at least, arrive in the order it was sent!

Geoff M.

Author Comment

ID: 11697682
Here's the current status.

The GifImage size problem is solved. TMemoryStream.Size is correct.  Apparently it got padded when written to disk.

My current problem is that there is not a "clean break" between BIN and TXT data.  After receiving a large BIN file (requiring several ReceiveBuf's), I get several more OnRead interrupts wherein the ReceiveBuf contains old parts on the BIN data.

If the connection is idle between BIN data and TXT data and if I flush the buffer (by TempString := Socket.ReceiveText, and TempString is usually empty), then all is well.

The problems arise when large BIN data is immediately followed by TXT data.

I don't know whether to persevere with this "counted byte string" approach, or to try another?

If another approach, then which one?  

(Indy doesn't seem to be much help.  It seems to use ServerType = stThreadBlocking, and my Server is already a thread controller)


Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A lot of questions regard threads in Delphi.   One of the more specific questions is how to show progress of the thread.   Updating a progressbar from inside a thread is a mistake. A solution to this would be to send a synchronized message to the…
Creating an auto free TStringList The TStringList is a basic and frequently used object in Delphi. On many occasions, you may want to create a temporary list, process some items in the list and be done with the list. In such cases, you have to…
With Secure Portal Encryption, the recipient is sent a link to their email address directing them to the email laundry delivery page. From there, the recipient will be required to enter a user name and password to enter the page. Once the recipient …

828 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question