How to mix text and binary data over TClient/ServerSockets

Question previously asked in Win Networking forum.  No response in two days.

I am programming in Delphi 6 using standard TClient/ServerSockets windows socket connections.  I am writing both the Client and the Server programs, using non-blocking connections.

How do I mix normal text messages with binary (eg image) data?

The basic problem is that I don't know how long each message is.

For solid text, I can put an "End of Transaction" marker in between each message.  No problem.

But for binary data, the data may naturally contain the EOT marker.  Also the binary files may be "large" (over 64K bytes).

As I see it, the alternatives are:

1. The start of each message does contain a "type" flag, like "TXT" or "BIN", then...

2. Use a VERY unusual, multi-byte EOT marker, like hex "00FF00FF" or something.  Not bullet-proof, but close.

3. Send the binary data as counted byte streams, that is: preceeded by the byte count.  I hesitate to do that with "all" data, since most of the data is text and a lost byte is usually no big deal.

4. On the Server side, wait until the Client's receive buffer is empty before sending BIN data.  Then wait again after sending.  That way, for BIN data, the Client can always read until the buffer is empty.

The question is:  What are other / better alternatives?


Phil Henningsen
Who is Participating?
danielluyoConnect With a Mentor Commented:
The easiest way is to include data type and data size information:


the protocol means:
Data type TXT= text
Data size 0005 = 5 bytes
next 5 bytes are data

Data type BIN= binary
Data size 0009 = 9 bytes
next 9 bytes are data
PHenningsenAuthor Commented:
Yesterday I implemented the "counted byte stream" approach, a fancier version of what you suggested.  Each message is preceeded with a text string that looks like:
in this case followed by (supposedly) 46545 bytes of a GIF image.  I'm using the GifImage component by Anders Melander.

Text works fine.  I'm having big problems with the size of the GIF image (which may be getting "off-topic")

The "sending code" looks like:
//=----------------------------------------- SendPictToClient ------------------
// currently all PICTs are GIFs
// for example, an entire transaction may look like:
// Hdr=031,Xmit=Pict,Length=46554,GIF87......
procedure TfmMain.SendPictToClient(SocketHandle: TSocket; anImage: TGifImage);
  iLth, iX: integer;
  sHdr: string;
  MemStream: TMemoryStream;
  MemStream := TMemoryStream.Create;
    iLth := MemStream.Size;
    sHdr := ',Xmit=Pict,Length='+IntToStr(iLth)+',';
    sHdr := 'Hdr='+PadLeft0(IntToStr(Length(sHdr)+7),3)+sHdr;
    MemStream.Position := 0;
    CommListBox1.Items.Insert(0,'Log=Sending> ,'+sHdr+'Remainder_Is=Omitted');
    with ServerSocket1 do
      for iX := 0 to Socket.ActiveConnections-1 do
        if (SocketHandle = 0) or          // =0 means broadcast to all client sockets
           (SocketHandle = Socket.Connections[iX].SocketHandle) then
            on E:ESocketError do
              if (SocketHandle = 0) then
                { nothing, ignore broadcast errors }
                raise Eg3Exception.Create('SendPictToClient, '+E.ClassName+': '+E.Message);
            on E:Exception do
              raise Eg3Exception.Create('SendPictToClient, '+E.ClassName+': '+E.Message);
          end;{if, try..except}
    //MemStream.Free;   // is Freed by the Windows socket object

In testing, the image is loaded from a GIF file on disk.  Win Explorer (and other methods) say that the file is 46554 bytes long.  "MemStream.Size", above, says that it is 46545 bytes, 9 bytes shorter.  MemStream.Position says the same.

Slick812Connect With a Mentor Commented:
hello PHenningsen, The relative size of the Gif Image will depend on the settings that you have for the GIF image, so the Size of the mem Stream may not be the same size as the original gif file , I do not beleive that a GIF image will save the original "File" bytes and then write them to the memory stream, instead it generates a NEW gif file and saves it to the mem stream, As far as I have seen, if it does not through an exception. then the file to stream is Good. . . you can test this by doing a save to file instead of a save to stream, and see if the new file is 46545 bytes,, , ,   OR you can do a test like this -

Gif1 := TGifImage.Create;
 MemStream := TMemoryStream.Create;

Gif1 := TGifImage.Create;
MemStream.Position := 0;
// show the Gif1 to see if it is OK

 - - - - -  - - and just a suggestion
you seem to use "Text"  ID and information in your  data segments, I think I would use "Binary" data for binary values, if your size is an integer, I would not use IntToStr text, but use an Integer
gmayoConnect With a Mentor Commented:
Another option is to encode the binary data into text form. For example, use the hex characters for each byte. This results in transfers exactly twice as big as the original file. Or try MIME encoding, as per binary attachments in emails.

Either way you can use appropriate terminators in data. I use ETX and STX.

I find that TClientSocket and TServerSocket lose data when a large amount is transmitted together, especially if the Internet is involved. The only way to solve that is to use a more advanced component (maybe Indy if that supports buffering) or do the buffering yourself. Also note that a block of X bytes sent in at one end results in multiple blocks A+B+C at the other - in other words, the OnClientRead event is called every time a trickle of information arrives, not necessarily when the whole lot arrives. It does, at least, arrive in the order it was sent!

Geoff M.
PHenningsenAuthor Commented:
Here's the current status.

The GifImage size problem is solved. TMemoryStream.Size is correct.  Apparently it got padded when written to disk.

My current problem is that there is not a "clean break" between BIN and TXT data.  After receiving a large BIN file (requiring several ReceiveBuf's), I get several more OnRead interrupts wherein the ReceiveBuf contains old parts on the BIN data.

If the connection is idle between BIN data and TXT data and if I flush the buffer (by TempString := Socket.ReceiveText, and TempString is usually empty), then all is well.

The problems arise when large BIN data is immediately followed by TXT data.

I don't know whether to persevere with this "counted byte string" approach, or to try another?

If another approach, then which one?  

(Indy doesn't seem to be much help.  It seems to use ServerType = stThreadBlocking, and my Server is already a thread controller)

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.