How to mix text and binary data over TClient/ServerSockets

Posted on 2004-07-30
Medium Priority
Last Modified: 2010-05-18
Question previously asked in Win Networking forum.  No response in two days.

I am programming in Delphi 6 using standard TClient/ServerSockets windows socket connections.  I am writing both the Client and the Server programs, using non-blocking connections.

How do I mix normal text messages with binary (eg image) data?

The basic problem is that I don't know how long each message is.

For solid text, I can put an "End of Transaction" marker in between each message.  No problem.

But for binary data, the data may naturally contain the EOT marker.  Also the binary files may be "large" (over 64K bytes).

As I see it, the alternatives are:

1. The start of each message does contain a "type" flag, like "TXT" or "BIN", then...

2. Use a VERY unusual, multi-byte EOT marker, like hex "00FF00FF" or something.  Not bullet-proof, but close.

3. Send the binary data as counted byte streams, that is: preceeded by the byte count.  I hesitate to do that with "all" data, since most of the data is text and a lost byte is usually no big deal.

4. On the Server side, wait until the Client's receive buffer is empty before sending BIN data.  Then wait again after sending.  That way, for BIN data, the Client can always read until the buffer is empty.

The question is:  What are other / better alternatives?


Phil Henningsen
Question by:PHenningsen

Accepted Solution

danielluyo earned 672 total points
ID: 11680564
The easiest way is to include data type and data size information:


the protocol means:
Data type TXT= text
Data size 0005 = 5 bytes
next 5 bytes are data

Data type BIN= binary
Data size 0009 = 9 bytes
next 9 bytes are data

Author Comment

ID: 11680751
Yesterday I implemented the "counted byte stream" approach, a fancier version of what you suggested.  Each message is preceeded with a text string that looks like:
in this case followed by (supposedly) 46545 bytes of a GIF image.  I'm using the GifImage component by Anders Melander.

Text works fine.  I'm having big problems with the size of the GIF image (which may be getting "off-topic")

The "sending code" looks like:
//=----------------------------------------- SendPictToClient ------------------
// currently all PICTs are GIFs
// for example, an entire transaction may look like:
// Hdr=031,Xmit=Pict,Length=46554,GIF87......
procedure TfmMain.SendPictToClient(SocketHandle: TSocket; anImage: TGifImage);
  iLth, iX: integer;
  sHdr: string;
  MemStream: TMemoryStream;
  MemStream := TMemoryStream.Create;
    iLth := MemStream.Size;
    sHdr := ',Xmit=Pict,Length='+IntToStr(iLth)+',';
    sHdr := 'Hdr='+PadLeft0(IntToStr(Length(sHdr)+7),3)+sHdr;
    MemStream.Position := 0;
    CommListBox1.Items.Insert(0,'Log=Sending> ,'+sHdr+'Remainder_Is=Omitted');
    with ServerSocket1 do
      for iX := 0 to Socket.ActiveConnections-1 do
        if (SocketHandle = 0) or          // =0 means broadcast to all client sockets
           (SocketHandle = Socket.Connections[iX].SocketHandle) then
            on E:ESocketError do
              if (SocketHandle = 0) then
                { nothing, ignore broadcast errors }
                raise Eg3Exception.Create('SendPictToClient, '+E.ClassName+': '+E.Message);
            on E:Exception do
              raise Eg3Exception.Create('SendPictToClient, '+E.ClassName+': '+E.Message);
          end;{if, try..except}
    //MemStream.Free;   // is Freed by the Windows socket object

In testing, the image is loaded from a GIF file on disk.  Win Explorer (and other methods) say that the file is 46554 bytes long.  "MemStream.Size", above, says that it is 46545 bytes, 9 bytes shorter.  MemStream.Position says the same.

LVL 34

Assisted Solution

Slick812 earned 664 total points
ID: 11682873
hello PHenningsen, The relative size of the Gif Image will depend on the settings that you have for the GIF image, so the Size of the mem Stream may not be the same size as the original gif file , I do not beleive that a GIF image will save the original "File" bytes and then write them to the memory stream, instead it generates a NEW gif file and saves it to the mem stream, As far as I have seen, if it does not through an exception. then the file to stream is Good. . . you can test this by doing a save to file instead of a save to stream, and see if the new file is 46545 bytes,, , ,   OR you can do a test like this -

Gif1 := TGifImage.Create;
 MemStream := TMemoryStream.Create;

Gif1 := TGifImage.Create;
MemStream.Position := 0;
// show the Gif1 to see if it is OK

 - - - - -  - - and just a suggestion
you seem to use "Text"  ID and information in your  data segments, I think I would use "Binary" data for binary values, if your size is an integer, I would not use IntToStr text, but use an Integer

Assisted Solution

gmayo earned 664 total points
ID: 11693439
Another option is to encode the binary data into text form. For example, use the hex characters for each byte. This results in transfers exactly twice as big as the original file. Or try MIME encoding, as per binary attachments in emails.

Either way you can use appropriate terminators in data. I use ETX and STX.

I find that TClientSocket and TServerSocket lose data when a large amount is transmitted together, especially if the Internet is involved. The only way to solve that is to use a more advanced component (maybe Indy if that supports buffering) or do the buffering yourself. Also note that a block of X bytes sent in at one end results in multiple blocks A+B+C at the other - in other words, the OnClientRead event is called every time a trickle of information arrives, not necessarily when the whole lot arrives. It does, at least, arrive in the order it was sent!

Geoff M.

Author Comment

ID: 11697682
Here's the current status.

The GifImage size problem is solved. TMemoryStream.Size is correct.  Apparently it got padded when written to disk.

My current problem is that there is not a "clean break" between BIN and TXT data.  After receiving a large BIN file (requiring several ReceiveBuf's), I get several more OnRead interrupts wherein the ReceiveBuf contains old parts on the BIN data.

If the connection is idle between BIN data and TXT data and if I flush the buffer (by TempString := Socket.ReceiveText, and TempString is usually empty), then all is well.

The problems arise when large BIN data is immediately followed by TXT data.

I don't know whether to persevere with this "counted byte string" approach, or to try another?

If another approach, then which one?  

(Indy doesn't seem to be much help.  It seems to use ServerType = stThreadBlocking, and my Server is already a thread controller)


Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article explains how to create forms/units independent of other forms/units object names in a delphi project. Have you ever created a form for user input in a Delphi project and then had the need to have that same form in a other Delphi proj…
Objective: - This article will help user in how to convert their numeric value become words. How to use 1. You can copy this code in your Unit as function 2. than you can perform your function by type this code The Code   (CODE) The Im…
Is your data getting by on basic protection measures? In today’s climate of debilitating malware and ransomware—like WannaCry—that may not be enough. You need to establish more than basics, like a recovery plan that protects both data and endpoints.…
Look below the covers at a subform control , and the form that is inside it. Explore properties and see how easy it is to aggregate, get statistics, and synchronize results for your data. A Microsoft Access subform is used to show relevant calcul…
Suggested Courses
Course of the Month15 days, 23 hours left to enroll

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question