[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now


How to mix text and binary data over TClient/ServerSockets

Posted on 2004-07-30
Medium Priority
Last Modified: 2010-05-18
Question previously asked in Win Networking forum.  No response in two days.

I am programming in Delphi 6 using standard TClient/ServerSockets windows socket connections.  I am writing both the Client and the Server programs, using non-blocking connections.

How do I mix normal text messages with binary (eg image) data?

The basic problem is that I don't know how long each message is.

For solid text, I can put an "End of Transaction" marker in between each message.  No problem.

But for binary data, the data may naturally contain the EOT marker.  Also the binary files may be "large" (over 64K bytes).

As I see it, the alternatives are:

1. The start of each message does contain a "type" flag, like "TXT" or "BIN", then...

2. Use a VERY unusual, multi-byte EOT marker, like hex "00FF00FF" or something.  Not bullet-proof, but close.

3. Send the binary data as counted byte streams, that is: preceeded by the byte count.  I hesitate to do that with "all" data, since most of the data is text and a lost byte is usually no big deal.

4. On the Server side, wait until the Client's receive buffer is empty before sending BIN data.  Then wait again after sending.  That way, for BIN data, the Client can always read until the buffer is empty.

The question is:  What are other / better alternatives?


Phil Henningsen
Question by:PHenningsen
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions

Accepted Solution

danielluyo earned 672 total points
ID: 11680564
The easiest way is to include data type and data size information:


the protocol means:
Data type TXT= text
Data size 0005 = 5 bytes
next 5 bytes are data

Data type BIN= binary
Data size 0009 = 9 bytes
next 9 bytes are data

Author Comment

ID: 11680751
Yesterday I implemented the "counted byte stream" approach, a fancier version of what you suggested.  Each message is preceeded with a text string that looks like:
in this case followed by (supposedly) 46545 bytes of a GIF image.  I'm using the GifImage component by Anders Melander.

Text works fine.  I'm having big problems with the size of the GIF image (which may be getting "off-topic")

The "sending code" looks like:
//=----------------------------------------- SendPictToClient ------------------
// currently all PICTs are GIFs
// for example, an entire transaction may look like:
// Hdr=031,Xmit=Pict,Length=46554,GIF87......
procedure TfmMain.SendPictToClient(SocketHandle: TSocket; anImage: TGifImage);
  iLth, iX: integer;
  sHdr: string;
  MemStream: TMemoryStream;
  MemStream := TMemoryStream.Create;
    iLth := MemStream.Size;
    sHdr := ',Xmit=Pict,Length='+IntToStr(iLth)+',';
    sHdr := 'Hdr='+PadLeft0(IntToStr(Length(sHdr)+7),3)+sHdr;
    MemStream.Position := 0;
    CommListBox1.Items.Insert(0,'Log=Sending> ,'+sHdr+'Remainder_Is=Omitted');
    with ServerSocket1 do
      for iX := 0 to Socket.ActiveConnections-1 do
        if (SocketHandle = 0) or          // =0 means broadcast to all client sockets
           (SocketHandle = Socket.Connections[iX].SocketHandle) then
            on E:ESocketError do
              if (SocketHandle = 0) then
                { nothing, ignore broadcast errors }
                raise Eg3Exception.Create('SendPictToClient, '+E.ClassName+': '+E.Message);
            on E:Exception do
              raise Eg3Exception.Create('SendPictToClient, '+E.ClassName+': '+E.Message);
          end;{if, try..except}
    //MemStream.Free;   // is Freed by the Windows socket object

In testing, the image is loaded from a GIF file on disk.  Win Explorer (and other methods) say that the file is 46554 bytes long.  "MemStream.Size", above, says that it is 46545 bytes, 9 bytes shorter.  MemStream.Position says the same.

LVL 34

Assisted Solution

Slick812 earned 664 total points
ID: 11682873
hello PHenningsen, The relative size of the Gif Image will depend on the settings that you have for the GIF image, so the Size of the mem Stream may not be the same size as the original gif file , I do not beleive that a GIF image will save the original "File" bytes and then write them to the memory stream, instead it generates a NEW gif file and saves it to the mem stream, As far as I have seen, if it does not through an exception. then the file to stream is Good. . . you can test this by doing a save to file instead of a save to stream, and see if the new file is 46545 bytes,, , ,   OR you can do a test like this -

Gif1 := TGifImage.Create;
 MemStream := TMemoryStream.Create;

Gif1 := TGifImage.Create;
MemStream.Position := 0;
// show the Gif1 to see if it is OK

 - - - - -  - - and just a suggestion
you seem to use "Text"  ID and information in your  data segments, I think I would use "Binary" data for binary values, if your size is an integer, I would not use IntToStr text, but use an Integer

Assisted Solution

gmayo earned 664 total points
ID: 11693439
Another option is to encode the binary data into text form. For example, use the hex characters for each byte. This results in transfers exactly twice as big as the original file. Or try MIME encoding, as per binary attachments in emails.

Either way you can use appropriate terminators in data. I use ETX and STX.

I find that TClientSocket and TServerSocket lose data when a large amount is transmitted together, especially if the Internet is involved. The only way to solve that is to use a more advanced component (maybe Indy if that supports buffering) or do the buffering yourself. Also note that a block of X bytes sent in at one end results in multiple blocks A+B+C at the other - in other words, the OnClientRead event is called every time a trickle of information arrives, not necessarily when the whole lot arrives. It does, at least, arrive in the order it was sent!

Geoff M.

Author Comment

ID: 11697682
Here's the current status.

The GifImage size problem is solved. TMemoryStream.Size is correct.  Apparently it got padded when written to disk.

My current problem is that there is not a "clean break" between BIN and TXT data.  After receiving a large BIN file (requiring several ReceiveBuf's), I get several more OnRead interrupts wherein the ReceiveBuf contains old parts on the BIN data.

If the connection is idle between BIN data and TXT data and if I flush the buffer (by TempString := Socket.ReceiveText, and TempString is usually empty), then all is well.

The problems arise when large BIN data is immediately followed by TXT data.

I don't know whether to persevere with this "counted byte string" approach, or to try another?

If another approach, then which one?  

(Indy doesn't seem to be much help.  It seems to use ServerType = stThreadBlocking, and my Server is already a thread controller)


Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Objective: - This article will help user in how to convert their numeric value become words. How to use 1. You can copy this code in your Unit as function 2. than you can perform your function by type this code The Code   (CODE) The Im…
Have you ever had your Delphi form/application just hanging while waiting for data to load? This is the article to read if you want to learn some things about adding threads for data loading in the background. First, I'll setup a general applica…
Video by: ITPro.TV
In this episode Don builds upon the troubleshooting techniques by demonstrating how to properly monitor a vSphere deployment to detect problems before they occur. He begins the show using tools found within the vSphere suite as ends the show demonst…
Are you ready to place your question in front of subject-matter experts for more timely responses? With the release of Priority Question, Premium Members, Team Accounts and Qualified Experts can now identify the emergent level of their issue, signal…
Suggested Courses

656 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question