Solved

How to mix text and binary data over TClient/ServerSockets

Posted on 2004-07-30
7
404 Views
Last Modified: 2010-05-18
Question previously asked in Win Networking forum.  No response in two days.

I am programming in Delphi 6 using standard TClient/ServerSockets windows socket connections.  I am writing both the Client and the Server programs, using non-blocking connections.

How do I mix normal text messages with binary (eg image) data?

The basic problem is that I don't know how long each message is.

For solid text, I can put an "End of Transaction" marker in between each message.  No problem.

But for binary data, the data may naturally contain the EOT marker.  Also the binary files may be "large" (over 64K bytes).

As I see it, the alternatives are:

1. The start of each message does contain a "type" flag, like "TXT" or "BIN", then...

2. Use a VERY unusual, multi-byte EOT marker, like hex "00FF00FF" or something.  Not bullet-proof, but close.

3. Send the binary data as counted byte streams, that is: preceeded by the byte count.  I hesitate to do that with "all" data, since most of the data is text and a lost byte is usually no big deal.

4. On the Server side, wait until the Client's receive buffer is empty before sending BIN data.  Then wait again after sending.  That way, for BIN data, the Client can always read until the buffer is empty.

The question is:  What are other / better alternatives?

Thanks,

Phil Henningsen
0
Comment
Question by:PHenningsen
7 Comments
 
LVL 3

Accepted Solution

by:
danielluyo earned 168 total points
Comment Utility
The easiest way is to include data type and data size information:

TXT0005helloBIN0009abcdefghi

the protocol means:
Data type TXT= text
Data size 0005 = 5 bytes
next 5 bytes are data

Data type BIN= binary
Data size 0009 = 9 bytes
next 9 bytes are data
0
 

Author Comment

by:PHenningsen
Comment Utility
Yesterday I implemented the "counted byte stream" approach, a fancier version of what you suggested.  Each message is preceeded with a text string that looks like:
Hdr=031,Xmit=Pict,Length=46545,
in this case followed by (supposedly) 46545 bytes of a GIF image.  I'm using the GifImage component by Anders Melander.

Text works fine.  I'm having big problems with the size of the GIF image (which may be getting "off-topic")

The "sending code" looks like:
//=----------------------------------------- SendPictToClient ------------------
// currently all PICTs are GIFs
// for example, an entire transaction may look like:
// Hdr=031,Xmit=Pict,Length=46554,GIF87......
procedure TfmMain.SendPictToClient(SocketHandle: TSocket; anImage: TGifImage);
var
  iLth, iX: integer;
  sHdr: string;
  MemStream: TMemoryStream;
begin
  MemStream := TMemoryStream.Create;
  try
    anImage.SaveToStream(MemStream);
    iLth := MemStream.Size;
    sHdr := ',Xmit=Pict,Length='+IntToStr(iLth)+',';
    sHdr := 'Hdr='+PadLeft0(IntToStr(Length(sHdr)+7),3)+sHdr;
    MemStream.Position := 0;
    CommListBox1.Items.Insert(0,'Log=Sending> ,'+sHdr+'Remainder_Is=Omitted');
    with ServerSocket1 do
      for iX := 0 to Socket.ActiveConnections-1 do
        begin
        if (SocketHandle = 0) or          // =0 means broadcast to all client sockets
           (SocketHandle = Socket.Connections[iX].SocketHandle) then
          try
            Socket.Connections[iX].SendText(sHdr);
            Socket.Connections[iX].SendStream(MemStream);
          except
            on E:ESocketError do
              if (SocketHandle = 0) then
                { nothing, ignore broadcast errors }
              else
                raise Eg3Exception.Create('SendPictToClient, '+E.ClassName+': '+E.Message);
            on E:Exception do
              raise Eg3Exception.Create('SendPictToClient, '+E.ClassName+': '+E.Message);
          end;{if, try..except}
        end;
  finally
    //MemStream.Free;   // is Freed by the Windows socket object
  end;
end;

In testing, the image is loaded from a GIF file on disk.  Win Explorer (and other methods) say that the file is 46554 bytes long.  "MemStream.Size", above, says that it is 46545 bytes, 9 bytes shorter.  MemStream.Position says the same.

0
 
LVL 33

Assisted Solution

by:Slick812
Slick812 earned 166 total points
Comment Utility
hello PHenningsen, The relative size of the Gif Image will depend on the settings that you have for the GIF image, so the Size of the mem Stream may not be the same size as the original gif file , I do not beleive that a GIF image will save the original "File" bytes and then write them to the memory stream, instead it generates a NEW gif file and saves it to the mem stream, As far as I have seen, if it does not through an exception. then the file to stream is Good. . . you can test this by doing a save to file instead of a save to stream, and see if the new file is 46545 bytes,, , ,   OR you can do a test like this -

Gif1 := TGifImage.Create;
Gif1.LoadFromFile('C:\someGif.gif');
 MemStream := TMemoryStream.Create;
 Gif1.SaveToStream(MemStream);
FreeAndNil(Gif1);

Gif1 := TGifImage.Create;
MemStream.Position := 0;
Gif1.LoadFromStream(MemStream);
FreeAndNil(MemStream);
// show the Gif1 to see if it is OK
Image1.Picture.Assign(Gif1);
FreeAndNil(Gif1);

 - - - - -  - - and just a suggestion
you seem to use "Text"  ID and information in your  data segments, I think I would use "Binary" data for binary values, if your size is an integer, I would not use IntToStr text, but use an Integer
0
 
LVL 8

Assisted Solution

by:gmayo
gmayo earned 166 total points
Comment Utility
Another option is to encode the binary data into text form. For example, use the hex characters for each byte. This results in transfers exactly twice as big as the original file. Or try MIME encoding, as per binary attachments in emails.

Either way you can use appropriate terminators in data. I use ETX and STX.

I find that TClientSocket and TServerSocket lose data when a large amount is transmitted together, especially if the Internet is involved. The only way to solve that is to use a more advanced component (maybe Indy if that supports buffering) or do the buffering yourself. Also note that a block of X bytes sent in at one end results in multiple blocks A+B+C at the other - in other words, the OnClientRead event is called every time a trickle of information arrives, not necessarily when the whole lot arrives. It does, at least, arrive in the order it was sent!

Geoff M.
0
 

Author Comment

by:PHenningsen
Comment Utility
Here's the current status.

The GifImage size problem is solved. TMemoryStream.Size is correct.  Apparently it got padded when written to disk.

My current problem is that there is not a "clean break" between BIN and TXT data.  After receiving a large BIN file (requiring several ReceiveBuf's), I get several more OnRead interrupts wherein the ReceiveBuf contains old parts on the BIN data.

If the connection is idle between BIN data and TXT data and if I flush the buffer (by TempString := Socket.ReceiveText, and TempString is usually empty), then all is well.

The problems arise when large BIN data is immediately followed by TXT data.

I don't know whether to persevere with this "counted byte string" approach, or to try another?

If another approach, then which one?  

(Indy doesn't seem to be much help.  It seems to use ServerType = stThreadBlocking, and my Server is already a thread controller)

0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Hello everybody This Article will show you how to validate number with TEdit control, What's the TEdit control? TEdit is a standard Windows edit control on a form, it allows to user to write, read and copy/paste single line of text. Usua…
Introduction Raise your hands if you were as upset with FireMonkey as I was when I discovered that there was no TListview.  I use TListView in almost all of my applications I've written, and I was not going to compromise by resorting to TStringGrid…
This video discusses moving either the default database or any database to a new volume.
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now