Solved

BinaryReader.Read function doesn't return the last couple of bytes in a NetworkStream (TcpListener) (Large amount of data 5MB+)

Posted on 2004-04-02
9
1,621 Views
Last Modified: 2012-06-21
I'm trying to read mixed data (text & binary) from a network stream. The data format i have chosen is similar to the mail format:

(Text in Brackets [ ] is substituted with appropriate data)

Message-ID: <[Hash]@[Sender]>
Date: [DateTime UTC]
Content-Type: multipart/mixed
Boundary: ------=_NextPart_[Hash]
Content-Part-Count: [Part Count];
{Additional headers}

[Message Text goes here]

------=_NextPart_[Hash]
Content-Length: [Length]
Content-Encoding: [Encoding]
Content-MD5Checksum: [MD5 Checksum]
{Additional headers}

[PartData (can be binary)]

------=_NextPart_[Hash]--

{Additional parts}

{EOF}

Reading the text part is easy. StreamReader.ReadLine() does the trick. However, when I come to the data parts, I'm switching to the BinaryReader. Since both readers use the underlaying NetworkStream I can use either one and be at the correct position within the stream. The problem occurs when I'm trying to read the binary data. I first read the "Content-Length" header of the part and then read the byte amount of that from the stream. I'm just reading the data from the stream, adding it to a byte array as we read. Since I'm reading over the network (Internet), I'm looping until I have read the right amount of bytes from the stream, then switching back to the StreamReader to read the next part, if there is one.
Having tried various things, I always seem to end up a few bytes short of data on the reciever side.

This is the code used to read the binary part of the message:

// read the data from the NetworkStream
[...]

// now read the data part

// read "Content-Length" from the header HashTable
int datalength = int.Parse(datapacket.GetHeaderValue("Content-Length"));

// create the byte array to hold the binary data
byte[] data = new byte[datalength];

int bytesRead = 0;
int bytesToRead = datalength;
int totalBytesRead = 0;

// set a timeout so we don't wait forever
TimeSpan timeout = new TimeSpan(0, 0, 5, 0, 0);
DateTime starttime = DateTime.Now;

// create the binary reader
BinaryReader datareader = new BinaryReader(networkStream, Encoding.UTF8);

// read until we have read all we need or until we readched our timeout
while((bytesToRead > 0) && (starttime.Add(timeout) > DateTime.Now))
{
      // read as long
      bytesRead = datareader.Read(data, bytesRead, bytesToRead);

      totalBytesRead += bytesRead;
      bytesToRead = datalength - totalBytesRead;

      // reset the timeout
      starttime = DateTime.Now;

      // let other processes do some work
      Thread.Sleep(10);
}

// check the data (MD5 checksum, length, etc.)
[...]

All my tries have failed. I always have about 430 bytes left to read (bytesToRead), even though the stream should contain a few bytes more than the binary data part. There should be a "------=_NextPart_[Hash]--" and a newline following.

Any ideas on what might be wrong?

The MSDN KnowledgeBase has an article on a bug in the BinaryReader.ReadChar() function at http://support.microsoft.com/default.aspx?scid=kb;en-us;318121 - Maybe this could be something similar?

Thanks for the help!
0
Comment
Question by:dave_the_brave
  • 4
  • 4
9 Comments
 
LVL 22

Expert Comment

by:_TAD_
ID: 10741062


you'll probably need to flush your data... that is, you sometimes need to explicitly tell your object to WRITE data to the byte[] and not just point to it.

  // read as long -- points to data
     bytesRead = datareader.Read(data, bytesRead, bytesToRead);
  // writes data to buffer (aka "data")
     datareader.BaseStream.Flush();  

   
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 10753051
The writer needs to flush - not the reader
Change this:  bytesRead = datareader.Read(data, bytesRead, bytesToRead);
to this:  bytesRead = datareader.Read(data, bytesRead, 2048);

Place this line:
starttime = DateTime.Now;
before the Read(..) - else the timeout will not work.
0
 

Author Comment

by:dave_the_brave
ID: 10756950
Thanks for the help!

It still doesn't work, though. I first thought I didn't write the data to the networkstream properly. I then just wrote the data i would send over the network to a file. the file is exactly the way it should be. Then I changed it back to write it to the networkstream (instead of the filestream).
Again! I still don't get all the data on the recipient side. I have no idea what the problem is. I did try to flush the data, however on a network stream, it didn't seem to do much.
So I did some research: according to the MSDN Library, a networkstream is not buffered, therefore the flush method doesn't do anything (it is reserved for future use).

I sent a 5MB file over the network (used BinaryWriter and BinaryReader) without the StreamReader or StreamWriter and it worked just fine.

Could it be that the use of a StreamReader and a BinaryReader on the same stream causes this strange effect?

I also use UTF8 Encoding. Maybe that could have something to do with it. I did a test on a 5MB file in different encodings. The file stayed the same size. So I'm guessing this doesn't have anything to do with it - or I'm doing something seriously wrong.
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 10757088
Why don't you use BinaryWriter/BinaryReader? You wrote that the stream contains binary data (not base64 encoded). May be the StreamReader causes troubles when reading the binary data.
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 

Author Comment

by:dave_the_brave
ID: 10757242
I do use both. I read the text with StreamReader.ReadLine() and then switch to the BinaryReader when I know the data starts. I only want to read in binary mode if I really have to. Being able to read whole lines at once is very comfortable. And since I do have headers which contain the content-length I just read "content-length" amount of bytes with the BinaryReader, then switch back to read in text mode again.

Is there any way to figure out how much data is going to be sent? (The NetworkStream doesn't support the Length property, but maybe the socket has one) That way I could at least figure out if the data is sent over the network or if there is still something "stuck" somewhere.
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 10757281
How do you know where to start reading binary data? You cannot rely on the stream position property since the StreamReader reads ahead!
0
 

Author Comment

by:dave_the_brave
ID: 10757313
Well, I have that header I add to every "data part" in the message. I can read the "Content-Length" header and then read until I find an empty newline. Then I switch, read "Content-Length" bytes from the stream in binary mode (BinaryReader) and then switch back to the StreamReader to read until I find another "data part" with an appropriate header or the end of the message is received.

I don't have any other idea on how to do this. Any ideas?
0
 
LVL 10

Accepted Solution

by:
ptmcomp earned 125 total points
ID: 10760109
You cannot seek a networkstream. May be it works if you read it in a memorystream and then use the procedure you described above to read the data from the memorystream. Since the StreamReader will read ahead, the position the BinaryReader should continue with is already skipped.
0
 

Author Comment

by:dave_the_brave
ID: 10763977
Thanks! Duh! It makes sense now. I just assumed that since the Stream stores the Position, the BinaryReader could just continue where the StreamReader left off. I'll just read everything in binary and loop through each line manually. Thanks for the help!
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Article by: Ivo
C# And Nullable Types Since 2.0 C# has Nullable(T) Generic Structure. The idea behind is to allow value type objects to have null values just like reference types have. This concerns scenarios where not all data sources have values (like a databa…
We all know that functional code is the leg that any good program stands on when it comes right down to it, however, if your program lacks a good user interface your product may not have the appeal needed to keep your customers happy. This issue can…
This Micro Tutorial will give you a basic overview how to record your screen with Microsoft Expression Encoder. This program is still free and open for the public to download. This will be demonstrated using Microsoft Expression Encoder 4.
Learn how to create flexible layouts using relative units in CSS.  New relative units added in CSS3 include vw(viewports width), vh(viewports height), vmin(minimum of viewports height and width), and vmax (maximum of viewports height and width).

920 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now