BinaryReader.Read function doesn't return the last couple of bytes in a NetworkStream (TcpListener) (Large amount of data 5MB+)

I'm trying to read mixed data (text & binary) from a network stream. The data format i have chosen is similar to the mail format:

(Text in Brackets [ ] is substituted with appropriate data)

Message-ID: <[Hash]@[Sender]>
Date: [DateTime UTC]
Content-Type: multipart/mixed
Boundary: ------=_NextPart_[Hash]
Content-Part-Count: [Part Count];
{Additional headers}

[Message Text goes here]

------=_NextPart_[Hash]
Content-Length: [Length]
Content-Encoding: [Encoding]
Content-MD5Checksum: [MD5 Checksum]
{Additional headers}

[PartData (can be binary)]

------=_NextPart_[Hash]--

{Additional parts}

{EOF}

Reading the text part is easy. StreamReader.ReadLine() does the trick. However, when I come to the data parts, I'm switching to the BinaryReader. Since both readers use the underlaying NetworkStream I can use either one and be at the correct position within the stream. The problem occurs when I'm trying to read the binary data. I first read the "Content-Length" header of the part and then read the byte amount of that from the stream. I'm just reading the data from the stream, adding it to a byte array as we read. Since I'm reading over the network (Internet), I'm looping until I have read the right amount of bytes from the stream, then switching back to the StreamReader to read the next part, if there is one.
Having tried various things, I always seem to end up a few bytes short of data on the reciever side.

This is the code used to read the binary part of the message:

// read the data from the NetworkStream
[...]

// now read the data part

// read "Content-Length" from the header HashTable
int datalength = int.Parse(datapacket.GetHeaderValue("Content-Length"));

// create the byte array to hold the binary data
byte[] data = new byte[datalength];

int bytesRead = 0;
int bytesToRead = datalength;
int totalBytesRead = 0;

// set a timeout so we don't wait forever
TimeSpan timeout = new TimeSpan(0, 0, 5, 0, 0);
DateTime starttime = DateTime.Now;

// create the binary reader
BinaryReader datareader = new BinaryReader(networkStream, Encoding.UTF8);

// read until we have read all we need or until we readched our timeout
while((bytesToRead > 0) && (starttime.Add(timeout) > DateTime.Now))
{
      // read as long
      bytesRead = datareader.Read(data, bytesRead, bytesToRead);

      totalBytesRead += bytesRead;
      bytesToRead = datalength - totalBytesRead;

      // reset the timeout
      starttime = DateTime.Now;

      // let other processes do some work
      Thread.Sleep(10);
}

// check the data (MD5 checksum, length, etc.)
[...]

All my tries have failed. I always have about 430 bytes left to read (bytesToRead), even though the stream should contain a few bytes more than the binary data part. There should be a "------=_NextPart_[Hash]--" and a newline following.

Any ideas on what might be wrong?

The MSDN KnowledgeBase has an article on a bug in the BinaryReader.ReadChar() function at http://support.microsoft.com/default.aspx?scid=kb;en-us;318121 - Maybe this could be something similar?

Thanks for the help!
dave_the_braveAsked:
Who is Participating?

[Webinar] Streamline your web hosting managementRegister Today

x
 
ptmcompConnect With a Mentor Commented:
You cannot seek a networkstream. May be it works if you read it in a memorystream and then use the procedure you described above to read the data from the memorystream. Since the StreamReader will read ahead, the position the BinaryReader should continue with is already skipped.
0
 
_TAD_Commented:


you'll probably need to flush your data... that is, you sometimes need to explicitly tell your object to WRITE data to the byte[] and not just point to it.

  // read as long -- points to data
     bytesRead = datareader.Read(data, bytesRead, bytesToRead);
  // writes data to buffer (aka "data")
     datareader.BaseStream.Flush();  

   
0
 
ptmcompCommented:
The writer needs to flush - not the reader
Change this:  bytesRead = datareader.Read(data, bytesRead, bytesToRead);
to this:  bytesRead = datareader.Read(data, bytesRead, 2048);

Place this line:
starttime = DateTime.Now;
before the Read(..) - else the timeout will not work.
0
The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

 
dave_the_braveAuthor Commented:
Thanks for the help!

It still doesn't work, though. I first thought I didn't write the data to the networkstream properly. I then just wrote the data i would send over the network to a file. the file is exactly the way it should be. Then I changed it back to write it to the networkstream (instead of the filestream).
Again! I still don't get all the data on the recipient side. I have no idea what the problem is. I did try to flush the data, however on a network stream, it didn't seem to do much.
So I did some research: according to the MSDN Library, a networkstream is not buffered, therefore the flush method doesn't do anything (it is reserved for future use).

I sent a 5MB file over the network (used BinaryWriter and BinaryReader) without the StreamReader or StreamWriter and it worked just fine.

Could it be that the use of a StreamReader and a BinaryReader on the same stream causes this strange effect?

I also use UTF8 Encoding. Maybe that could have something to do with it. I did a test on a 5MB file in different encodings. The file stayed the same size. So I'm guessing this doesn't have anything to do with it - or I'm doing something seriously wrong.
0
 
ptmcompCommented:
Why don't you use BinaryWriter/BinaryReader? You wrote that the stream contains binary data (not base64 encoded). May be the StreamReader causes troubles when reading the binary data.
0
 
dave_the_braveAuthor Commented:
I do use both. I read the text with StreamReader.ReadLine() and then switch to the BinaryReader when I know the data starts. I only want to read in binary mode if I really have to. Being able to read whole lines at once is very comfortable. And since I do have headers which contain the content-length I just read "content-length" amount of bytes with the BinaryReader, then switch back to read in text mode again.

Is there any way to figure out how much data is going to be sent? (The NetworkStream doesn't support the Length property, but maybe the socket has one) That way I could at least figure out if the data is sent over the network or if there is still something "stuck" somewhere.
0
 
ptmcompCommented:
How do you know where to start reading binary data? You cannot rely on the stream position property since the StreamReader reads ahead!
0
 
dave_the_braveAuthor Commented:
Well, I have that header I add to every "data part" in the message. I can read the "Content-Length" header and then read until I find an empty newline. Then I switch, read "Content-Length" bytes from the stream in binary mode (BinaryReader) and then switch back to the StreamReader to read until I find another "data part" with an appropriate header or the end of the message is received.

I don't have any other idea on how to do this. Any ideas?
0
 
dave_the_braveAuthor Commented:
Thanks! Duh! It makes sense now. I just assumed that since the Stream stores the Position, the BinaryReader could just continue where the StreamReader left off. I'll just read everything in binary and loop through each line manually. Thanks for the help!
0
All Courses

From novice to tech pro — start learning today.