Solved

BinaryReader.Read function doesn't return the last couple of bytes in a NetworkStream (TcpListener) (Large amount of data 5MB+)

Posted on 2004-04-02
9
1,618 Views
Last Modified: 2012-06-21
I'm trying to read mixed data (text & binary) from a network stream. The data format i have chosen is similar to the mail format:

(Text in Brackets [ ] is substituted with appropriate data)

Message-ID: <[Hash]@[Sender]>
Date: [DateTime UTC]
Content-Type: multipart/mixed
Boundary: ------=_NextPart_[Hash]
Content-Part-Count: [Part Count];
{Additional headers}

[Message Text goes here]

------=_NextPart_[Hash]
Content-Length: [Length]
Content-Encoding: [Encoding]
Content-MD5Checksum: [MD5 Checksum]
{Additional headers}

[PartData (can be binary)]

------=_NextPart_[Hash]--

{Additional parts}

{EOF}

Reading the text part is easy. StreamReader.ReadLine() does the trick. However, when I come to the data parts, I'm switching to the BinaryReader. Since both readers use the underlaying NetworkStream I can use either one and be at the correct position within the stream. The problem occurs when I'm trying to read the binary data. I first read the "Content-Length" header of the part and then read the byte amount of that from the stream. I'm just reading the data from the stream, adding it to a byte array as we read. Since I'm reading over the network (Internet), I'm looping until I have read the right amount of bytes from the stream, then switching back to the StreamReader to read the next part, if there is one.
Having tried various things, I always seem to end up a few bytes short of data on the reciever side.

This is the code used to read the binary part of the message:

// read the data from the NetworkStream
[...]

// now read the data part

// read "Content-Length" from the header HashTable
int datalength = int.Parse(datapacket.GetHeaderValue("Content-Length"));

// create the byte array to hold the binary data
byte[] data = new byte[datalength];

int bytesRead = 0;
int bytesToRead = datalength;
int totalBytesRead = 0;

// set a timeout so we don't wait forever
TimeSpan timeout = new TimeSpan(0, 0, 5, 0, 0);
DateTime starttime = DateTime.Now;

// create the binary reader
BinaryReader datareader = new BinaryReader(networkStream, Encoding.UTF8);

// read until we have read all we need or until we readched our timeout
while((bytesToRead > 0) && (starttime.Add(timeout) > DateTime.Now))
{
      // read as long
      bytesRead = datareader.Read(data, bytesRead, bytesToRead);

      totalBytesRead += bytesRead;
      bytesToRead = datalength - totalBytesRead;

      // reset the timeout
      starttime = DateTime.Now;

      // let other processes do some work
      Thread.Sleep(10);
}

// check the data (MD5 checksum, length, etc.)
[...]

All my tries have failed. I always have about 430 bytes left to read (bytesToRead), even though the stream should contain a few bytes more than the binary data part. There should be a "------=_NextPart_[Hash]--" and a newline following.

Any ideas on what might be wrong?

The MSDN KnowledgeBase has an article on a bug in the BinaryReader.ReadChar() function at http://support.microsoft.com/default.aspx?scid=kb;en-us;318121 - Maybe this could be something similar?

Thanks for the help!
0
Comment
Question by:dave_the_brave
  • 4
  • 4
9 Comments
 
LVL 22

Expert Comment

by:_TAD_
Comment Utility


you'll probably need to flush your data... that is, you sometimes need to explicitly tell your object to WRITE data to the byte[] and not just point to it.

  // read as long -- points to data
     bytesRead = datareader.Read(data, bytesRead, bytesToRead);
  // writes data to buffer (aka "data")
     datareader.BaseStream.Flush();  

   
0
 
LVL 10

Expert Comment

by:ptmcomp
Comment Utility
The writer needs to flush - not the reader
Change this:  bytesRead = datareader.Read(data, bytesRead, bytesToRead);
to this:  bytesRead = datareader.Read(data, bytesRead, 2048);

Place this line:
starttime = DateTime.Now;
before the Read(..) - else the timeout will not work.
0
 

Author Comment

by:dave_the_brave
Comment Utility
Thanks for the help!

It still doesn't work, though. I first thought I didn't write the data to the networkstream properly. I then just wrote the data i would send over the network to a file. the file is exactly the way it should be. Then I changed it back to write it to the networkstream (instead of the filestream).
Again! I still don't get all the data on the recipient side. I have no idea what the problem is. I did try to flush the data, however on a network stream, it didn't seem to do much.
So I did some research: according to the MSDN Library, a networkstream is not buffered, therefore the flush method doesn't do anything (it is reserved for future use).

I sent a 5MB file over the network (used BinaryWriter and BinaryReader) without the StreamReader or StreamWriter and it worked just fine.

Could it be that the use of a StreamReader and a BinaryReader on the same stream causes this strange effect?

I also use UTF8 Encoding. Maybe that could have something to do with it. I did a test on a 5MB file in different encodings. The file stayed the same size. So I'm guessing this doesn't have anything to do with it - or I'm doing something seriously wrong.
0
 
LVL 10

Expert Comment

by:ptmcomp
Comment Utility
Why don't you use BinaryWriter/BinaryReader? You wrote that the stream contains binary data (not base64 encoded). May be the StreamReader causes troubles when reading the binary data.
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 

Author Comment

by:dave_the_brave
Comment Utility
I do use both. I read the text with StreamReader.ReadLine() and then switch to the BinaryReader when I know the data starts. I only want to read in binary mode if I really have to. Being able to read whole lines at once is very comfortable. And since I do have headers which contain the content-length I just read "content-length" amount of bytes with the BinaryReader, then switch back to read in text mode again.

Is there any way to figure out how much data is going to be sent? (The NetworkStream doesn't support the Length property, but maybe the socket has one) That way I could at least figure out if the data is sent over the network or if there is still something "stuck" somewhere.
0
 
LVL 10

Expert Comment

by:ptmcomp
Comment Utility
How do you know where to start reading binary data? You cannot rely on the stream position property since the StreamReader reads ahead!
0
 

Author Comment

by:dave_the_brave
Comment Utility
Well, I have that header I add to every "data part" in the message. I can read the "Content-Length" header and then read until I find an empty newline. Then I switch, read "Content-Length" bytes from the stream in binary mode (BinaryReader) and then switch back to the StreamReader to read until I find another "data part" with an appropriate header or the end of the message is received.

I don't have any other idea on how to do this. Any ideas?
0
 
LVL 10

Accepted Solution

by:
ptmcomp earned 125 total points
Comment Utility
You cannot seek a networkstream. May be it works if you read it in a memorystream and then use the procedure you described above to read the data from the memorystream. Since the StreamReader will read ahead, the position the BinaryReader should continue with is already skipped.
0
 

Author Comment

by:dave_the_brave
Comment Utility
Thanks! Duh! It makes sense now. I just assumed that since the Stream stores the Position, the BinaryReader could just continue where the StreamReader left off. I'll just read everything in binary and loop through each line manually. Thanks for the help!
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Introduction This article series is supposed to shed some light on the use of IDisposable and objects that inherit from it. In essence, a more apt title for this article would be: using (IDisposable) {}. I’m just not sure how many people would ge…
Introduction Although it is an old technology, serial ports are still being used by many hardware manufacturers. If you develop applications in C#, Microsoft .NET framework has SerialPort class to communicate with the serial ports.  I needed to…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
This video explains how to create simple products associated to Magento configurable product and offers fast way of their generation with Store Manager for Magento tool.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

6 Experts available now in Live!

Get 1:1 Help Now