Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

BinaryReader.Read function doesn't return the last couple of bytes in a NetworkStream (TcpListener) (Large amount of data 5MB+)

Posted on 2004-04-02
9
Medium Priority
?
1,638 Views
Last Modified: 2012-06-21
I'm trying to read mixed data (text & binary) from a network stream. The data format i have chosen is similar to the mail format:

(Text in Brackets [ ] is substituted with appropriate data)

Message-ID: <[Hash]@[Sender]>
Date: [DateTime UTC]
Content-Type: multipart/mixed
Boundary: ------=_NextPart_[Hash]
Content-Part-Count: [Part Count];
{Additional headers}

[Message Text goes here]

------=_NextPart_[Hash]
Content-Length: [Length]
Content-Encoding: [Encoding]
Content-MD5Checksum: [MD5 Checksum]
{Additional headers}

[PartData (can be binary)]

------=_NextPart_[Hash]--

{Additional parts}

{EOF}

Reading the text part is easy. StreamReader.ReadLine() does the trick. However, when I come to the data parts, I'm switching to the BinaryReader. Since both readers use the underlaying NetworkStream I can use either one and be at the correct position within the stream. The problem occurs when I'm trying to read the binary data. I first read the "Content-Length" header of the part and then read the byte amount of that from the stream. I'm just reading the data from the stream, adding it to a byte array as we read. Since I'm reading over the network (Internet), I'm looping until I have read the right amount of bytes from the stream, then switching back to the StreamReader to read the next part, if there is one.
Having tried various things, I always seem to end up a few bytes short of data on the reciever side.

This is the code used to read the binary part of the message:

// read the data from the NetworkStream
[...]

// now read the data part

// read "Content-Length" from the header HashTable
int datalength = int.Parse(datapacket.GetHeaderValue("Content-Length"));

// create the byte array to hold the binary data
byte[] data = new byte[datalength];

int bytesRead = 0;
int bytesToRead = datalength;
int totalBytesRead = 0;

// set a timeout so we don't wait forever
TimeSpan timeout = new TimeSpan(0, 0, 5, 0, 0);
DateTime starttime = DateTime.Now;

// create the binary reader
BinaryReader datareader = new BinaryReader(networkStream, Encoding.UTF8);

// read until we have read all we need or until we readched our timeout
while((bytesToRead > 0) && (starttime.Add(timeout) > DateTime.Now))
{
      // read as long
      bytesRead = datareader.Read(data, bytesRead, bytesToRead);

      totalBytesRead += bytesRead;
      bytesToRead = datalength - totalBytesRead;

      // reset the timeout
      starttime = DateTime.Now;

      // let other processes do some work
      Thread.Sleep(10);
}

// check the data (MD5 checksum, length, etc.)
[...]

All my tries have failed. I always have about 430 bytes left to read (bytesToRead), even though the stream should contain a few bytes more than the binary data part. There should be a "------=_NextPart_[Hash]--" and a newline following.

Any ideas on what might be wrong?

The MSDN KnowledgeBase has an article on a bug in the BinaryReader.ReadChar() function at http://support.microsoft.com/default.aspx?scid=kb;en-us;318121 - Maybe this could be something similar?

Thanks for the help!
0
Comment
Question by:dave_the_brave
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 4
9 Comments
 
LVL 22

Expert Comment

by:_TAD_
ID: 10741062


you'll probably need to flush your data... that is, you sometimes need to explicitly tell your object to WRITE data to the byte[] and not just point to it.

  // read as long -- points to data
     bytesRead = datareader.Read(data, bytesRead, bytesToRead);
  // writes data to buffer (aka "data")
     datareader.BaseStream.Flush();  

   
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 10753051
The writer needs to flush - not the reader
Change this:  bytesRead = datareader.Read(data, bytesRead, bytesToRead);
to this:  bytesRead = datareader.Read(data, bytesRead, 2048);

Place this line:
starttime = DateTime.Now;
before the Read(..) - else the timeout will not work.
0
 

Author Comment

by:dave_the_brave
ID: 10756950
Thanks for the help!

It still doesn't work, though. I first thought I didn't write the data to the networkstream properly. I then just wrote the data i would send over the network to a file. the file is exactly the way it should be. Then I changed it back to write it to the networkstream (instead of the filestream).
Again! I still don't get all the data on the recipient side. I have no idea what the problem is. I did try to flush the data, however on a network stream, it didn't seem to do much.
So I did some research: according to the MSDN Library, a networkstream is not buffered, therefore the flush method doesn't do anything (it is reserved for future use).

I sent a 5MB file over the network (used BinaryWriter and BinaryReader) without the StreamReader or StreamWriter and it worked just fine.

Could it be that the use of a StreamReader and a BinaryReader on the same stream causes this strange effect?

I also use UTF8 Encoding. Maybe that could have something to do with it. I did a test on a 5MB file in different encodings. The file stayed the same size. So I'm guessing this doesn't have anything to do with it - or I'm doing something seriously wrong.
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
LVL 10

Expert Comment

by:ptmcomp
ID: 10757088
Why don't you use BinaryWriter/BinaryReader? You wrote that the stream contains binary data (not base64 encoded). May be the StreamReader causes troubles when reading the binary data.
0
 

Author Comment

by:dave_the_brave
ID: 10757242
I do use both. I read the text with StreamReader.ReadLine() and then switch to the BinaryReader when I know the data starts. I only want to read in binary mode if I really have to. Being able to read whole lines at once is very comfortable. And since I do have headers which contain the content-length I just read "content-length" amount of bytes with the BinaryReader, then switch back to read in text mode again.

Is there any way to figure out how much data is going to be sent? (The NetworkStream doesn't support the Length property, but maybe the socket has one) That way I could at least figure out if the data is sent over the network or if there is still something "stuck" somewhere.
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 10757281
How do you know where to start reading binary data? You cannot rely on the stream position property since the StreamReader reads ahead!
0
 

Author Comment

by:dave_the_brave
ID: 10757313
Well, I have that header I add to every "data part" in the message. I can read the "Content-Length" header and then read until I find an empty newline. Then I switch, read "Content-Length" bytes from the stream in binary mode (BinaryReader) and then switch back to the StreamReader to read until I find another "data part" with an appropriate header or the end of the message is received.

I don't have any other idea on how to do this. Any ideas?
0
 
LVL 10

Accepted Solution

by:
ptmcomp earned 500 total points
ID: 10760109
You cannot seek a networkstream. May be it works if you read it in a memorystream and then use the procedure you described above to read the data from the memorystream. Since the StreamReader will read ahead, the position the BinaryReader should continue with is already skipped.
0
 

Author Comment

by:dave_the_brave
ID: 10763977
Thanks! Duh! It makes sense now. I just assumed that since the Stream stores the Position, the BinaryReader could just continue where the StreamReader left off. I'll just read everything in binary and loop through each line manually. Thanks for the help!
0

Featured Post

New feature and membership benefit!

New feature! Upgrade and increase expert visibility of your issues with Priority Questions.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In order to hide the "ugly" records selectors (triangles) in the rowheaders, here are some suggestions. Microsoft doesn't have a direct method/property to do it. You can only hide the rowheader column. First solution, the easy way The first sol…
Real-time is more about the business, not the technology. In day-to-day life, to make real-time decisions like buying or investing, business needs the latest information(e.g. Gold Rate/Stock Rate). Unlike traditional days, you need not wait for a fe…
In this video you will find out how to export Office 365 mailboxes using the built in eDiscovery tool. Bear in mind that although this method might be useful in some cases, using PST files as Office 365 backup is troublesome in a long run (more on t…
In this video, Percona Director of Solution Engineering Jon Tobin discusses the function and features of Percona Server for MongoDB. How Percona can help Percona can help you determine if Percona Server for MongoDB is the right solution for …
Suggested Courses

618 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question