Solved

BinaryReader.Read function doesn't return the last couple of bytes in a NetworkStream (TcpListener) (Large amount of data 5MB+)

Posted on 2004-04-02
9
1,625 Views
Last Modified: 2012-06-21
I'm trying to read mixed data (text & binary) from a network stream. The data format i have chosen is similar to the mail format:

(Text in Brackets [ ] is substituted with appropriate data)

Message-ID: <[Hash]@[Sender]>
Date: [DateTime UTC]
Content-Type: multipart/mixed
Boundary: ------=_NextPart_[Hash]
Content-Part-Count: [Part Count];
{Additional headers}

[Message Text goes here]

------=_NextPart_[Hash]
Content-Length: [Length]
Content-Encoding: [Encoding]
Content-MD5Checksum: [MD5 Checksum]
{Additional headers}

[PartData (can be binary)]

------=_NextPart_[Hash]--

{Additional parts}

{EOF}

Reading the text part is easy. StreamReader.ReadLine() does the trick. However, when I come to the data parts, I'm switching to the BinaryReader. Since both readers use the underlaying NetworkStream I can use either one and be at the correct position within the stream. The problem occurs when I'm trying to read the binary data. I first read the "Content-Length" header of the part and then read the byte amount of that from the stream. I'm just reading the data from the stream, adding it to a byte array as we read. Since I'm reading over the network (Internet), I'm looping until I have read the right amount of bytes from the stream, then switching back to the StreamReader to read the next part, if there is one.
Having tried various things, I always seem to end up a few bytes short of data on the reciever side.

This is the code used to read the binary part of the message:

// read the data from the NetworkStream
[...]

// now read the data part

// read "Content-Length" from the header HashTable
int datalength = int.Parse(datapacket.GetHeaderValue("Content-Length"));

// create the byte array to hold the binary data
byte[] data = new byte[datalength];

int bytesRead = 0;
int bytesToRead = datalength;
int totalBytesRead = 0;

// set a timeout so we don't wait forever
TimeSpan timeout = new TimeSpan(0, 0, 5, 0, 0);
DateTime starttime = DateTime.Now;

// create the binary reader
BinaryReader datareader = new BinaryReader(networkStream, Encoding.UTF8);

// read until we have read all we need or until we readched our timeout
while((bytesToRead > 0) && (starttime.Add(timeout) > DateTime.Now))
{
      // read as long
      bytesRead = datareader.Read(data, bytesRead, bytesToRead);

      totalBytesRead += bytesRead;
      bytesToRead = datalength - totalBytesRead;

      // reset the timeout
      starttime = DateTime.Now;

      // let other processes do some work
      Thread.Sleep(10);
}

// check the data (MD5 checksum, length, etc.)
[...]

All my tries have failed. I always have about 430 bytes left to read (bytesToRead), even though the stream should contain a few bytes more than the binary data part. There should be a "------=_NextPart_[Hash]--" and a newline following.

Any ideas on what might be wrong?

The MSDN KnowledgeBase has an article on a bug in the BinaryReader.ReadChar() function at http://support.microsoft.com/default.aspx?scid=kb;en-us;318121 - Maybe this could be something similar?

Thanks for the help!
0
Comment
Question by:dave_the_brave
  • 4
  • 4
9 Comments
 
LVL 22

Expert Comment

by:_TAD_
ID: 10741062


you'll probably need to flush your data... that is, you sometimes need to explicitly tell your object to WRITE data to the byte[] and not just point to it.

  // read as long -- points to data
     bytesRead = datareader.Read(data, bytesRead, bytesToRead);
  // writes data to buffer (aka "data")
     datareader.BaseStream.Flush();  

   
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 10753051
The writer needs to flush - not the reader
Change this:  bytesRead = datareader.Read(data, bytesRead, bytesToRead);
to this:  bytesRead = datareader.Read(data, bytesRead, 2048);

Place this line:
starttime = DateTime.Now;
before the Read(..) - else the timeout will not work.
0
 

Author Comment

by:dave_the_brave
ID: 10756950
Thanks for the help!

It still doesn't work, though. I first thought I didn't write the data to the networkstream properly. I then just wrote the data i would send over the network to a file. the file is exactly the way it should be. Then I changed it back to write it to the networkstream (instead of the filestream).
Again! I still don't get all the data on the recipient side. I have no idea what the problem is. I did try to flush the data, however on a network stream, it didn't seem to do much.
So I did some research: according to the MSDN Library, a networkstream is not buffered, therefore the flush method doesn't do anything (it is reserved for future use).

I sent a 5MB file over the network (used BinaryWriter and BinaryReader) without the StreamReader or StreamWriter and it worked just fine.

Could it be that the use of a StreamReader and a BinaryReader on the same stream causes this strange effect?

I also use UTF8 Encoding. Maybe that could have something to do with it. I did a test on a 5MB file in different encodings. The file stayed the same size. So I'm guessing this doesn't have anything to do with it - or I'm doing something seriously wrong.
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 10

Expert Comment

by:ptmcomp
ID: 10757088
Why don't you use BinaryWriter/BinaryReader? You wrote that the stream contains binary data (not base64 encoded). May be the StreamReader causes troubles when reading the binary data.
0
 

Author Comment

by:dave_the_brave
ID: 10757242
I do use both. I read the text with StreamReader.ReadLine() and then switch to the BinaryReader when I know the data starts. I only want to read in binary mode if I really have to. Being able to read whole lines at once is very comfortable. And since I do have headers which contain the content-length I just read "content-length" amount of bytes with the BinaryReader, then switch back to read in text mode again.

Is there any way to figure out how much data is going to be sent? (The NetworkStream doesn't support the Length property, but maybe the socket has one) That way I could at least figure out if the data is sent over the network or if there is still something "stuck" somewhere.
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 10757281
How do you know where to start reading binary data? You cannot rely on the stream position property since the StreamReader reads ahead!
0
 

Author Comment

by:dave_the_brave
ID: 10757313
Well, I have that header I add to every "data part" in the message. I can read the "Content-Length" header and then read until I find an empty newline. Then I switch, read "Content-Length" bytes from the stream in binary mode (BinaryReader) and then switch back to the StreamReader to read until I find another "data part" with an appropriate header or the end of the message is received.

I don't have any other idea on how to do this. Any ideas?
0
 
LVL 10

Accepted Solution

by:
ptmcomp earned 125 total points
ID: 10760109
You cannot seek a networkstream. May be it works if you read it in a memorystream and then use the procedure you described above to read the data from the memorystream. Since the StreamReader will read ahead, the position the BinaryReader should continue with is already skipped.
0
 

Author Comment

by:dave_the_brave
ID: 10763977
Thanks! Duh! It makes sense now. I just assumed that since the Stream stores the Position, the BinaryReader could just continue where the StreamReader left off. I'll just read everything in binary and loop through each line manually. Thanks for the help!
0

Featured Post

ScreenConnect 6.0 Free Trial

At ScreenConnect, partner feedback doesn't fall on deaf ears. We collected partner suggestions off of their virtual wish list and transformed them into one game-changing release: ScreenConnect 6.0. Explore all of the extras and enhancements for yourself!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
VB.NET (2008) - Refactoring Question 2 21
PrintingFoundIt(Almost!(Damn!)) 1 27
C# Desktop Application 3 36
Call windows 10 virtual keyboard from windows forms app 2 36
This article introduced a TextBox that supports transparent background.   Introduction TextBox is the most widely used control component in GUI design. Most GUI controls do not support transparent background and more or less do not have the…
This article aims to explain the working of CircularLogArchiver. This tool was designed to solve the buildup of log file in cases where systems do not support circular logging or where circular logging is not enabled
Established in 1997, Technology Architects has become one of the most reputable technology solutions companies in the country. TA have been providing businesses with cost effective state-of-the-art solutions and unparalleled service that is designed…
With Secure Portal Encryption, the recipient is sent a link to their email address directing them to the email laundry delivery page. From there, the recipient will be required to enter a user name and password to enter the page. Once the recipient …

821 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question