?
Solved

Stream Reader out of memory exception

Posted on 2011-03-14
15
Medium Priority
?
1,924 Views
Last Modified: 2012-06-27
I am using StreamReader to read in a text file in VB.net 2005
My problem is at the.last line 1 text file is malformed
it does not have a LFCR
so it keeps reading

I end up with 150 Megs read in and then it throws this error

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.String.GetStringForStringBuilder(String value, Int32 startIndex, Int32 length, Int32 capacity)
   at System.Text.StringBuilder.GetNewString(String currentString, Int32 requiredLength)
   at System.Text.StringBuilder.Append(Char[] value, Int32 startIndex, Int32 charCount)
   at System.IO.StreamReader.ReadLine()


My lines are at most 164 characters not counting the LFCR

I need a way to tell stream reader to stop reading after a 165 characters
any idea


Dim sr As StreamReader = File.OpenText(filename)
Do Until sr.Peek = -1
linecontent = sr.ReadLine
If Trim(linecontent) <> "" Then
   

If Trim(Mid(linecontent, 79, 8)) <> "" Then
aDate1 = Mid(linecontent, 79, 8)
aDate1 = Mid(aDate1, 5, 2) & "/" & Mid(aDate1, 7, 2) & "/" & Mid(aDate1, 1, 4)
Else
aDate1 = "12/31/1849"
 End If

If CDate(aDate1) >= dDate
sGuest = Mid(linecontent, 1, 6) & "0"
End IF
'''''''''''''''''

Other code here that works fine
'''''''''''''''''

End IF
linecontent = String.Empty
Loop

Open in new window

0
Comment
Question by:hainansyndrome
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 5
  • 2
  • +1
15 Comments
 
LVL 9

Expert Comment

by:Hawkvalley1
ID: 35133201
Try this method:

Using sr As New StreamReader(<path>)
  While Not sr.EndOfStream
     'your code here for parsing
  End While
End Using 'this closes and disposes the stream

Open in new window


PS be sure to close your streams.
0
 

Author Comment

by:hainansyndrome
ID: 35133277
Hawkvalley1:  
how do I setup the Stream Reader part?
I am trying

            Dim sr As StreamReader
            Using sr As New StreamReader=File.OpenText((destination & "\" & filename)

but i get end of Statement expected?
0
 

Author Comment

by:hainansyndrome
ID: 35133329
even
sing sr As New StreamReader=File.OpenText((destination & "\" & filename)


is not working
0
Get free NFR key for Veeam Availability Suite 9.5

Veeam is happy to provide a free NFR license (1 year, 2 sockets) to all certified IT Pros. The license allows for the non-production use of Veeam Availability Suite v9.5 in your home lab, without any feature limitations. It works for both VMware and Hyper-V environments

 

Author Comment

by:hainansyndrome
ID: 35133398
ok never mind i have dain bramage

but that did not help
it still hit the last line and blew up

see the last line is about 170 megs since there are no LFCR
0
 
LVL 86

Expert Comment

by:Mike Tomlinson
ID: 35133404
May we see all the code?...
0
 

Author Comment

by:hainansyndrome
ID: 35133439
I have shown all the relevant code
the problem is on the last line of the text file
0
 
LVL 86

Expert Comment

by:Mike Tomlinson
ID: 35133473
You've shown all the code that YOU think is relevant...but you were unable to solve the problem.  If you show us all the code we may spot a problem that you are unaware of.  Just saying...
0
 

Author Comment

by:hainansyndrome
ID: 35133532
Idle_Mind: I have show you all the relevant code
I know with 10000000% certainty the remaining code has no flaws
I have trappend this error
I know when and where it is occuring
0
 
LVL 86

Expert Comment

by:Mike Tomlinson
ID: 35133927
The "OutOfMemoryException" part throws up a red flag.  It indicates that you were low on memory, which shouldn't be the case if you are only reading LINE BY LINE and each line is no longer than 164 chars.

Are you by chance, in that code you are sure is not the problem, building up a huge value with simple string concatenation?

    str = str & someDataExtractedFromEachLine

Or in short hand:

    str &= someDataExtractedFromEachLine
0
 
LVL 9

Expert Comment

by:Hawkvalley1
ID: 35134895
Why is the last line 170 megs? Maybe you need to rethink the file design.
0
 
LVL 86

Expert Comment

by:Mike Tomlinson
ID: 35134917
If you need to control how much is read at a time then use a straight FileStream() and then you can read by a specified number of bytes:
http://msdn.microsoft.com/en-us/library/system.io.filestream.read.aspx

...but then you lose the built-in ability to read only the next line.  You have to manually look for the CRLF 's in the returned array of bytes and break up the data yourself.
0
 
LVL 18

Accepted Solution

by:
deighton earned 500 total points
ID: 35136902
do you need to use a binary reader?
0
 

Author Comment

by:hainansyndrome
ID: 35137444
I fixed it
must have been the fact that I had 8 VS windows open or that I have not rebooted in 4 days
after reboot the  .substring showed up

I was able to use
Dim sLen As Integer = 160
                   
linecontent = sr.ReadLine.Substring(0, sLen)
I then later incremented up to 164 characters using a Try Catch
0
 
LVL 86

Expert Comment

by:Mike Tomlinson
ID: 35137548
Just for clarification, though, if the last line really is ~super~ long...your method directly above is NOT reading just 160 characters.  It is still reading the ENTIRE line and then Substring() is simply returning the requested number of chars.  This apparently is only working because you have enough memory available.  To truly read only the limited number of chars you'd have to use the lower level FileStream.Read() method as I mentioned earlier, or you something similar like BinaryReader() as deighton suggests.
0
 

Author Comment

by:hainansyndrome
ID: 35139286
I stand corrected
same problem
it hits the last line and bam

0

Featured Post

PowerShell Core for Advanced Linux Administrators

Understand advanced principals around Powershell Core with a focus on the Linux Administrator.  This course covers how to administer numerous environments across multiple platforms including Linux, Azure, AWS, and Google Cloud from a single shell instance.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In my previous article (http://www.experts-exchange.com/Programming/Languages/.NET/.NET_Framework_3.x/A_4362-Serialization-in-NET-1.html) we saw the basics of serialization and how types/objects can be serialized to Binary format. In this blog we wi…
A long time ago (May 2011), I have written an article showing you how to create a DLL using Visual Studio 2005 to be hosted in SQL Server 2005. That was valid at that time and it is still valid if you are still using these versions. You can still re…
In this video you will find out how to export Office 365 mailboxes using the built in eDiscovery tool. Bear in mind that although this method might be useful in some cases, using PST files as Office 365 backup is troublesome in a long run (more on t…
Have you created a query with information for a calendar? ... and then, abra-cadabra, the calendar is done?! I am going to show you how to make that happen. Visualize your data!  ... really see it To use the code to create a calendar from a q…
Suggested Courses

801 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question