Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1341
  • Last Modified:

Filestream Out of Memory Error

I am running VB.NET my machine has 2GB of ram. My program extracts and sorts large amounts of financial data (1.5GB+). I have tried using the filestream and stremreader methods. I consistantly recieve 'System.OutOfMemoryException'. I have duplicated this error on blank form with only a command button containg the code. I hit the button after each iteration (replicating the loop in my program) and watch the RAM in the task manager keep going up until it hits the 2GB marker, then I receive the out of memory exception.This happens 9/10 times. Sometimes the ram gets to about 1.5GB then the system finally releases the unused resources. There is no pattern or set amount of time to when it decides to finally realease the unused resources. It seems like the OS isn't receiving the message to release the data. The file tradefile.l is 385MB. It is from the New York Stock Exchange TAQ Database and was obtained via FTP. I ran this code using the FSO in VB 6.0 without error for months. I would rather use the filestream method because it is about 5X faster than the FSO. Thank You.

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
  Dim s As String
  Dim readBytes(91000000) As Byte
  Dim fsstream As New FileStream("D:\TRADE DATA\TESTING\TEMP _
  FILES\tradefile.l", FileMode.Open, FileAccess.Read)
  fsstream.Read(readBytes, 0, 90999999)

  s = System.Text.Encoding.ASCII.GetString(readBytes)

  fsstream.Flush()
  fsstream.Close()
  s = ""
  Erase readBytes
End Sub



0
EBarefoot
Asked:
EBarefoot
  • 2
  • 2
1 Solution
 
BrianGEFF719Commented:
do you really need to load the entire 1.5gb into memory, is this really neccesarry, can you not load it in 15 100mb chunks and process it as you go...


-Brian
0
 
EBarefootAuthor Commented:
I am already doing that. The code above is what I am using in the program and if I am correct it is just less then 100MB. Thank you for the suggestion.
0
 
EBarefootAuthor Commented:
SOLUTION:

System.GC.Collect()

The garbage collector is a service that automatically reclaims unused memory.
The Collect() method forces a garbage collection rather letting the system pick the best time.

How do I close this thread? Thank You.
0
 
BrianGEFF719Commented:
Go ahead and goto the Community Support area and post a link to this question requesting that it be delete and points refunded.


-Brian
0
 
NetminderCommented:
User resolved; closed, 250 points refunded.

Netminder
Site Admin
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now