Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Filestream Out of Memory Error

Posted on 2004-04-26
7
Medium Priority
?
1,288 Views
Last Modified: 2012-05-04
I am running VB.NET my machine has 2GB of ram. My program extracts and sorts large amounts of financial data (1.5GB+). I have tried using the filestream and stremreader methods. I consistantly recieve 'System.OutOfMemoryException'. I have duplicated this error on blank form with only a command button containg the code. I hit the button after each iteration (replicating the loop in my program) and watch the RAM in the task manager keep going up until it hits the 2GB marker, then I receive the out of memory exception.This happens 9/10 times. Sometimes the ram gets to about 1.5GB then the system finally releases the unused resources. There is no pattern or set amount of time to when it decides to finally realease the unused resources. It seems like the OS isn't receiving the message to release the data. The file tradefile.l is 385MB. It is from the New York Stock Exchange TAQ Database and was obtained via FTP. I ran this code using the FSO in VB 6.0 without error for months. I would rather use the filestream method because it is about 5X faster than the FSO. Thank You.

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
  Dim s As String
  Dim readBytes(91000000) As Byte
  Dim fsstream As New FileStream("D:\TRADE DATA\TESTING\TEMP _
  FILES\tradefile.l", FileMode.Open, FileAccess.Read)
  fsstream.Read(readBytes, 0, 90999999)

  s = System.Text.Encoding.ASCII.GetString(readBytes)

  fsstream.Flush()
  fsstream.Close()
  s = ""
  Erase readBytes
End Sub



0
Comment
Question by:EBarefoot
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
7 Comments
 
LVL 19

Expert Comment

by:BrianGEFF719
ID: 10924915
do you really need to load the entire 1.5gb into memory, is this really neccesarry, can you not load it in 15 100mb chunks and process it as you go...


-Brian
0
 

Author Comment

by:EBarefoot
ID: 10925103
I am already doing that. The code above is what I am using in the program and if I am correct it is just less then 100MB. Thank you for the suggestion.
0
 

Author Comment

by:EBarefoot
ID: 10925372
SOLUTION:

System.GC.Collect()

The garbage collector is a service that automatically reclaims unused memory.
The Collect() method forces a garbage collection rather letting the system pick the best time.

How do I close this thread? Thank You.
0
 
LVL 19

Expert Comment

by:BrianGEFF719
ID: 10925547
Go ahead and goto the Community Support area and post a link to this question requesting that it be delete and points refunded.


-Brian
0
 
LVL 5

Accepted Solution

by:
Netminder earned 0 total points
ID: 11256534
User resolved; closed, 250 points refunded.

Netminder
Site Admin
0

Featured Post

Enroll in October's Free Course of the Month

Do you work with and analyze data? Enroll in October's Course of the Month for 7+ hours of SQL training, allowing you to quickly and efficiently store or retrieve data. It's free for Premium Members, Team Accounts, and Qualified Experts!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Since upgrading to Office 2013 or higher installing the Smart Indenter addin will fail. This article will explain how to install it so it will work regardless of the Office version installed.
If you need to start windows update installation remotely or as a scheduled task you will find this very helpful.
Get people started with the process of using Access VBA to control Excel using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Excel. Using automation, an Access application can laun…
Get people started with the utilization of class modules. Class modules can be a powerful tool in Microsoft Access. They allow you to create self-contained objects that encapsulate functionality. They can easily hide the complexity of a process from…
Suggested Courses

597 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question