Solved

'Out of Memory' for LoadFromFile

Posted on 2013-10-28
11
2,069 Views
Last Modified: 2013-10-29
I need to load a 200MB disk text file into TStringlist. This gives 'Out of memory' at times.
I am using Delphi 2007 on win 7 with 16GB Ram computer.
0
Comment
Question by:Allan_Fernandes
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4
  • 2
11 Comments
 
LVL 14

Expert Comment

by:Pierre Cornelius
ID: 39607911
Can you explain more on what you need to do with the text file.

Also it would help if you tell us more about the file i.e. does it have a structure(e.g. fixed length records, csv, just a bunch of text, etc...)?

If you just need to read it and process the contents, then why not just use a TFilestream and process it buffer by buffer until you reach end of the file?
0
 
LVL 27

Expert Comment

by:Sinisa Vuk
ID: 39607913
Why you need to load such large file?
Use something like ATViewer for showing large file only.
0
 

Author Comment

by:Allan_Fernandes
ID: 39607915
I have backup application and I need to store details on several files in a text file for quick reference. The incremental patch (full file names , date, size ) has to be merged into the earlier text file. Sorting is required therefore stringlist is required.
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 27

Expert Comment

by:Sinisa Vuk
ID: 39607920
No, wrong way. For sorting purpose is using stringlist  very bad for such large text. Use simple database - maybe some standalone like SQLite3. Create table with indexes on field which will be sorted.
Then you have sorted text and easy filtering if you like.
0
 
LVL 14

Expert Comment

by:Pierre Cornelius
ID: 39607948
If you are already using a database, then rather store the file info in a table and use sql select commands to retrieve info and sort.

If you do not want to go the db route then you could build an index using an array or linked list to store the info you need e.g. filename, size, folder, etc. Then sort the array or linked list using a Quicksort and QuickFind combo. Let me know if you are interested in this approach so I can provide more info.
0
 

Author Comment

by:Allan_Fernandes
ID: 39607959
I am using custom sort and the efficiency if very good.
The same logic has been going on over 1000 machines for last few years without any issue.
Even 500,000 lines has not been a problem.
Suddenly this customer has started getting this issue at time of loading the Stringlist for which I am helpless. Also this customer has only about 100,000 files.
0
 
LVL 14

Expert Comment

by:Pierre Cornelius
ID: 39607978
what version of delphi are you using?

Can you show:
1) how and where your stringlist object is declared
2) the code loading the text file

Side-note: memory leak? is the file loaded once? or frequently? Are you freeing the stringlist object at some point?
0
 

Author Comment

by:Allan_Fernandes
ID: 39608067
I am using Delphi 2007 on win 7 with 16GB Ram computer.
The stringlist is declared within the procedure and the lines are
procedure TForm1.MergeControllerFile;
var
  sl_ControllerFile:TStringlist;
begin
  try
    sl_ControllerFile:=TStringlist.Create ;
    sl_ControllerFile.LoadFromFile(wFileName) ;  // The program does not go to next line
    ...
    ...
  finally
    FreeAndNil(sl_ControllerFile) ;
  end;

Open in new window

0
 
LVL 14

Accepted Solution

by:
Pierre Cornelius earned 500 total points
ID: 39608160
Looking at the code of TStringList, I noticed the LoadFromFile tries to load the entire thing in one go. I put together a procedure for you which will load it in numerous reads instead of just one big read of everything. try it and see if it solves your problem.

procedure SL_LoadFromFile(AFilename: string; AStringList: TStringList);
var fs: TFileStream;
    br, i, sz: integer;
    Buffer: TBytes;
    e, de: TEncoding;
begin
  //load the file data into the string in stages
  fs:= TFileStream.Create(AFilename, fmOpenRead OR fmShareDenyWrite);
  try
    i:= 0;
    SetLength(Buffer, fs.Size);
    repeat
      //let's read 65535 bytes at a time
      br:= fs.Read(buffer[i], 65535);
      i:= i + br;
    until (br < 65535);
  finally
    fs.Free;
  end;

  with AStringList do
  begin
    BeginUpdate;
    try
      e:= nil;
      de:= DefaultEncoding;
      sz := Encoding.GetBufferEncoding(Buffer, e, de);
      text:= e.GetString(Buffer, sz, Length(Buffer) - sz);
    finally
      EndUpdate;
    end;
  end;
end;

Open in new window


so instead of
sl_ControllerFile.LoadFromFile(wFileName) ;

use
SL_LoadFromFile(sl_ControllerFile, wFileName) ;

side-note
you should swop lines 5 and 6 in your extract i.e.

    sl_ControllerFile:=TStringlist.Create ;
    try
...
    finally
      sl_ControllerFile.free;
    end;
0
 

Author Closing Comment

by:Allan_Fernandes
ID: 39608194
This code is not working in Delphi 2007 but works in Xe2. Decided to upgrade anyways due to unicode issues. Therefore will test and get back, if required.
Since issue is at customers place I will have to upgrade entire application and only then can I test. Solution seems practical and should work.
0
 
LVL 14

Expert Comment

by:Pierre Cornelius
ID: 39608226
update:
  with AStringList do
  begin
    e:= nil;
    de:= DefaultEncoding;
    sz := Encoding.GetBufferEncoding(Buffer, e, de);
    text:= e.GetString(Buffer, sz, Length(Buffer) - sz);
  end;

Open in new window


Notes:
1) If there is truly not enough memory, I would expect an error on this line:
    SetLength(Buffer, fs.Size);

2) If you can't test in your development environment using debug then you should consider wrapping each line (or certain ) within try except block so you can see exactly where the error is occuring.

3) Another thought came to mind: do you have any event handlers attached to your stringlist object (e.g. OnChange)? Thinking if so, there might be a problem with that code instead. Maybe an infinite loop causing you to run out of memory?
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this tutorial I will show you how to use the Windows Speech API in Delphi. I will only cover basic functions such as text to speech and controlling the speed of the speech. SAPI Installation First you need to install the SAPI type library, th…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
There's a multitude of different network monitoring solutions out there, and you're probably wondering what makes NetCrunch so special. It's completely agentless, but does let you create an agent, if you desire. It offers powerful scalability …
Do you want to know how to make a graph with Microsoft Access? First, create a query with the data for the chart. Then make a blank form and add a chart control. This video also shows how to change what data is displayed on the graph as well as form…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question