Solved

'Out of Memory' for LoadFromFile

Posted on 2013-10-28
11
1,797 Views
Last Modified: 2013-10-29
I need to load a 200MB disk text file into TStringlist. This gives 'Out of memory' at times.
I am using Delphi 2007 on win 7 with 16GB Ram computer.
0
Comment
Question by:Allan_Fernandes
  • 5
  • 4
  • 2
11 Comments
 
LVL 14

Expert Comment

by:Pierre Cornelius
Comment Utility
Can you explain more on what you need to do with the text file.

Also it would help if you tell us more about the file i.e. does it have a structure(e.g. fixed length records, csv, just a bunch of text, etc...)?

If you just need to read it and process the contents, then why not just use a TFilestream and process it buffer by buffer until you reach end of the file?
0
 
LVL 25

Expert Comment

by:Sinisa Vuk
Comment Utility
Why you need to load such large file?
Use something like ATViewer for showing large file only.
0
 

Author Comment

by:Allan_Fernandes
Comment Utility
I have backup application and I need to store details on several files in a text file for quick reference. The incremental patch (full file names , date, size ) has to be merged into the earlier text file. Sorting is required therefore stringlist is required.
0
 
LVL 25

Expert Comment

by:Sinisa Vuk
Comment Utility
No, wrong way. For sorting purpose is using stringlist  very bad for such large text. Use simple database - maybe some standalone like SQLite3. Create table with indexes on field which will be sorted.
Then you have sorted text and easy filtering if you like.
0
 
LVL 14

Expert Comment

by:Pierre Cornelius
Comment Utility
If you are already using a database, then rather store the file info in a table and use sql select commands to retrieve info and sort.

If you do not want to go the db route then you could build an index using an array or linked list to store the info you need e.g. filename, size, folder, etc. Then sort the array or linked list using a Quicksort and QuickFind combo. Let me know if you are interested in this approach so I can provide more info.
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 

Author Comment

by:Allan_Fernandes
Comment Utility
I am using custom sort and the efficiency if very good.
The same logic has been going on over 1000 machines for last few years without any issue.
Even 500,000 lines has not been a problem.
Suddenly this customer has started getting this issue at time of loading the Stringlist for which I am helpless. Also this customer has only about 100,000 files.
0
 
LVL 14

Expert Comment

by:Pierre Cornelius
Comment Utility
what version of delphi are you using?

Can you show:
1) how and where your stringlist object is declared
2) the code loading the text file

Side-note: memory leak? is the file loaded once? or frequently? Are you freeing the stringlist object at some point?
0
 

Author Comment

by:Allan_Fernandes
Comment Utility
I am using Delphi 2007 on win 7 with 16GB Ram computer.
The stringlist is declared within the procedure and the lines are
procedure TForm1.MergeControllerFile;
var
  sl_ControllerFile:TStringlist;
begin
  try
    sl_ControllerFile:=TStringlist.Create ;
    sl_ControllerFile.LoadFromFile(wFileName) ;  // The program does not go to next line
    ...
    ...
  finally
    FreeAndNil(sl_ControllerFile) ;
  end;

Open in new window

0
 
LVL 14

Accepted Solution

by:
Pierre Cornelius earned 500 total points
Comment Utility
Looking at the code of TStringList, I noticed the LoadFromFile tries to load the entire thing in one go. I put together a procedure for you which will load it in numerous reads instead of just one big read of everything. try it and see if it solves your problem.

procedure SL_LoadFromFile(AFilename: string; AStringList: TStringList);
var fs: TFileStream;
    br, i, sz: integer;
    Buffer: TBytes;
    e, de: TEncoding;
begin
  //load the file data into the string in stages
  fs:= TFileStream.Create(AFilename, fmOpenRead OR fmShareDenyWrite);
  try
    i:= 0;
    SetLength(Buffer, fs.Size);
    repeat
      //let's read 65535 bytes at a time
      br:= fs.Read(buffer[i], 65535);
      i:= i + br;
    until (br < 65535);
  finally
    fs.Free;
  end;

  with AStringList do
  begin
    BeginUpdate;
    try
      e:= nil;
      de:= DefaultEncoding;
      sz := Encoding.GetBufferEncoding(Buffer, e, de);
      text:= e.GetString(Buffer, sz, Length(Buffer) - sz);
    finally
      EndUpdate;
    end;
  end;
end;

Open in new window


so instead of
sl_ControllerFile.LoadFromFile(wFileName) ;

use
SL_LoadFromFile(sl_ControllerFile, wFileName) ;

side-note
you should swop lines 5 and 6 in your extract i.e.

    sl_ControllerFile:=TStringlist.Create ;
    try
...
    finally
      sl_ControllerFile.free;
    end;
0
 

Author Closing Comment

by:Allan_Fernandes
Comment Utility
This code is not working in Delphi 2007 but works in Xe2. Decided to upgrade anyways due to unicode issues. Therefore will test and get back, if required.
Since issue is at customers place I will have to upgrade entire application and only then can I test. Solution seems practical and should work.
0
 
LVL 14

Expert Comment

by:Pierre Cornelius
Comment Utility
update:
  with AStringList do
  begin
    e:= nil;
    de:= DefaultEncoding;
    sz := Encoding.GetBufferEncoding(Buffer, e, de);
    text:= e.GetString(Buffer, sz, Length(Buffer) - sz);
  end;

Open in new window


Notes:
1) If there is truly not enough memory, I would expect an error on this line:
    SetLength(Buffer, fs.Size);

2) If you can't test in your development environment using debug then you should consider wrapping each line (or certain ) within try except block so you can see exactly where the error is occuring.

3) Another thought came to mind: do you have any event handlers attached to your stringlist object (e.g. OnChange)? Thinking if so, there might be a problem with that code instead. Maybe an infinite loop causing you to run out of memory?
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

The uses clause is one of those things that just tends to grow and grow. Most of the time this is in the main form, as it's from this form that all others are called. If you have a big application (including many forms), the uses clause in the in…
Introduction The parallel port is a very commonly known port, it was widely used to connect a printer to the PC, if you look at the back of your computer, for those who don't have newer computers, there will be a port with 25 pins and a small print…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
You have products, that come in variants and want to set different prices for them? Watch this micro tutorial that describes how to configure prices for Magento super attributes. Assigning simple products to configurable: We assigned simple products…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now