Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

'Out of Memory' for LoadFromFile

Posted on 2013-10-28
11
1,947 Views
Last Modified: 2013-10-29
I need to load a 200MB disk text file into TStringlist. This gives 'Out of memory' at times.
I am using Delphi 2007 on win 7 with 16GB Ram computer.
0
Comment
Question by:Allan_Fernandes
  • 5
  • 4
  • 2
11 Comments
 
LVL 14

Expert Comment

by:Pierre Cornelius
ID: 39607911
Can you explain more on what you need to do with the text file.

Also it would help if you tell us more about the file i.e. does it have a structure(e.g. fixed length records, csv, just a bunch of text, etc...)?

If you just need to read it and process the contents, then why not just use a TFilestream and process it buffer by buffer until you reach end of the file?
0
 
LVL 26

Expert Comment

by:Sinisa Vuk
ID: 39607913
Why you need to load such large file?
Use something like ATViewer for showing large file only.
0
 

Author Comment

by:Allan_Fernandes
ID: 39607915
I have backup application and I need to store details on several files in a text file for quick reference. The incremental patch (full file names , date, size ) has to be merged into the earlier text file. Sorting is required therefore stringlist is required.
0
Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
LVL 26

Expert Comment

by:Sinisa Vuk
ID: 39607920
No, wrong way. For sorting purpose is using stringlist  very bad for such large text. Use simple database - maybe some standalone like SQLite3. Create table with indexes on field which will be sorted.
Then you have sorted text and easy filtering if you like.
0
 
LVL 14

Expert Comment

by:Pierre Cornelius
ID: 39607948
If you are already using a database, then rather store the file info in a table and use sql select commands to retrieve info and sort.

If you do not want to go the db route then you could build an index using an array or linked list to store the info you need e.g. filename, size, folder, etc. Then sort the array or linked list using a Quicksort and QuickFind combo. Let me know if you are interested in this approach so I can provide more info.
0
 

Author Comment

by:Allan_Fernandes
ID: 39607959
I am using custom sort and the efficiency if very good.
The same logic has been going on over 1000 machines for last few years without any issue.
Even 500,000 lines has not been a problem.
Suddenly this customer has started getting this issue at time of loading the Stringlist for which I am helpless. Also this customer has only about 100,000 files.
0
 
LVL 14

Expert Comment

by:Pierre Cornelius
ID: 39607978
what version of delphi are you using?

Can you show:
1) how and where your stringlist object is declared
2) the code loading the text file

Side-note: memory leak? is the file loaded once? or frequently? Are you freeing the stringlist object at some point?
0
 

Author Comment

by:Allan_Fernandes
ID: 39608067
I am using Delphi 2007 on win 7 with 16GB Ram computer.
The stringlist is declared within the procedure and the lines are
procedure TForm1.MergeControllerFile;
var
  sl_ControllerFile:TStringlist;
begin
  try
    sl_ControllerFile:=TStringlist.Create ;
    sl_ControllerFile.LoadFromFile(wFileName) ;  // The program does not go to next line
    ...
    ...
  finally
    FreeAndNil(sl_ControllerFile) ;
  end;

Open in new window

0
 
LVL 14

Accepted Solution

by:
Pierre Cornelius earned 500 total points
ID: 39608160
Looking at the code of TStringList, I noticed the LoadFromFile tries to load the entire thing in one go. I put together a procedure for you which will load it in numerous reads instead of just one big read of everything. try it and see if it solves your problem.

procedure SL_LoadFromFile(AFilename: string; AStringList: TStringList);
var fs: TFileStream;
    br, i, sz: integer;
    Buffer: TBytes;
    e, de: TEncoding;
begin
  //load the file data into the string in stages
  fs:= TFileStream.Create(AFilename, fmOpenRead OR fmShareDenyWrite);
  try
    i:= 0;
    SetLength(Buffer, fs.Size);
    repeat
      //let's read 65535 bytes at a time
      br:= fs.Read(buffer[i], 65535);
      i:= i + br;
    until (br < 65535);
  finally
    fs.Free;
  end;

  with AStringList do
  begin
    BeginUpdate;
    try
      e:= nil;
      de:= DefaultEncoding;
      sz := Encoding.GetBufferEncoding(Buffer, e, de);
      text:= e.GetString(Buffer, sz, Length(Buffer) - sz);
    finally
      EndUpdate;
    end;
  end;
end;

Open in new window


so instead of
sl_ControllerFile.LoadFromFile(wFileName) ;

use
SL_LoadFromFile(sl_ControllerFile, wFileName) ;

side-note
you should swop lines 5 and 6 in your extract i.e.

    sl_ControllerFile:=TStringlist.Create ;
    try
...
    finally
      sl_ControllerFile.free;
    end;
0
 

Author Closing Comment

by:Allan_Fernandes
ID: 39608194
This code is not working in Delphi 2007 but works in Xe2. Decided to upgrade anyways due to unicode issues. Therefore will test and get back, if required.
Since issue is at customers place I will have to upgrade entire application and only then can I test. Solution seems practical and should work.
0
 
LVL 14

Expert Comment

by:Pierre Cornelius
ID: 39608226
update:
  with AStringList do
  begin
    e:= nil;
    de:= DefaultEncoding;
    sz := Encoding.GetBufferEncoding(Buffer, e, de);
    text:= e.GetString(Buffer, sz, Length(Buffer) - sz);
  end;

Open in new window


Notes:
1) If there is truly not enough memory, I would expect an error on this line:
    SetLength(Buffer, fs.Size);

2) If you can't test in your development environment using debug then you should consider wrapping each line (or certain ) within try except block so you can see exactly where the error is occuring.

3) Another thought came to mind: do you have any event handlers attached to your stringlist object (e.g. OnChange)? Thinking if so, there might be a problem with that code instead. Maybe an infinite loop causing you to run out of memory?
0

Featured Post

Active Directory Webinar

We all know we need to protect and secure our privileges, but where to start? Join Experts Exchange and ManageEngine on Tuesday, April 11, 2017 10:00 AM PDT to learn how to track and secure privileged users in Active Directory.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

This article explains how to create forms/units independent of other forms/units object names in a delphi project. Have you ever created a form for user input in a Delphi project and then had the need to have that same form in a other Delphi proj…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…

829 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question