Solved

'Out of Memory' for LoadFromFile

Posted on 2013-10-28
11
1,902 Views
Last Modified: 2013-10-29
I need to load a 200MB disk text file into TStringlist. This gives 'Out of memory' at times.
I am using Delphi 2007 on win 7 with 16GB Ram computer.
0
Comment
Question by:Allan_Fernandes
  • 5
  • 4
  • 2
11 Comments
 
LVL 14

Expert Comment

by:Pierre Cornelius
ID: 39607911
Can you explain more on what you need to do with the text file.

Also it would help if you tell us more about the file i.e. does it have a structure(e.g. fixed length records, csv, just a bunch of text, etc...)?

If you just need to read it and process the contents, then why not just use a TFilestream and process it buffer by buffer until you reach end of the file?
0
 
LVL 26

Expert Comment

by:Sinisa Vuk
ID: 39607913
Why you need to load such large file?
Use something like ATViewer for showing large file only.
0
 

Author Comment

by:Allan_Fernandes
ID: 39607915
I have backup application and I need to store details on several files in a text file for quick reference. The incremental patch (full file names , date, size ) has to be merged into the earlier text file. Sorting is required therefore stringlist is required.
0
3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

 
LVL 26

Expert Comment

by:Sinisa Vuk
ID: 39607920
No, wrong way. For sorting purpose is using stringlist  very bad for such large text. Use simple database - maybe some standalone like SQLite3. Create table with indexes on field which will be sorted.
Then you have sorted text and easy filtering if you like.
0
 
LVL 14

Expert Comment

by:Pierre Cornelius
ID: 39607948
If you are already using a database, then rather store the file info in a table and use sql select commands to retrieve info and sort.

If you do not want to go the db route then you could build an index using an array or linked list to store the info you need e.g. filename, size, folder, etc. Then sort the array or linked list using a Quicksort and QuickFind combo. Let me know if you are interested in this approach so I can provide more info.
0
 

Author Comment

by:Allan_Fernandes
ID: 39607959
I am using custom sort and the efficiency if very good.
The same logic has been going on over 1000 machines for last few years without any issue.
Even 500,000 lines has not been a problem.
Suddenly this customer has started getting this issue at time of loading the Stringlist for which I am helpless. Also this customer has only about 100,000 files.
0
 
LVL 14

Expert Comment

by:Pierre Cornelius
ID: 39607978
what version of delphi are you using?

Can you show:
1) how and where your stringlist object is declared
2) the code loading the text file

Side-note: memory leak? is the file loaded once? or frequently? Are you freeing the stringlist object at some point?
0
 

Author Comment

by:Allan_Fernandes
ID: 39608067
I am using Delphi 2007 on win 7 with 16GB Ram computer.
The stringlist is declared within the procedure and the lines are
procedure TForm1.MergeControllerFile;
var
  sl_ControllerFile:TStringlist;
begin
  try
    sl_ControllerFile:=TStringlist.Create ;
    sl_ControllerFile.LoadFromFile(wFileName) ;  // The program does not go to next line
    ...
    ...
  finally
    FreeAndNil(sl_ControllerFile) ;
  end;

Open in new window

0
 
LVL 14

Accepted Solution

by:
Pierre Cornelius earned 500 total points
ID: 39608160
Looking at the code of TStringList, I noticed the LoadFromFile tries to load the entire thing in one go. I put together a procedure for you which will load it in numerous reads instead of just one big read of everything. try it and see if it solves your problem.

procedure SL_LoadFromFile(AFilename: string; AStringList: TStringList);
var fs: TFileStream;
    br, i, sz: integer;
    Buffer: TBytes;
    e, de: TEncoding;
begin
  //load the file data into the string in stages
  fs:= TFileStream.Create(AFilename, fmOpenRead OR fmShareDenyWrite);
  try
    i:= 0;
    SetLength(Buffer, fs.Size);
    repeat
      //let's read 65535 bytes at a time
      br:= fs.Read(buffer[i], 65535);
      i:= i + br;
    until (br < 65535);
  finally
    fs.Free;
  end;

  with AStringList do
  begin
    BeginUpdate;
    try
      e:= nil;
      de:= DefaultEncoding;
      sz := Encoding.GetBufferEncoding(Buffer, e, de);
      text:= e.GetString(Buffer, sz, Length(Buffer) - sz);
    finally
      EndUpdate;
    end;
  end;
end;

Open in new window


so instead of
sl_ControllerFile.LoadFromFile(wFileName) ;

use
SL_LoadFromFile(sl_ControllerFile, wFileName) ;

side-note
you should swop lines 5 and 6 in your extract i.e.

    sl_ControllerFile:=TStringlist.Create ;
    try
...
    finally
      sl_ControllerFile.free;
    end;
0
 

Author Closing Comment

by:Allan_Fernandes
ID: 39608194
This code is not working in Delphi 2007 but works in Xe2. Decided to upgrade anyways due to unicode issues. Therefore will test and get back, if required.
Since issue is at customers place I will have to upgrade entire application and only then can I test. Solution seems practical and should work.
0
 
LVL 14

Expert Comment

by:Pierre Cornelius
ID: 39608226
update:
  with AStringList do
  begin
    e:= nil;
    de:= DefaultEncoding;
    sz := Encoding.GetBufferEncoding(Buffer, e, de);
    text:= e.GetString(Buffer, sz, Length(Buffer) - sz);
  end;

Open in new window


Notes:
1) If there is truly not enough memory, I would expect an error on this line:
    SetLength(Buffer, fs.Size);

2) If you can't test in your development environment using debug then you should consider wrapping each line (or certain ) within try except block so you can see exactly where the error is occuring.

3) Another thought came to mind: do you have any event handlers attached to your stringlist object (e.g. OnChange)? Thinking if so, there might be a problem with that code instead. Maybe an infinite loop causing you to run out of memory?
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

The uses clause is one of those things that just tends to grow and grow. Most of the time this is in the main form, as it's from this form that all others are called. If you have a big application (including many forms), the uses clause in the in…
Hello everybody This Article will show you how to validate number with TEdit control, What's the TEdit control? TEdit is a standard Windows edit control on a form, it allows to user to write, read and copy/paste single line of text. Usua…
This Micro Tutorial demonstrates using Microsoft Excel pivot tables, how to reverse engineer competitors' marketing strategies through backlinks.
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question