• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1486
  • Last Modified:

Loading large text files

Hi all and everyone.
I'm wondering what is the best tecnique to load a large text file, let's say about 300.000 lines (26Mb)

I tried to use TStringList.LoadFromFile, AssignFile, TStreamReader, but it's always taking a lot and I end to kill the process becaus It's not acceptable a user wait 10 minutes for haveing the text displayed.

I need to load the whole file because I search in the file all non existent paths and after having added them to a TRichEdit, I highlight non existent paths with red color.

I'lll be grateful for your help.
Cheers
Marco
0
Marco Gasi
Asked:
Marco Gasi
  • 5
  • 3
  • 2
  • +2
3 Solutions
 
ThommyCommented:
You should try TFileStream or Memory-mapped file...
How to read very large text files fast
0
 
Marco GasiFreelancerAuthor Commented:
Thanks for your reply, Thommy. I'll give it a try asap :-)
0
 
Tony GiangrecoCommented:
You can also try UltraEdit. I've used it for years and it works great on large files also.

http://www.ultraedit.com/
0
Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

 
aikimarkCommented:
Are you setting the Capacity property of your TStringList before you invoke the LoadFromFile method?
0
 
Marco GasiFreelancerAuthor Commented:
@Thommy: I tried the first snippet but it's still taking a long, long time. I'll try the second, but I begin to suspect that there is some other bottleneck in my code even if it raise up only with large files...

@TG-TIS: I'm trying to make this within a my program. Notepad++ works fine to open this file :-)

@aikimark: I admit - no: I'm going to look for that immediately! Thanks
0
 
Sinisa VukCommented:
Agree, memory mapped files is right direction. There is good unit (unFileMapping) in this blog:

http://delphi-snippets.blogspot.com/2006/04/fast-reading-of-files-using-memory.html

this way you will get pointer to memory.... cast it to PChar when you want to pass it
to  fast pos function. some of fast resources:
http://fastcode.sourceforge.net/

I recommend BMfind (Boyer-Moore search) for this:
http://delphidabbler.com/tips/42

note 2: Before all process disable updating:
RichEdit1.Lines.BeginUpdate;  

Open in new window

...and enable it on the end:
RichEdit1.Lines.EndUpdate;

Open in new window


note 3: add FastMM in project too - this might help you more
0
 
aikimarkCommented:
Let's not forget kbmMemTable.  Fast and powerful.
http://www.components4programmers.com/products/kbmmemtable/
0
 
ThommyCommented:
As I have already suggested in my first post, memory-mapped files should give you the best performance.

But you can also try playing around with SetTextBuf together with AssignFile...
System.SetTextBuf Function
Delphi in a Nutshell
0
 
Marco GasiFreelancerAuthor Commented:
Hi, guys. Thank you all for your replies.

After having experimented all suggestions I discovered the problem was not to load the file in the StringList but display the stringlist in a RichEdit:

Nor
  RichEdit1.Lines.BeginUpdate;
  for I := 0 to sl.Count-1 do
    RichEdit1.Lines.Add(sl[I]);
  RichEdit1.Lines.EndUpdate;

Open in new window


nor
 
RichEdit1.Lines.BeginUpdate;
RichEdit1.Lines.Assign(sl);
RichEdit1.Lines.EndUpdate;

Open in new window


seem to work. The program freezes, and I finally have to kill it. Now I'm trying to wait to see if at the end the strings are loaded in the Richedit, but it is too slow. Any idea?
0
 
Marco GasiFreelancerAuthor Commented:
Maybe you wish I open a new question?
0
 
aikimarkCommented:
Yes.  I think the fast population of a richedit control warrants a new question.  You might need to replace the control with something that's faster.
0
 
Marco GasiFreelancerAuthor Commented:
Thanks for your suggestion about the problm, but I found the issue was another and I opened a new question about here: http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_28408002.html

Thanks to all
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

  • 5
  • 3
  • 2
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now