Solved

Loading large text files

Posted on 2014-04-08
12
937 Views
Last Modified: 2014-04-08
Hi all and everyone.
I'm wondering what is the best tecnique to load a large text file, let's say about 300.000 lines (26Mb)

I tried to use TStringList.LoadFromFile, AssignFile, TStreamReader, but it's always taking a lot and I end to kill the process becaus It's not acceptable a user wait 10 minutes for haveing the text displayed.

I need to load the whole file because I search in the file all non existent paths and after having added them to a TRichEdit, I highlight non existent paths with red color.

I'lll be grateful for your help.
Cheers
Marco
0
Comment
Question by:Marco Gasi
  • 5
  • 3
  • 2
  • +2
12 Comments
 
LVL 19

Accepted Solution

by:
Thommy earned 200 total points
ID: 39985718
You should try TFileStream or Memory-mapped file...
How to read very large text files fast
0
 
LVL 31

Author Comment

by:Marco Gasi
ID: 39985805
Thanks for your reply, Thommy. I'll give it a try asap :-)
0
 
LVL 25

Expert Comment

by:Tony Giangreco
ID: 39985834
You can also try UltraEdit. I've used it for years and it works great on large files also.

http://www.ultraedit.com/
0
 
LVL 45

Expert Comment

by:aikimark
ID: 39986023
Are you setting the Capacity property of your TStringList before you invoke the LoadFromFile method?
0
 
LVL 31

Author Comment

by:Marco Gasi
ID: 39986041
@Thommy: I tried the first snippet but it's still taking a long, long time. I'll try the second, but I begin to suspect that there is some other bottleneck in my code even if it raise up only with large files...

@TG-TIS: I'm trying to make this within a my program. Notepad++ works fine to open this file :-)

@aikimark: I admit - no: I'm going to look for that immediately! Thanks
0
 
LVL 26

Assisted Solution

by:Sinisa Vuk
Sinisa Vuk earned 150 total points
ID: 39986102
Agree, memory mapped files is right direction. There is good unit (unFileMapping) in this blog:

http://delphi-snippets.blogspot.com/2006/04/fast-reading-of-files-using-memory.html

this way you will get pointer to memory.... cast it to PChar when you want to pass it
to  fast pos function. some of fast resources:
http://fastcode.sourceforge.net/

I recommend BMfind (Boyer-Moore search) for this:
http://delphidabbler.com/tips/42

note 2: Before all process disable updating:
RichEdit1.Lines.BeginUpdate;  

Open in new window

...and enable it on the end:
RichEdit1.Lines.EndUpdate;

Open in new window


note 3: add FastMM in project too - this might help you more
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 
LVL 45

Assisted Solution

by:aikimark
aikimark earned 150 total points
ID: 39986121
Let's not forget kbmMemTable.  Fast and powerful.
http://www.components4programmers.com/products/kbmmemtable/
0
 
LVL 19

Expert Comment

by:Thommy
ID: 39986150
As I have already suggested in my first post, memory-mapped files should give you the best performance.

But you can also try playing around with SetTextBuf together with AssignFile...
System.SetTextBuf Function
Delphi in a Nutshell
0
 
LVL 31

Author Comment

by:Marco Gasi
ID: 39986719
Hi, guys. Thank you all for your replies.

After having experimented all suggestions I discovered the problem was not to load the file in the StringList but display the stringlist in a RichEdit:

Nor
  RichEdit1.Lines.BeginUpdate;
  for I := 0 to sl.Count-1 do
    RichEdit1.Lines.Add(sl[I]);
  RichEdit1.Lines.EndUpdate;

Open in new window


nor
 
RichEdit1.Lines.BeginUpdate;
RichEdit1.Lines.Assign(sl);
RichEdit1.Lines.EndUpdate;

Open in new window


seem to work. The program freezes, and I finally have to kill it. Now I'm trying to wait to see if at the end the strings are loaded in the Richedit, but it is too slow. Any idea?
0
 
LVL 31

Author Comment

by:Marco Gasi
ID: 39986720
Maybe you wish I open a new question?
0
 
LVL 45

Expert Comment

by:aikimark
ID: 39986818
Yes.  I think the fast population of a richedit control warrants a new question.  You might need to replace the control with something that's faster.
0
 
LVL 31

Author Closing Comment

by:Marco Gasi
ID: 39987020
Thanks for your suggestion about the problm, but I found the issue was another and I opened a new question about here: http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_28408002.html

Thanks to all
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

This article explains how to create forms/units independent of other forms/units object names in a delphi project. Have you ever created a form for user input in a Delphi project and then had the need to have that same form in a other Delphi proj…
Have you ever had your Delphi form/application just hanging while waiting for data to load? This is the article to read if you want to learn some things about adding threads for data loading in the background. First, I'll setup a general applica…
In this video I am going to show you how to back up and restore Office 365 mailboxes using CodeTwo Backup for Office 365. Learn more about the tool used in this video here: http://www.codetwo.com/backup-for-office-365/ (http://www.codetwo.com/ba…
As a trusted technology advisor to your customers you are likely getting the daily question of, ‘should I put this in the cloud?’ As customer demands for cloud services increases, companies will see a shift from traditional buying patterns to new…

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

28 Experts available now in Live!

Get 1:1 Help Now