Solved

Loading large text files

Posted on 2014-04-08
12
919 Views
Last Modified: 2014-04-08
Hi all and everyone.
I'm wondering what is the best tecnique to load a large text file, let's say about 300.000 lines (26Mb)

I tried to use TStringList.LoadFromFile, AssignFile, TStreamReader, but it's always taking a lot and I end to kill the process becaus It's not acceptable a user wait 10 minutes for haveing the text displayed.

I need to load the whole file because I search in the file all non existent paths and after having added them to a TRichEdit, I highlight non existent paths with red color.

I'lll be grateful for your help.
Cheers
Marco
0
Comment
Question by:Marco Gasi
  • 5
  • 3
  • 2
  • +2
12 Comments
 
LVL 19

Accepted Solution

by:
Thommy earned 200 total points
ID: 39985718
You should try TFileStream or Memory-mapped file...
How to read very large text files fast
0
 
LVL 30

Author Comment

by:Marco Gasi
ID: 39985805
Thanks for your reply, Thommy. I'll give it a try asap :-)
0
 
LVL 25

Expert Comment

by:Tony Giangreco
ID: 39985834
You can also try UltraEdit. I've used it for years and it works great on large files also.

http://www.ultraedit.com/
0
 
LVL 45

Expert Comment

by:aikimark
ID: 39986023
Are you setting the Capacity property of your TStringList before you invoke the LoadFromFile method?
0
 
LVL 30

Author Comment

by:Marco Gasi
ID: 39986041
@Thommy: I tried the first snippet but it's still taking a long, long time. I'll try the second, but I begin to suspect that there is some other bottleneck in my code even if it raise up only with large files...

@TG-TIS: I'm trying to make this within a my program. Notepad++ works fine to open this file :-)

@aikimark: I admit - no: I'm going to look for that immediately! Thanks
0
 
LVL 25

Assisted Solution

by:Sinisa Vuk
Sinisa Vuk earned 150 total points
ID: 39986102
Agree, memory mapped files is right direction. There is good unit (unFileMapping) in this blog:

http://delphi-snippets.blogspot.com/2006/04/fast-reading-of-files-using-memory.html

this way you will get pointer to memory.... cast it to PChar when you want to pass it
to  fast pos function. some of fast resources:
http://fastcode.sourceforge.net/

I recommend BMfind (Boyer-Moore search) for this:
http://delphidabbler.com/tips/42

note 2: Before all process disable updating:
RichEdit1.Lines.BeginUpdate;  

Open in new window

...and enable it on the end:
RichEdit1.Lines.EndUpdate;

Open in new window


note 3: add FastMM in project too - this might help you more
0
Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

 
LVL 45

Assisted Solution

by:aikimark
aikimark earned 150 total points
ID: 39986121
Let's not forget kbmMemTable.  Fast and powerful.
http://www.components4programmers.com/products/kbmmemtable/
0
 
LVL 19

Expert Comment

by:Thommy
ID: 39986150
As I have already suggested in my first post, memory-mapped files should give you the best performance.

But you can also try playing around with SetTextBuf together with AssignFile...
System.SetTextBuf Function
Delphi in a Nutshell
0
 
LVL 30

Author Comment

by:Marco Gasi
ID: 39986719
Hi, guys. Thank you all for your replies.

After having experimented all suggestions I discovered the problem was not to load the file in the StringList but display the stringlist in a RichEdit:

Nor
  RichEdit1.Lines.BeginUpdate;
  for I := 0 to sl.Count-1 do
    RichEdit1.Lines.Add(sl[I]);
  RichEdit1.Lines.EndUpdate;

Open in new window


nor
 
RichEdit1.Lines.BeginUpdate;
RichEdit1.Lines.Assign(sl);
RichEdit1.Lines.EndUpdate;

Open in new window


seem to work. The program freezes, and I finally have to kill it. Now I'm trying to wait to see if at the end the strings are loaded in the Richedit, but it is too slow. Any idea?
0
 
LVL 30

Author Comment

by:Marco Gasi
ID: 39986720
Maybe you wish I open a new question?
0
 
LVL 45

Expert Comment

by:aikimark
ID: 39986818
Yes.  I think the fast population of a richedit control warrants a new question.  You might need to replace the control with something that's faster.
0
 
LVL 30

Author Closing Comment

by:Marco Gasi
ID: 39987020
Thanks for your suggestion about the problm, but I found the issue was another and I opened a new question about here: http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_28408002.html

Thanks to all
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

This article explains how to create forms/units independent of other forms/units object names in a delphi project. Have you ever created a form for user input in a Delphi project and then had the need to have that same form in a other Delphi proj…
Introduction The parallel port is a very commonly known port, it was widely used to connect a printer to the PC, if you look at the back of your computer, for those who don't have newer computers, there will be a port with 25 pins and a small print…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…
You have products, that come in variants and want to set different prices for them? Watch this micro tutorial that describes how to configure prices for Magento super attributes. Assigning simple products to configurable: We assigned simple products…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now