Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Loading large text files

Posted on 2014-04-08
12
Medium Priority
?
1,241 Views
Last Modified: 2014-04-08
Hi all and everyone.
I'm wondering what is the best tecnique to load a large text file, let's say about 300.000 lines (26Mb)

I tried to use TStringList.LoadFromFile, AssignFile, TStreamReader, but it's always taking a lot and I end to kill the process becaus It's not acceptable a user wait 10 minutes for haveing the text displayed.

I need to load the whole file because I search in the file all non existent paths and after having added them to a TRichEdit, I highlight non existent paths with red color.

I'lll be grateful for your help.
Cheers
Marco
0
Comment
Question by:Marco Gasi
  • 5
  • 3
  • 2
  • +2
12 Comments
 
LVL 19

Accepted Solution

by:
Thommy earned 800 total points
ID: 39985718
You should try TFileStream or Memory-mapped file...
How to read very large text files fast
0
 
LVL 31

Author Comment

by:Marco Gasi
ID: 39985805
Thanks for your reply, Thommy. I'll give it a try asap :-)
0
 
LVL 25

Expert Comment

by:Tony Giangreco
ID: 39985834
You can also try UltraEdit. I've used it for years and it works great on large files also.

http://www.ultraedit.com/
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
LVL 46

Expert Comment

by:aikimark
ID: 39986023
Are you setting the Capacity property of your TStringList before you invoke the LoadFromFile method?
0
 
LVL 31

Author Comment

by:Marco Gasi
ID: 39986041
@Thommy: I tried the first snippet but it's still taking a long, long time. I'll try the second, but I begin to suspect that there is some other bottleneck in my code even if it raise up only with large files...

@TG-TIS: I'm trying to make this within a my program. Notepad++ works fine to open this file :-)

@aikimark: I admit - no: I'm going to look for that immediately! Thanks
0
 
LVL 28

Assisted Solution

by:Sinisa Vuk
Sinisa Vuk earned 600 total points
ID: 39986102
Agree, memory mapped files is right direction. There is good unit (unFileMapping) in this blog:

http://delphi-snippets.blogspot.com/2006/04/fast-reading-of-files-using-memory.html

this way you will get pointer to memory.... cast it to PChar when you want to pass it
to  fast pos function. some of fast resources:
http://fastcode.sourceforge.net/

I recommend BMfind (Boyer-Moore search) for this:
http://delphidabbler.com/tips/42

note 2: Before all process disable updating:
RichEdit1.Lines.BeginUpdate;  

Open in new window

...and enable it on the end:
RichEdit1.Lines.EndUpdate;

Open in new window


note 3: add FastMM in project too - this might help you more
0
 
LVL 46

Assisted Solution

by:aikimark
aikimark earned 600 total points
ID: 39986121
Let's not forget kbmMemTable.  Fast and powerful.
http://www.components4programmers.com/products/kbmmemtable/
0
 
LVL 19

Expert Comment

by:Thommy
ID: 39986150
As I have already suggested in my first post, memory-mapped files should give you the best performance.

But you can also try playing around with SetTextBuf together with AssignFile...
System.SetTextBuf Function
Delphi in a Nutshell
0
 
LVL 31

Author Comment

by:Marco Gasi
ID: 39986719
Hi, guys. Thank you all for your replies.

After having experimented all suggestions I discovered the problem was not to load the file in the StringList but display the stringlist in a RichEdit:

Nor
  RichEdit1.Lines.BeginUpdate;
  for I := 0 to sl.Count-1 do
    RichEdit1.Lines.Add(sl[I]);
  RichEdit1.Lines.EndUpdate;

Open in new window


nor
 
RichEdit1.Lines.BeginUpdate;
RichEdit1.Lines.Assign(sl);
RichEdit1.Lines.EndUpdate;

Open in new window


seem to work. The program freezes, and I finally have to kill it. Now I'm trying to wait to see if at the end the strings are loaded in the Richedit, but it is too slow. Any idea?
0
 
LVL 31

Author Comment

by:Marco Gasi
ID: 39986720
Maybe you wish I open a new question?
0
 
LVL 46

Expert Comment

by:aikimark
ID: 39986818
Yes.  I think the fast population of a richedit control warrants a new question.  You might need to replace the control with something that's faster.
0
 
LVL 31

Author Closing Comment

by:Marco Gasi
ID: 39987020
Thanks for your suggestion about the problm, but I found the issue was another and I opened a new question about here: http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_28408002.html

Thanks to all
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction The parallel port is a very commonly known port, it was widely used to connect a printer to the PC, if you look at the back of your computer, for those who don't have newer computers, there will be a port with 25 pins and a small print…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an anti-spam), the admin…
Despite its rising prevalence in the business world, "the cloud" is still misunderstood. Some companies still believe common misconceptions about lack of security in cloud solutions and many misuses of cloud storage options still occur every day. …
Suggested Courses

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question