Solved

Loading large text files

Posted on 2014-04-08
12
952 Views
Last Modified: 2014-04-08
Hi all and everyone.
I'm wondering what is the best tecnique to load a large text file, let's say about 300.000 lines (26Mb)

I tried to use TStringList.LoadFromFile, AssignFile, TStreamReader, but it's always taking a lot and I end to kill the process becaus It's not acceptable a user wait 10 minutes for haveing the text displayed.

I need to load the whole file because I search in the file all non existent paths and after having added them to a TRichEdit, I highlight non existent paths with red color.

I'lll be grateful for your help.
Cheers
Marco
0
Comment
Question by:Marco Gasi
  • 5
  • 3
  • 2
  • +2
12 Comments
 
LVL 19

Accepted Solution

by:
Thommy earned 200 total points
ID: 39985718
You should try TFileStream or Memory-mapped file...
How to read very large text files fast
0
 
LVL 31

Author Comment

by:Marco Gasi
ID: 39985805
Thanks for your reply, Thommy. I'll give it a try asap :-)
0
 
LVL 25

Expert Comment

by:Tony Giangreco
ID: 39985834
You can also try UltraEdit. I've used it for years and it works great on large files also.

http://www.ultraedit.com/
0
Courses: Start Training Online With Pros, Today

Brush up on the basics or master the advanced techniques required to earn essential industry certifications, with Courses. Enroll in a course and start learning today. Training topics range from Android App Dev to the Xen Virtualization Platform.

 
LVL 45

Expert Comment

by:aikimark
ID: 39986023
Are you setting the Capacity property of your TStringList before you invoke the LoadFromFile method?
0
 
LVL 31

Author Comment

by:Marco Gasi
ID: 39986041
@Thommy: I tried the first snippet but it's still taking a long, long time. I'll try the second, but I begin to suspect that there is some other bottleneck in my code even if it raise up only with large files...

@TG-TIS: I'm trying to make this within a my program. Notepad++ works fine to open this file :-)

@aikimark: I admit - no: I'm going to look for that immediately! Thanks
0
 
LVL 26

Assisted Solution

by:Sinisa Vuk
Sinisa Vuk earned 150 total points
ID: 39986102
Agree, memory mapped files is right direction. There is good unit (unFileMapping) in this blog:

http://delphi-snippets.blogspot.com/2006/04/fast-reading-of-files-using-memory.html

this way you will get pointer to memory.... cast it to PChar when you want to pass it
to  fast pos function. some of fast resources:
http://fastcode.sourceforge.net/

I recommend BMfind (Boyer-Moore search) for this:
http://delphidabbler.com/tips/42

note 2: Before all process disable updating:
RichEdit1.Lines.BeginUpdate;  

Open in new window

...and enable it on the end:
RichEdit1.Lines.EndUpdate;

Open in new window


note 3: add FastMM in project too - this might help you more
0
 
LVL 45

Assisted Solution

by:aikimark
aikimark earned 150 total points
ID: 39986121
Let's not forget kbmMemTable.  Fast and powerful.
http://www.components4programmers.com/products/kbmmemtable/
0
 
LVL 19

Expert Comment

by:Thommy
ID: 39986150
As I have already suggested in my first post, memory-mapped files should give you the best performance.

But you can also try playing around with SetTextBuf together with AssignFile...
System.SetTextBuf Function
Delphi in a Nutshell
0
 
LVL 31

Author Comment

by:Marco Gasi
ID: 39986719
Hi, guys. Thank you all for your replies.

After having experimented all suggestions I discovered the problem was not to load the file in the StringList but display the stringlist in a RichEdit:

Nor
  RichEdit1.Lines.BeginUpdate;
  for I := 0 to sl.Count-1 do
    RichEdit1.Lines.Add(sl[I]);
  RichEdit1.Lines.EndUpdate;

Open in new window


nor
 
RichEdit1.Lines.BeginUpdate;
RichEdit1.Lines.Assign(sl);
RichEdit1.Lines.EndUpdate;

Open in new window


seem to work. The program freezes, and I finally have to kill it. Now I'm trying to wait to see if at the end the strings are loaded in the Richedit, but it is too slow. Any idea?
0
 
LVL 31

Author Comment

by:Marco Gasi
ID: 39986720
Maybe you wish I open a new question?
0
 
LVL 45

Expert Comment

by:aikimark
ID: 39986818
Yes.  I think the fast population of a richedit control warrants a new question.  You might need to replace the control with something that's faster.
0
 
LVL 31

Author Closing Comment

by:Marco Gasi
ID: 39987020
Thanks for your suggestion about the problm, but I found the issue was another and I opened a new question about here: http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_28408002.html

Thanks to all
0

Featured Post

Gigs: Get Your Project Delivered by an Expert

Select from freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Using idhttp to login to instagram 2 89
Delphi Form ownership 4 87
Delphi 2 59
PHP preg_replace code convert to Delphi 14 55
In this tutorial I will show you how to use the Windows Speech API in Delphi. I will only cover basic functions such as text to speech and controlling the speed of the speech. SAPI Installation First you need to install the SAPI type library, th…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
Windows 10 is mostly good. However the one thing that annoys me is how many clicks you have to do to dial a VPN connection. You have to go to settings from the start menu, (2 clicks), Network and Internet (1 click), Click VPN (another click) then fi…
Established in 1997, Technology Architects has become one of the most reputable technology solutions companies in the country. TA have been providing businesses with cost effective state-of-the-art solutions and unparalleled service that is designed…

785 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question