Solved

FindFirst/Next to slow

Posted on 2001-09-15
13
832 Views
Last Modified: 2008-02-01
Hi!
I've tried several sort routines and I even tried to skip sorting, and still it takes some time to process large directories. Now I beleve it's the FindFirst/Next functions that's slow?
Is there a faster way to create a file-list?
0
Comment
Question by:Fraction
13 Comments
 
LVL 6

Expert Comment

by:DrDelphi
ID: 6485205
There are a few things that you might consider:

1: If you are rendering this directory in a listbox (or some other windowed control) as you are building it, you might want to consider retrieving all the files first into an offscreen reposititory such as a TstringList and then assign that to you windowed control once it is done. This way you avoid constantly having to refresh the Listbox with each new file added. This some help performance.

2. There is a really decent FileListBox control which ships with Delphi. It's under the Win3.1 tab.


Good luck!!
0
 
LVL 1

Author Comment

by:Fraction
ID: 6485216
I need more information to be displayed to each file than the FileListBox control offers, size, date, attrib, etc.

I store all file-info in a datastructure, sort it using the common combsort routine before adding it to a TListView (with the BeginUpdate turned on). The TListView update seems to be quiet fast, running without the sort routine does not affect the performance that much, I really think it's then reading that slows things down.
0
 
LVL 44

Expert Comment

by:CrazyOne
ID: 6485386
Hmmm I don't know but I really think the FindFirst/Next is rather fast. I have used it for moving across the entire disk and I found it to be as fast as anything else I have seen. I think as DrDelphi said it probably has to do with how you are passing the info to your controls. :>)

You might look into this API. I have never used it so I don't have any code to show you on it.

SHGetFileInfo


The Crazy One
0
 

Expert Comment

by:duti
ID: 6485545
Maybe you could post your code snippet, so we could try to figure out where it goes wrong.
0
 
LVL 1

Author Comment

by:Fraction
ID: 6485702
I'm Looking on the SHGetFileInfo, anyway, here is a snipped piece of my code (did'nt include the sort routine, but that part is fast):

//-- Begin ----------->

//-- Data structure to store files info -----------
type
  TDirItem = record
    Icon: integer;
    Name: string;
    Size: int64;
    Date: TDateTime;
    Attr: integer;
  end;
  PDirItems = ^TDirItems;
  TDirItems = record
    Count: integer;
    Items: array[0..4000] of TDirItem;
  end;



//-- The ListView Update routines (Quiet fast)-----------
procedure UpdateFileList;
var
  i: integer;
  cr: TCursor;
  Name, Ext, Size, SDate, FAttr: string;
begin
  cr := Screen.Cursor;
  Screen.Cursor := crHourGlass;
  FileList.Items.BeginUpdate;
  FileList.Items.Clear;
  FileList.Columns.Clear;
  DirPanel.Caption := FileListDir;
  for i:=0 to 4 do begin
    with FileList.Columns.Add do begin
      Caption := DefColCaption[i].Caption;
      Width := DefColCaption[i].Width;
      Alignment := DefColCaption[i].Align;
      Tag := i;
      OnClick := ColumnClick;
    end;
  end;
  if FileListContent^.Count>0 then begin
    for i := 0 to FileListContent^.Count - 1 do begin
      if (FileListContent^.Items[i].Attr and faDirectory)<>faDirectory then begin
        Ext := ExtractFileExt(FileListContent^.Items[i].Name);
        Name := FileListContent^.Items[i].Name;
        if Ext>'' then begin
          delete(Name, Length(Name)-Length(Ext)+1, length(Ext));
          delete(Ext, 1, 1);
        end;
        Size := SizeToString(FileListContent^.Items[i].Size);
      end
      else begin
        Ext := '';
        Name := FileListContent^.Items[i].Name;
        Size := '<DIR>';
      end;
      SDate := FormatDateTime(DateFormat, FileListContent^.Items[i].Date);
      if (FileListContent^.Items[i].Attr and faReadOnly)=faReadOnly then FAttr := 'r' else FAttr := '-';
      if (FileListContent^.Items[i].Attr and faArchive)=faArchive then FAttr := FAttr + 'a' else FAttr := FAttr + '-';
      if (FileListContent^.Items[i].Attr and faHidden)=faHidden then FAttr := FAttr + 'h' else FAttr := FAttr + '-';
      if (FileListContent^.Items[i].Attr and faSysFile)=faSysFile then FAttr := FAttr + 's' else FAttr := FAttr + '-';
      with FileList.Items.Add do begin
        ImageIndex := FileListContent^.Items[i].Icon;
        Caption := Name;
        SubItems.Add(Ext);
        SubItems.Add(Size);
        SubItems.Add(SDate);
        SubItems.Add(FAttr);
      end;
    end;
  end;
  Screen.Cursor := cr;
  FileList.Items.EndUpdate;
end;



//-- The Read Dir routines (Slow)-----------
prodecure ReadDir(Dir: string);
var
  i: integer;
  SR: TSearchRec;
  FileInfo: TSHFILEINFO;
begin
  if Dir>'' then begin
    try
      ChDir(Dir);
    except
      ShowMess(false, ic_Info, Format('"%s" - Path not found!', [Dir]));
      exit;
    end;
    if Dir[Length(Dir)]<>'\' then Dir := Dir + '\';
    i := Ord(UpCase(Dir[1])) - 65;
    Paths[i] := Dir;
    FileListContent^.Count := 0;
    i := FindFirst(Dir + '*.*', faAnyFile, SR);
    while i=0 do begin
      if ((SR.Attr and faDirectory)=faDirectory) and (SR.Name <> '.') then begin
        inc(FileListContent^.Count);
        FileListContent^.Items[FileListContent^.Count-1].Name := SR.Name;
        FileListContent^.Items[FileListContent^.Count-1].Size := 0;
        FileListContent^.Items[FileListContent^.Count-1].Date := FileDateToDateTime(SR.Time);
        FileListContent^.Items[FileListContent^.Count-1].Attr := SR.Attr;
      end
      else if SR.Name<>'.' then begin
        inc(FileListContent^.Count);
        FileListContent^.Items[FileListContent^.Count-1].Name := SR.Name;
        FileListContent^.Items[FileListContent^.Count-1].Size := SR.Size;
        FileListContent^.Items[FileListContent^.Count-1].Date := FileDateToDateTime(SR.Time);
        FileListContent^.Items[FileListContent^.Count-1].Attr := SR.Attr;
      end;
      if SR.Name <> '.' then begin
        if SR.Name = '..' then FileListContent^.Items[FileListContent^.Count-1].Icon := 4
        else begin
          ShGetFileInfo(PChar(Dir+SR.Name), 0, FileInfo, SizeOf(FileInfo),SHGFI_SMALLICON or SHGFI_SYSICONINDEX or SHGFI_TYPENAME);
          FileListContent^.Items[FileListContent^.Count-1].Icon := FileInfo.iIcon;
        end;
      end;
      i := FindNext(SR);
    end;
    FindClose(SR);
    Sort;
  end;
  UpdateFileList;
end;

//-- End ----------->
0
 
LVL 1

Author Comment

by:Fraction
ID: 6485706
PS (I forgott):

var
  FileListContent: PDirItems;
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 
LVL 14

Expert Comment

by:AvonWyss
ID: 6485929
I believe that the call to SHGetFileInfo is slowing things down; you have most information you need already in the search record (attributes, size, name, date...), so try to avoid calling it.


Using WITH could also help the compiler generate better code; instead of:

     if ((SR.Attr and faDirectory)=faDirectory) and (SR.Name <> '.') then begin
          inc(FileListContent^.Count);
          FileListContent^.Items[FileListContent^.Count-1].Name := SR.Name;
          FileListContent^.Items[FileListContent^.Count-1].Size := 0;
          FileListContent^.Items[FileListContent^.Count-1].Date := FileDateToDateTime(SR.Time);
          FileListContent^.Items[FileListContent^.Count-1].Attr := SR.Attr;
     end

try this:

     with FileListContent^ do
          if ((SR.Attr and faDirectory)=faDirectory) and (SR.Name <> '.') then begin
               with Items[Count] do begin
                    Name := SR.Name;
                    Size := 0;
                    FileDateToDateTime(SR.Time);
                    Attr := SR.Attr;
               end;
               inc(Count);
          end


Also be aware that FindClose should only be called upon a successfull FindFirst; if FindFirst was unsuccessful, you should NOT call FindClose on that search record. Therefore, rather do your search like this:

     if FindFirst([...],SR)=0 then try
          repeat
               [...]
          until FindNext(SR)<>0;
     finally
          FindClose(SR);
     end;
0
 
LVL 1

Author Comment

by:Fraction
ID: 6485973
You got a point there Avon, is there a simpler way of finding out icon index of a filetype other than using SHGetFileInfo?

Right now I'm looking at D5 demo "Virtual Listview", fast as lightning, hard to understand, but I think this is the piece of code I want.
0
 
LVL 1

Author Comment

by:Fraction
ID: 6485977
Sorry, You got a few points there. My code is quiet uggly, I know ;-)
0
 
LVL 3

Expert Comment

by:cubud
ID: 6485978
This is what I use

procedure GetFiles(List : TStrings; ASearchString : String; SubDirs : Boolean; IncludeDirs : Boolean = False);
var
  R       : Integer;
  SR      : TSearchRec;
  Dir,
  Mask    : String;
begin
  if List = nil then
    raise Exception.Create('Stringlist has not been created.');

  List.Clear;
  //Get files
  R := FindFirst(aSearchString, faAnyFile - faDirectory, SR);
  while R = 0 do begin
    List.Add(ExtractFilePath(aSearchString) + SR.Name);
    R := FindNext(SR);
  end;
  FindClose(SR);


  //Get directories
  if SubDirs or IncludeDirs then begin
    R := FindFirst(ExtractFilePath(aSearchString) + '*.*', faDirectory, SR);
    while R = 0 do begin
      if SR.Attr and faDirectory <> 0 then begin
        Dir := ExtractFilePath(aSearchString);
        Mask := ExtractFileName(aSearchString);
        if (SR.Name <> '.') and (SR.Name <> '..') then begin
          if IncludeDirs then List.Add(Dir + SR.Name + '\');
          If SubDirs then GetFiles(List, Dir + SR.Name + '\' + Mask, True);
        end;
      end;
      R := FindNext(SR);
    end;
    FindClose(SR);
  end;
end;

Try that and see if it is fast or not.
If it is slow then it is the ListView.  I believe ListViews + Delphi are slow anyway.

Pete
====
http://www.HowToDoThings.com (Articles)
http://www.Stuckindoors.com/delphi (Open source)
0
 
LVL 14

Accepted Solution

by:
AvonWyss earned 100 total points
ID: 6485980
Fraction, in the very first version of Winfows 95, the Explorer windows also did both the listing and the icon retrieval at the same time. It was slow as hell, especially on network shares or slow disks. They moved on and separed the two things: they first read and list all files with a generic icon, and only later (in a thread or sao to avoid interference with thew user interface) to retrieve and draw the correct icons.

I'd suggest a similar approach: read the complete list first, and then start reading the icons while the user already is able to navigate in the UI. This will make the response time much better while not cutting down on functionnality.
0
 
LVL 1

Author Comment

by:Fraction
ID: 6486060
Thanx Avon! This really speed things up.

Sometimes the solution is to simple to figure out ;-)
0
 
LVL 14

Expert Comment

by:AvonWyss
ID: 6486160
Glad to help!
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

Have you ever had your Delphi form/application just hanging while waiting for data to load? This is the article to read if you want to learn some things about adding threads for data loading in the background. First, I'll setup a general applica…
Hello everybody This Article will show you how to validate number with TEdit control, What's the TEdit control? TEdit is a standard Windows edit control on a form, it allows to user to write, read and copy/paste single line of text. Usua…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
Access reports are powerful and flexible. Learn how to create a query and then a grouped report using the wizard. Modify the report design after the wizard is done to make it look better. There will be another video to explain how to put the final p…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now