Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 867
  • Last Modified:

FindFirst/Next to slow

Hi!
I've tried several sort routines and I even tried to skip sorting, and still it takes some time to process large directories. Now I beleve it's the FindFirst/Next functions that's slow?
Is there a faster way to create a file-list?
0
Fraction
Asked:
Fraction
1 Solution
 
DrDelphiCommented:
There are a few things that you might consider:

1: If you are rendering this directory in a listbox (or some other windowed control) as you are building it, you might want to consider retrieving all the files first into an offscreen reposititory such as a TstringList and then assign that to you windowed control once it is done. This way you avoid constantly having to refresh the Listbox with each new file added. This some help performance.

2. There is a really decent FileListBox control which ships with Delphi. It's under the Win3.1 tab.


Good luck!!
0
 
FractionAuthor Commented:
I need more information to be displayed to each file than the FileListBox control offers, size, date, attrib, etc.

I store all file-info in a datastructure, sort it using the common combsort routine before adding it to a TListView (with the BeginUpdate turned on). The TListView update seems to be quiet fast, running without the sort routine does not affect the performance that much, I really think it's then reading that slows things down.
0
 
CrazyOneCommented:
Hmmm I don't know but I really think the FindFirst/Next is rather fast. I have used it for moving across the entire disk and I found it to be as fast as anything else I have seen. I think as DrDelphi said it probably has to do with how you are passing the info to your controls. :>)

You might look into this API. I have never used it so I don't have any code to show you on it.

SHGetFileInfo


The Crazy One
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
dutiCommented:
Maybe you could post your code snippet, so we could try to figure out where it goes wrong.
0
 
FractionAuthor Commented:
I'm Looking on the SHGetFileInfo, anyway, here is a snipped piece of my code (did'nt include the sort routine, but that part is fast):

//-- Begin ----------->

//-- Data structure to store files info -----------
type
  TDirItem = record
    Icon: integer;
    Name: string;
    Size: int64;
    Date: TDateTime;
    Attr: integer;
  end;
  PDirItems = ^TDirItems;
  TDirItems = record
    Count: integer;
    Items: array[0..4000] of TDirItem;
  end;



//-- The ListView Update routines (Quiet fast)-----------
procedure UpdateFileList;
var
  i: integer;
  cr: TCursor;
  Name, Ext, Size, SDate, FAttr: string;
begin
  cr := Screen.Cursor;
  Screen.Cursor := crHourGlass;
  FileList.Items.BeginUpdate;
  FileList.Items.Clear;
  FileList.Columns.Clear;
  DirPanel.Caption := FileListDir;
  for i:=0 to 4 do begin
    with FileList.Columns.Add do begin
      Caption := DefColCaption[i].Caption;
      Width := DefColCaption[i].Width;
      Alignment := DefColCaption[i].Align;
      Tag := i;
      OnClick := ColumnClick;
    end;
  end;
  if FileListContent^.Count>0 then begin
    for i := 0 to FileListContent^.Count - 1 do begin
      if (FileListContent^.Items[i].Attr and faDirectory)<>faDirectory then begin
        Ext := ExtractFileExt(FileListContent^.Items[i].Name);
        Name := FileListContent^.Items[i].Name;
        if Ext>'' then begin
          delete(Name, Length(Name)-Length(Ext)+1, length(Ext));
          delete(Ext, 1, 1);
        end;
        Size := SizeToString(FileListContent^.Items[i].Size);
      end
      else begin
        Ext := '';
        Name := FileListContent^.Items[i].Name;
        Size := '<DIR>';
      end;
      SDate := FormatDateTime(DateFormat, FileListContent^.Items[i].Date);
      if (FileListContent^.Items[i].Attr and faReadOnly)=faReadOnly then FAttr := 'r' else FAttr := '-';
      if (FileListContent^.Items[i].Attr and faArchive)=faArchive then FAttr := FAttr + 'a' else FAttr := FAttr + '-';
      if (FileListContent^.Items[i].Attr and faHidden)=faHidden then FAttr := FAttr + 'h' else FAttr := FAttr + '-';
      if (FileListContent^.Items[i].Attr and faSysFile)=faSysFile then FAttr := FAttr + 's' else FAttr := FAttr + '-';
      with FileList.Items.Add do begin
        ImageIndex := FileListContent^.Items[i].Icon;
        Caption := Name;
        SubItems.Add(Ext);
        SubItems.Add(Size);
        SubItems.Add(SDate);
        SubItems.Add(FAttr);
      end;
    end;
  end;
  Screen.Cursor := cr;
  FileList.Items.EndUpdate;
end;



//-- The Read Dir routines (Slow)-----------
prodecure ReadDir(Dir: string);
var
  i: integer;
  SR: TSearchRec;
  FileInfo: TSHFILEINFO;
begin
  if Dir>'' then begin
    try
      ChDir(Dir);
    except
      ShowMess(false, ic_Info, Format('"%s" - Path not found!', [Dir]));
      exit;
    end;
    if Dir[Length(Dir)]<>'\' then Dir := Dir + '\';
    i := Ord(UpCase(Dir[1])) - 65;
    Paths[i] := Dir;
    FileListContent^.Count := 0;
    i := FindFirst(Dir + '*.*', faAnyFile, SR);
    while i=0 do begin
      if ((SR.Attr and faDirectory)=faDirectory) and (SR.Name <> '.') then begin
        inc(FileListContent^.Count);
        FileListContent^.Items[FileListContent^.Count-1].Name := SR.Name;
        FileListContent^.Items[FileListContent^.Count-1].Size := 0;
        FileListContent^.Items[FileListContent^.Count-1].Date := FileDateToDateTime(SR.Time);
        FileListContent^.Items[FileListContent^.Count-1].Attr := SR.Attr;
      end
      else if SR.Name<>'.' then begin
        inc(FileListContent^.Count);
        FileListContent^.Items[FileListContent^.Count-1].Name := SR.Name;
        FileListContent^.Items[FileListContent^.Count-1].Size := SR.Size;
        FileListContent^.Items[FileListContent^.Count-1].Date := FileDateToDateTime(SR.Time);
        FileListContent^.Items[FileListContent^.Count-1].Attr := SR.Attr;
      end;
      if SR.Name <> '.' then begin
        if SR.Name = '..' then FileListContent^.Items[FileListContent^.Count-1].Icon := 4
        else begin
          ShGetFileInfo(PChar(Dir+SR.Name), 0, FileInfo, SizeOf(FileInfo),SHGFI_SMALLICON or SHGFI_SYSICONINDEX or SHGFI_TYPENAME);
          FileListContent^.Items[FileListContent^.Count-1].Icon := FileInfo.iIcon;
        end;
      end;
      i := FindNext(SR);
    end;
    FindClose(SR);
    Sort;
  end;
  UpdateFileList;
end;

//-- End ----------->
0
 
FractionAuthor Commented:
PS (I forgott):

var
  FileListContent: PDirItems;
0
 
AvonWyssCommented:
I believe that the call to SHGetFileInfo is slowing things down; you have most information you need already in the search record (attributes, size, name, date...), so try to avoid calling it.


Using WITH could also help the compiler generate better code; instead of:

     if ((SR.Attr and faDirectory)=faDirectory) and (SR.Name <> '.') then begin
          inc(FileListContent^.Count);
          FileListContent^.Items[FileListContent^.Count-1].Name := SR.Name;
          FileListContent^.Items[FileListContent^.Count-1].Size := 0;
          FileListContent^.Items[FileListContent^.Count-1].Date := FileDateToDateTime(SR.Time);
          FileListContent^.Items[FileListContent^.Count-1].Attr := SR.Attr;
     end

try this:

     with FileListContent^ do
          if ((SR.Attr and faDirectory)=faDirectory) and (SR.Name <> '.') then begin
               with Items[Count] do begin
                    Name := SR.Name;
                    Size := 0;
                    FileDateToDateTime(SR.Time);
                    Attr := SR.Attr;
               end;
               inc(Count);
          end


Also be aware that FindClose should only be called upon a successfull FindFirst; if FindFirst was unsuccessful, you should NOT call FindClose on that search record. Therefore, rather do your search like this:

     if FindFirst([...],SR)=0 then try
          repeat
               [...]
          until FindNext(SR)<>0;
     finally
          FindClose(SR);
     end;
0
 
FractionAuthor Commented:
You got a point there Avon, is there a simpler way of finding out icon index of a filetype other than using SHGetFileInfo?

Right now I'm looking at D5 demo "Virtual Listview", fast as lightning, hard to understand, but I think this is the piece of code I want.
0
 
FractionAuthor Commented:
Sorry, You got a few points there. My code is quiet uggly, I know ;-)
0
 
cubudCommented:
This is what I use

procedure GetFiles(List : TStrings; ASearchString : String; SubDirs : Boolean; IncludeDirs : Boolean = False);
var
  R       : Integer;
  SR      : TSearchRec;
  Dir,
  Mask    : String;
begin
  if List = nil then
    raise Exception.Create('Stringlist has not been created.');

  List.Clear;
  //Get files
  R := FindFirst(aSearchString, faAnyFile - faDirectory, SR);
  while R = 0 do begin
    List.Add(ExtractFilePath(aSearchString) + SR.Name);
    R := FindNext(SR);
  end;
  FindClose(SR);


  //Get directories
  if SubDirs or IncludeDirs then begin
    R := FindFirst(ExtractFilePath(aSearchString) + '*.*', faDirectory, SR);
    while R = 0 do begin
      if SR.Attr and faDirectory <> 0 then begin
        Dir := ExtractFilePath(aSearchString);
        Mask := ExtractFileName(aSearchString);
        if (SR.Name <> '.') and (SR.Name <> '..') then begin
          if IncludeDirs then List.Add(Dir + SR.Name + '\');
          If SubDirs then GetFiles(List, Dir + SR.Name + '\' + Mask, True);
        end;
      end;
      R := FindNext(SR);
    end;
    FindClose(SR);
  end;
end;

Try that and see if it is fast or not.
If it is slow then it is the ListView.  I believe ListViews + Delphi are slow anyway.

Pete
====
http://www.HowToDoThings.com (Articles)
http://www.Stuckindoors.com/delphi (Open source)
0
 
AvonWyssCommented:
Fraction, in the very first version of Winfows 95, the Explorer windows also did both the listing and the icon retrieval at the same time. It was slow as hell, especially on network shares or slow disks. They moved on and separed the two things: they first read and list all files with a generic icon, and only later (in a thread or sao to avoid interference with thew user interface) to retrieve and draw the correct icons.

I'd suggest a similar approach: read the complete list first, and then start reading the icons while the user already is able to navigate in the UI. This will make the response time much better while not cutting down on functionnality.
0
 
FractionAuthor Commented:
Thanx Avon! This really speed things up.

Sometimes the solution is to simple to figure out ;-)
0
 
AvonWyssCommented:
Glad to help!
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now