Solved

With more than 100,000 Record, Locate Methode is Lame duck...How to Get higher speed in searching progress....

Posted on 2001-09-04
6
288 Views
Last Modified: 2010-04-06
          I?m writing a code with A Medium size table, ?Member? Table. Problems I have encounter is that I try to create Search-Form to provide user interface by letting user key in MemberId in Edit Box and write OnChange Event of EditBox. Form only have a DBGrid with 3 Columns of MemberId, MemberName, MemberSurname. But searching speed suffer user too much and make my Search-Form lame duck...By using a Locate Methode, I?ve created Index Name ?Member01? in which indexed on Field ?MemberId? but it?s still Run with too bad speed
        I wonder if there?s anyway to write a Search-Form to provide user interface by entering a MemberId and Dataset move to Such Record first found
        For example, Table contain 3 Record
          Record 1              MemberId = 0001  MemberName = Name01 ....
          Record 2              MemberId = 0101 MemberName = Name02
          Record 3              MemberId = 0201 MemberName = Name03
     If User Key in 01   Form Must move RecordPointer to Record 2, if User Key in 02 RecordPointer must move to Record 3 and So on...
           What I want is that Record 999       Form Must move Record Pointer to Record of MemebrId 99900 Within 2 Seconds At least on Pentium II-400  , say, must be faster depend on Spec of Computer running it. Any real expert out there Show me how to do so, I?ll give you 200 points
0
Comment
Question by:DOOM_DAEMON
6 Comments
 
LVL 10

Expert Comment

by:Jacco
ID: 6453093
What component are you using?

TQuery.Locate never uses index

Regards Jacco
0
 
LVL 3

Expert Comment

by:karouri
ID: 6453794
Listening.
I just hope that these 100,000 records are not in a Microsoft Access database..
0
 
LVL 27

Expert Comment

by:kretzschmar
ID: 6455848
listening
(multiple ways possible, but some more info needed)
0
Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

 
LVL 3

Accepted Solution

by:
oraelbis earned 200 total points
ID: 6457587
I recomend to use TClientDataSet(from Midas tab) and create indexes on the fly(creating indexes very fast).
After creating indexes you can use them for fast searching, or filtering. This component support additional filter options.(Note: before close it you may free all created indexes).
There is some code from my TClientDataset descendant (you cand find this components sourcecode on http://bs-elbis.chat.ru) with supporting search function. Use Tedit control on your form such as:
procedure TfrmEmployee.edtSearchChange(Sender: TObject);
begin
  //cdsEmloyee TBS_Client dataset
  cdsEmloyee.SearchCdsRecord(edtSearch);
end;


function TBS_ClientDataSet.SearchCdsRecord(const AedtSearch:Tedit):boolean;
var
  sFieldsNames:String;
  i:integer;
  theSearchText:String;
  aSubStr:String;
  theSearchArray:Variant;

begin
  sFieldsNames:='';
  theSearchText:=AedtSearch.text;
  Result:=False;
  try
    theSearchArray := VarArrayCreate([0, Self.IndexFieldCount - 1], varVariant);
    for i := 0 to Self.IndexFieldCount - 1 do
      begin
        if sFieldsNames<>'' then
          sFieldsNames:=sFieldsNames+';';
        sFieldsNames:=sFieldsNames + Self.IndexFields[i].FieldName;
        if Pos(';',theSearchText)>0 then
          begin
            aSubStr:=Copy(theSearchText,1,Pos(';',theSearchText)-1);
            theSearchText:=Copy(theSearchText,Pos(';',theSearchText)+1,
                           Length(theSearchText)-(Pos(';',theSearchText)));
          end
        else
          begin
            aSubStr:=theSearchText;
            theSearchText:='';
          end;
        try
          case Self.IndexFields[i].DataType of
            ftSmallint,ftInteger,ftBytes,ftVarBytes,
            ftAutoInc,ftLargeint,ftTypedBinary:
              if aSubStr<>'' then
                theSearchArray[i]:=StrToInt(aSubStr)
              else
                theSearchArray[i]:=0;
            ftWord,ftFloat,ftBCD,ftCurrency:
              if aSubStr<>'' then
                theSearchArray[i]:=StrToFloat(aSubStr)
              else
                theSearchArray[i]:=0;
            ftDate, ftTime, ftDateTime:
              theSearchArray[i]:=StrToDate(aSubStr);
          else
            theSearchArray[i]:=aSubStr;
          end;
        except
          theSearchArray[i]:=aSubStr;
        end;
      end;

    if Self.Locate(sFieldsNames, theSearchArray,
       [loCaseInsensitive,loPartialKey]) then
      begin
        AedtSearch.Font.Color:=clWindowText;
      end
    else
      AedtSearch.Font.Color:=clRed; {Can't find}
    Result:=True;
  except      //Try restore last position
    Result:=False;
  end;
end;

procedure TBS_ClientDataSet.FreeCreatedIndexes;
var
  i:integer;
begin
  if Self.Active then
  try
    IndexDefs.Update; //Refresh
    for i := 0 to IndexDefs.Count - 1 do
      if (IndexDefs.Items[i].Name<>'DEFAULT_ORDER') and (IndexDefs.Items[i].Name<>'CHANGEINDEX')
         and (IndexDefs.Items[i].Name<>'PRIMARY_KEY') then
           begin
             if (Self.IndexName=IndexDefs.Items[i].Name) then
               Self.IndexName:='';
             DeleteIndex(IndexDefs.Items[i].Name);
           end;
  except
    raise;
  end;
end;

////////////////////////////////////////////////////
I use TDBGreedEh descendant, for automatic index creating.
There is part of code from it:
function TBS_DBGridEh.BS_MultiSort:boolean;
var
  i                   :integer;
  theIndexName        :String;
  theIndexFields      :String;
  theDescFields       :String;
  theOldCursor        :TCursor;
  theBS_ClientDataSet :TBS_ClientDataSet;
  fIndexPresent       :boolean;
  s                   :String;

 function DeleteStr(str:String; substr:String): String;
 var i:Integer;
 begin
   i := Pos(substr,str);
   if i <> 0 then
     Delete(str,i,Length(substr));
   Result := str;
 end;

begin
 Result:=False;
 theOldCursor:=Screen.Cursor;
 Screen.Cursor:=crHourGlass;
 try
  if Assigned(Self.DataSource.DataSet as TBS_ClientDataSet)
    and Self.DataSource.DataSet.Active then
    begin
      theBS_ClientDataSet:=TBS_ClientDataSet(Self.DataSource.DataSet);

      theIndexName := '';
      theDescFields:= '';
      theIndexFields:= '';
      for i := 0 to Self.SortMarkedColumns.Count-1 do
        if Self.SortMarkedColumns[i].Title.SortMarker = smUpEh then
          begin
            theIndexFields:=theIndexFields+ Self.SortMarkedColumns[i].FieldName+';';
            theIndexName := theIndexName + Self.SortMarkedColumns[i].FieldName + '_DESC_';
          end
        else
          begin
            theIndexFields:=theIndexFields+ Self.SortMarkedColumns[i].FieldName+';';
            theIndexName := theIndexName + Self.SortMarkedColumns[i].FieldName + '_';
            theDescFields:=theDescFields+ Self.SortMarkedColumns[i].FieldName+';';
          end;

        if theIndexName <> '' then theIndexName := 'idx' + Copy(theIndexName,1,Length(theIndexName)-1);
          theIndexName := DeleteStr(theIndexName,'1');
        if theIndexFields <> '' then theIndexFields := Copy(theIndexFields,1,Length(theIndexFields)-1);
          theIndexFields := DeleteStr(theIndexFields,'1');
        if theDescFields <> '' then theDescFields := Copy(theDescFields,1,Length(theDescFields)-1);
          theDescFields := DeleteStr(theDescFields,'1');

        fIndexPresent:=False;
        theBS_ClientDataSet.IndexDefs.Update;//Refresh
        //If index exist, then set it
        for i := 0 to theBS_ClientDataSet.IndexDefs.Count - 1 do
        if LowerCase(theBS_ClientDataSet.IndexDefs.Items[i].Name) = LowerCase(theIndexName) then
          begin
            fIndexPresent:=True; // Sign
            theBS_ClientDataSet.IndexName := theBS_ClientDataSet.IndexDefs.Items[i].Name;
          end;
        //If index not exist, then try create it
        if Not(fIndexPresent) then
          begin
            theBS_ClientDataSet.AddIndex(theIndexName, theIndexFields,[ixCaseInsensitive],theDescFields,'',0);
          end;

        theBS_ClientDataSet.IndexName := theIndexName;
        Result:=True;
    end;
  finally
    Screen.Cursor:=theOldCursor;
  end;
end;

////////////////////////////////////////////////////////////////////
And there is using filter for searching example(Wark very fast):
procedure TfrmSPORG.edtSearchChange(Sender: TObject);
var
  sFieldsNames  : String;
  i             : integer;
begin
  sFieldsNames:='';
  for i := 0 to FinDM.cdsOrganization.IndexFieldCount - 1 do
    if sFieldsNames='' then
      begin
        sFieldsNames:=sFieldsNames + FinDM.cdsOrganization.IndexFields[i].FieldName;
        break;
      end;
  {$D+L+}
  case FinDM.cdsOrganization.IndexFields[i].DataType of
    ftString, ftFixedChar, ftWideString, ftGuid:
    begin

      if (edtSearch.Text='') or (sFieldsNames='') then
        begin
          FinDM.cdsOrganization.Filtered:=False;
          FinDM.cdsOrganization.First;
        end
      else
        begin
          FinDM.cdsOrganization.Filter:='';
           FinDM.cdsOrganization.Filter:='('+sFieldsNames
             +' LIKE '+ #39+'%'+edtSearch.Text+'%'+#39+')'
             +' or ('+sFieldsNames+'='+#39+edtSearch.Text+#39+')';
        end;
        FinDM.cdsOrganization.Filtered:=True;
        FinDM.cdsOrganization.First;
        FinDM.cdsOrganization.SearchCdsRecord(edtSearch);
    end
  else
    FinDM.cdsOrganization.Filtered:=False;
    FinDM.cdsOrganization.SearchCdsRecord(edtSearch);
  end
end;
 
0
 
LVL 22

Expert Comment

by:Mohammed Nasman
ID: 6468790
as I tested before (but with oracle database)
ADOTable and ADOQuery are much faster than the Table and Query
that's mean use ADO it will be faster than BDE
but still more information needed

0
 
LVL 1

Expert Comment

by:Moondancer
ID: 6976068
ADMINISTRATION WILL BE CONTACTING YOU SHORTLY.  Moderators Computer101 or Netminder will return to finalize these if they are still open in 14 days.  Experts, please post closing recommendations before that time.

Below are your open questions as of today.  Questions which have been inactive for 21 days or longer are considered to be abandoned and for those, your options are:
1. Accept a Comment As Answer (use the button next to the Expert's name).
2. Close the question if the information was not useful to you, but may help others. You must tell the participants why you wish to do this, and allow for Expert response.  This choice will include a refund to you, and will move this question to our PAQ (Previously Asked Question) database.  If you found information outside this question thread, please add it.
3. Ask Community Support to help split points between participating experts, or just comment here with details and we'll respond with the process.
4. Delete the question (if it has no potential value for others).
   --> Post comments for expert of your intention to delete and why
   --> YOU CANNOT DELETE A QUESTION with comments; special handling by a Moderator is required.

For special handling needs, please post a zero point question in the link below and include the URL (question QID/link) that it regards with details.
http://www.experts-exchange.com/jsp/qList.jsp?ta=commspt
 
Please click this link for Help Desk, Guidelines/Member Agreement and the Question/Answer process.  http://www.experts-exchange.com/jsp/cmtyHelpDesk.jsp

Click you Member Profile to view your question history and please keep them updated. If you are a KnowledgePro user, use the Power Search option to find them.  

Questions which are LOCKED with a Proposed Answer but do not help you, should be rejected with comments added.  When you grade the question less than an A, please comment as to why.  This helps all involved, as well as others who may access this item in the future.  PLEASE DO NOT AWARD POINTS TO ME.

To view your open questions, please click the following link(s) and keep them all current with updates.
http://www.experts-exchange.com/questions/Q.20110714.html
http://www.experts-exchange.com/questions/Q.20173874.html
http://www.experts-exchange.com/questions/Q.20170377.html
http://www.experts-exchange.com/questions/Q.20178176.html
http://www.experts-exchange.com/questions/Q.20178177.html
http://www.experts-exchange.com/questions/Q.20229473.html
http://www.experts-exchange.com/questions/Q.20279642.html

To view your locked questions, please click the following link(s) and evaluate the proposed answer.
http://www.experts-exchange.com/questions/Q.20220086.html

*****  E X P E R T S    P L E A S E  ******  Leave your closing recommendations.
If you are interested in the cleanup effort, please click this link
http://www.experts-exchange.com/jsp/qManageQuestion.jsp?ta=commspt&qid=20274643 
POINTS FOR EXPERTS awaiting comments are listed in the link below
http://www.experts-exchange.com/commspt/Q.20277028.html
 
Moderators will finalize this question if in @14 days Asker has not responded.  This will be moved to the PAQ (Previously Asked Questions) at zero points, deleted or awarded.
 
Thanks everyone.
Moondancer
Moderator @ Experts Exchange
0

Featured Post

Free Tool: Postgres Monitoring System

A PHP and Perl based system to collect and display usage statistics from PostgreSQL databases.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Working with hours 3 65
Press three keys together and trigger a function 3 61
control image tags in a string ? 12 142
DBCtrlGrid, Delphi, Scroll 7 26
This article explains how to create forms/units independent of other forms/units object names in a delphi project. Have you ever created a form for user input in a Delphi project and then had the need to have that same form in a other Delphi proj…
Have you ever had your Delphi form/application just hanging while waiting for data to load? This is the article to read if you want to learn some things about adding threads for data loading in the background. First, I'll setup a general applica…
This video shows how to quickly and easily add an email signature for all users on Exchange 2016. The resulting signature is applied on a server level by Exchange Online. The email signature template has been downloaded from: www.mail-signatures…
I've attached the XLSM Excel spreadsheet I used in the video and also text files containing the macros used below. https://filedb.experts-exchange.com/incoming/2017/03_w12/1151775/Permutations.txt https://filedb.experts-exchange.com/incoming/201…

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question