Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

Read a large database into memory more efficiently

Posted on 2006-03-27
6
Medium Priority
?
209 Views
Last Modified: 2010-04-05
I have a program that I would like to make more efficient. A large part of the processing time is spent reading two large databases into memory (about 150,000 records each) so that I can do computations on them. I am reading them in with the construction:

mydatabasetable.first;
while not mydatabasetable.eof do begin
<assign the database fields to various variables>.
mydatabasetable.next;
end;

I am finding that it takes several minutes to read each database. Is there a faster way to get the data into memory?
0
Comment
Question by:riskassessor
6 Comments
 

Expert Comment

by:pilotz
ID: 16309685
i think part "<assign the database fields to various variables>" takes the largest amount of time. check that, e.g. just iterate through mydatabasetable. If so, think, maybe you can change the assigning part, e.g. some computations or operations should be done later on.
0
 
LVL 6

Expert Comment

by:Hardi
ID: 16310629
If you use mydatabasetable['fieldname'], change it to mydatabasetable.FieldByName('fieldname').As...
It may be a little faster
0
 

Author Comment

by:riskassessor
ID: 16329602
My code already implements both the above suggestions. Thanks.
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 1

Expert Comment

by:Kristao
ID: 16341395
unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls, DB, ADODB;

type
  TForm1 = class(TForm)
    ado: TADOQuery;
    Button1: TButton;
    procedure FormCreate(Sender: TObject);
    procedure FormDestroy(Sender: TObject);
    procedure Button1Click(Sender: TObject);
  private
    { Private declarations }
  public
    Buffer: TList;
  end;

type
  pDbData = ^rDbData;
  rDbData = record
    Name: string[255];
    Age: integer;
    Telephone: integer;
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

procedure TForm1.FormCreate(Sender: TObject);
begin
  Buffer := TList.Create;
end;

procedure TForm1.FormDestroy(Sender: TObject);
begin
  while Buffer.Count <> 0 do
  begin
{$I-}
    dispose(Buffer[0]);
    Buffer.Delete(0);
{$I+}
  end;
  freeandnil(Buffer);
end;

procedure TForm1.Button1Click(Sender: TObject);
var
  k: pDbData;
begin
  ado.Open;
  while not (ado.Eof) do
  begin
    new(k);
    k.Name := ado.Fields.Fields[0].AsString;
    k.Age := ado.Fields.Fields[1].AsInteger;
    k.Telephone := ado.Fields.Fields[2].AsInteger;
    Buffer.Add(k);
    ado.Next;
  end;
end;

end.

ado.Fields.Fields[X].assomething my help a little bit. But the datasets are big have you tried to cut datasets in more then one part so mybe you could cut datasets in more parts and work with datasets in more than one thread.
0
 

Author Comment

by:riskassessor
ID: 16347235
Kristao, thanks for algorithm and ideas.

The algorithm you propose is essentially the same as that already in my program.

Regarding your suggestion of working with more than one thread, I have not tried it but would not have thought it would be any quicker on a single-processor machine.

In reply to several commenters, I find it makes little difference to the speed whether I use Fields.Fields[X].As... or FieldByName('fieldname').As... or FieldValues('fieldname').
0
 
LVL 1

Accepted Solution

by:
Kristao earned 200 total points
ID: 16495787
Ok is there is big datasets, i supose you need to use thouse data wich are in dataset. This idea could make your soft a litle bit quicker.

One process reads data from dataset in buffer

Second process take data from bufer and works with them, in this way you don't need to wait until all dataset is loaded in memory.

There will be litle collision in dataput(data:pointer) and dataget(var data:pointer), becouse in multithread u need to use TCriticalSection. > "TCriticalSection allows a thread in a multi-threaded application to temporarily block other threads from accessing a block of code."

I'm using this kind of tehnology my self. My soft get very big dataset there is more than 80 000 records in it, i can't wait until all data are in memory. I Start reading dataset, put the info in buffer, othere process just takes the data from and starts to work with data :). In my case there is 1 datareader and up to 10 dataworkers :)

regards
Kristao.
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Creating an auto free TStringList The TStringList is a basic and frequently used object in Delphi. On many occasions, you may want to create a temporary list, process some items in the list and be done with the list. In such cases, you have to…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
This lesson discusses how to use a Mainform + Subforms in Microsoft Access to find and enter data for payments on orders. The sample data comes from a custom shop that builds and sells movable storage structures that are delivered to your property. …
When cloud platforms entered the scene, users and companies jumped on board to take advantage of the many benefits, like the ability to work and connect with company information from various locations. What many didn't foresee was the increased risk…
Suggested Courses

581 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question