Solved

Quick save to ADO

Posted on 2004-08-05
4
482 Views
Last Modified: 2013-11-23
We have an application that does a lot of data manipulation in memory using dynamic arrays. We have recently tried using ADO as the data storage mechanism, replacing 20 files with a single Access database with 20 tables.

It's great, except saving using APPEND/EDIT,{code}, then  POST is so SLOW when you want to do a bulk save (all records) to a single table. This is not a standard data-aware application that saves as you go. Is there a quick way to do a bulk save (savetotable?) using ADO.

   
0
Comment
Question by:Dennis_Treacy
4 Comments
 
LVL 3

Accepted Solution

by:
huferry earned 50 total points
ID: 11725137
Try using TADOCommand and inserting the records using an SQL Command:

INSERT INTO MyTable (Field1,Field2..) VALUES (Value1,Value2...)

It might help, it saves a few statements. If you use the Post method, you
probably populate the fields one by one:

   FieldValues['Field1'] := Value1;
   etc...

regards,
huferry
0
 
LVL 17

Assisted Solution

by:Wim ten Brink
Wim ten Brink earned 100 total points
ID: 11727765
I have to disappoint you... Any database action will be slower than just file manipulation actions. Databases have a lot of overhead for handling all the records. If you really need speed, don't use databases...
I had to write an application once that had to read half a million records from a textfile to send them to either an Oracle or SQL Server database. The whole reading, parsing and checking of the records in the textfile was done within two minutes. Sending the records to the database did add another half hour to an hour to the whole process. And there wasn't much that could be done about it. If I would just write it to another textfile, the whole application would run within 3 to 4 minutes...

Databases are powerful but there's always a cost. And this cost is the speed...

But okay, a quick option might be to build a large SQL query. For every record you need to add, add an "Insert into xxx (...) values (...);" to a string. This way, build a string with as many insert statements as you have records. Then execute that single query... This might or might not speed things up a bit.
0
 
LVL 4

Assisted Solution

by:Greg Rowland
Greg Rowland earned 100 total points
ID: 11730375
You could try using ClientDataSets and replacing your Arrays with DBGrids “whether or not they are visible”.
Then Apply updates periodically at appropriate moments during processing.
The nice thing ClientDatasets.ApplyUpdates will marshal the changed data for you. Although ClientDataSets can be a little tricky at first, surely the crew here at EE can help there.

Typically I would start with and ADOConnection component, connected to an ADOQuery, or ADOTable then a DataSetProvider connected to a ClientDataSet, finally a DataSource and DBGrid. The grids data can be index/enumerated just like an array; multiple grids can be layered to provided multi-dimensionality. Because ClientDataSets are in memory the performance should be pretty good. StringGrids also work real well if you what to view the processing results, and don’t mind doing type conversions on the fly.

If you do try this, pay careful attention the individual fields provider flags. While very useful limit the number of fields in the “where clause”.
“Date” data types also have some rather interesting anomalies when used with ClientDataSet.

Just an Idea.
 


0
 

Author Comment

by:Dennis_Treacy
ID: 11732566
Thanks for your comments.

huferry - the extra time taken is with the Append/Edit funtion, not Post. I expect what you have suggested will take the same time.

Alex - you have confirmed what I thought was the case. I was hoping I was wrong.

Surfer Joe - thanks. Worth a try.

Split points ok?
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Have you ever had your Delphi form/application just hanging while waiting for data to load? This is the article to read if you want to learn some things about adding threads for data loading in the background. First, I'll setup a general applica…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
Email security requires an ever evolving service that stays up to date with counter-evolving threats. The Email Laundry perform Research and Development to ensure their email security service evolves faster than cyber criminals. We apply our Threat…
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…

828 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question