Solved

Quick save to ADO

Posted on 2004-08-05
4
474 Views
Last Modified: 2013-11-23
We have an application that does a lot of data manipulation in memory using dynamic arrays. We have recently tried using ADO as the data storage mechanism, replacing 20 files with a single Access database with 20 tables.

It's great, except saving using APPEND/EDIT,{code}, then  POST is so SLOW when you want to do a bulk save (all records) to a single table. This is not a standard data-aware application that saves as you go. Is there a quick way to do a bulk save (savetotable?) using ADO.

   
0
Comment
Question by:Dennis_Treacy
4 Comments
 
LVL 3

Accepted Solution

by:
huferry earned 50 total points
ID: 11725137
Try using TADOCommand and inserting the records using an SQL Command:

INSERT INTO MyTable (Field1,Field2..) VALUES (Value1,Value2...)

It might help, it saves a few statements. If you use the Post method, you
probably populate the fields one by one:

   FieldValues['Field1'] := Value1;
   etc...

regards,
huferry
0
 
LVL 17

Assisted Solution

by:Wim ten Brink
Wim ten Brink earned 100 total points
ID: 11727765
I have to disappoint you... Any database action will be slower than just file manipulation actions. Databases have a lot of overhead for handling all the records. If you really need speed, don't use databases...
I had to write an application once that had to read half a million records from a textfile to send them to either an Oracle or SQL Server database. The whole reading, parsing and checking of the records in the textfile was done within two minutes. Sending the records to the database did add another half hour to an hour to the whole process. And there wasn't much that could be done about it. If I would just write it to another textfile, the whole application would run within 3 to 4 minutes...

Databases are powerful but there's always a cost. And this cost is the speed...

But okay, a quick option might be to build a large SQL query. For every record you need to add, add an "Insert into xxx (...) values (...);" to a string. This way, build a string with as many insert statements as you have records. Then execute that single query... This might or might not speed things up a bit.
0
 
LVL 4

Assisted Solution

by:SurferJoe
SurferJoe earned 100 total points
ID: 11730375
You could try using ClientDataSets and replacing your Arrays with DBGrids “whether or not they are visible”.
Then Apply updates periodically at appropriate moments during processing.
The nice thing ClientDatasets.ApplyUpdates will marshal the changed data for you. Although ClientDataSets can be a little tricky at first, surely the crew here at EE can help there.

Typically I would start with and ADOConnection component, connected to an ADOQuery, or ADOTable then a DataSetProvider connected to a ClientDataSet, finally a DataSource and DBGrid. The grids data can be index/enumerated just like an array; multiple grids can be layered to provided multi-dimensionality. Because ClientDataSets are in memory the performance should be pretty good. StringGrids also work real well if you what to view the processing results, and don’t mind doing type conversions on the fly.

If you do try this, pay careful attention the individual fields provider flags. While very useful limit the number of fields in the “where clause”.
“Date” data types also have some rather interesting anomalies when used with ClientDataSet.

Just an Idea.
 


0
 

Author Comment

by:Dennis_Treacy
ID: 11732566
Thanks for your comments.

huferry - the extra time taken is with the Append/Edit funtion, not Post. I expect what you have suggested will take the same time.

Alex - you have confirmed what I thought was the case. I was hoping I was wrong.

Surfer Joe - thanks. Worth a try.

Split points ok?
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Have you ever had your Delphi form/application just hanging while waiting for data to load? This is the article to read if you want to learn some things about adding threads for data loading in the background. First, I'll setup a general applica…
Introduction Raise your hands if you were as upset with FireMonkey as I was when I discovered that there was no TListview.  I use TListView in almost all of my applications I've written, and I was not going to compromise by resorting to TStringGrid…
Access reports are powerful and flexible. Learn how to create a query and then a grouped report using the wizard. Modify the report design after the wizard is done to make it look better. There will be another video to explain how to put the final p…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now