Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Quick save to ADO

Posted on 2004-08-05
4
Medium Priority
?
492 Views
Last Modified: 2013-11-23
We have an application that does a lot of data manipulation in memory using dynamic arrays. We have recently tried using ADO as the data storage mechanism, replacing 20 files with a single Access database with 20 tables.

It's great, except saving using APPEND/EDIT,{code}, then  POST is so SLOW when you want to do a bulk save (all records) to a single table. This is not a standard data-aware application that saves as you go. Is there a quick way to do a bulk save (savetotable?) using ADO.

   
0
Comment
Question by:Dennis_Treacy
4 Comments
 
LVL 3

Accepted Solution

by:
huferry earned 150 total points
ID: 11725137
Try using TADOCommand and inserting the records using an SQL Command:

INSERT INTO MyTable (Field1,Field2..) VALUES (Value1,Value2...)

It might help, it saves a few statements. If you use the Post method, you
probably populate the fields one by one:

   FieldValues['Field1'] := Value1;
   etc...

regards,
huferry
0
 
LVL 17

Assisted Solution

by:Wim ten Brink
Wim ten Brink earned 300 total points
ID: 11727765
I have to disappoint you... Any database action will be slower than just file manipulation actions. Databases have a lot of overhead for handling all the records. If you really need speed, don't use databases...
I had to write an application once that had to read half a million records from a textfile to send them to either an Oracle or SQL Server database. The whole reading, parsing and checking of the records in the textfile was done within two minutes. Sending the records to the database did add another half hour to an hour to the whole process. And there wasn't much that could be done about it. If I would just write it to another textfile, the whole application would run within 3 to 4 minutes...

Databases are powerful but there's always a cost. And this cost is the speed...

But okay, a quick option might be to build a large SQL query. For every record you need to add, add an "Insert into xxx (...) values (...);" to a string. This way, build a string with as many insert statements as you have records. Then execute that single query... This might or might not speed things up a bit.
0
 
LVL 4

Assisted Solution

by:Greg Rowland
Greg Rowland earned 300 total points
ID: 11730375
You could try using ClientDataSets and replacing your Arrays with DBGrids “whether or not they are visible”.
Then Apply updates periodically at appropriate moments during processing.
The nice thing ClientDatasets.ApplyUpdates will marshal the changed data for you. Although ClientDataSets can be a little tricky at first, surely the crew here at EE can help there.

Typically I would start with and ADOConnection component, connected to an ADOQuery, or ADOTable then a DataSetProvider connected to a ClientDataSet, finally a DataSource and DBGrid. The grids data can be index/enumerated just like an array; multiple grids can be layered to provided multi-dimensionality. Because ClientDataSets are in memory the performance should be pretty good. StringGrids also work real well if you what to view the processing results, and don’t mind doing type conversions on the fly.

If you do try this, pay careful attention the individual fields provider flags. While very useful limit the number of fields in the “where clause”.
“Date” data types also have some rather interesting anomalies when used with ClientDataSet.

Just an Idea.
 


0
 

Author Comment

by:Dennis_Treacy
ID: 11732566
Thanks for your comments.

huferry - the extra time taken is with the Append/Edit funtion, not Post. I expect what you have suggested will take the same time.

Alex - you have confirmed what I thought was the case. I was hoping I was wrong.

Surfer Joe - thanks. Worth a try.

Split points ok?
0

Featured Post

How to Use the Help Bell

Need to boost the visibility of your question for solutions? Use the Experts Exchange Help Bell to confirm priority levels and contact subject-matter experts for question attention.  Check out this how-to article for more information.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction The parallel port is a very commonly known port, it was widely used to connect a printer to the PC, if you look at the back of your computer, for those who don't have newer computers, there will be a port with 25 pins and a small print…
Creating an auto free TStringList The TStringList is a basic and frequently used object in Delphi. On many occasions, you may want to create a temporary list, process some items in the list and be done with the list. In such cases, you have to…
this video summaries big data hadoop online training demo (http://onlineitguru.com/big-data-hadoop-online-training-placement.html) , and covers basics in big data hadoop .
Integration Management Part 2

877 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question