• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 497
  • Last Modified:

Quick save to ADO

We have an application that does a lot of data manipulation in memory using dynamic arrays. We have recently tried using ADO as the data storage mechanism, replacing 20 files with a single Access database with 20 tables.

It's great, except saving using APPEND/EDIT,{code}, then  POST is so SLOW when you want to do a bulk save (all records) to a single table. This is not a standard data-aware application that saves as you go. Is there a quick way to do a bulk save (savetotable?) using ADO.

   
0
Dennis_Treacy
Asked:
Dennis_Treacy
3 Solutions
 
huferryCommented:
Try using TADOCommand and inserting the records using an SQL Command:

INSERT INTO MyTable (Field1,Field2..) VALUES (Value1,Value2...)

It might help, it saves a few statements. If you use the Post method, you
probably populate the fields one by one:

   FieldValues['Field1'] := Value1;
   etc...

regards,
huferry
0
 
Wim ten BrinkSelf-employed developerCommented:
I have to disappoint you... Any database action will be slower than just file manipulation actions. Databases have a lot of overhead for handling all the records. If you really need speed, don't use databases...
I had to write an application once that had to read half a million records from a textfile to send them to either an Oracle or SQL Server database. The whole reading, parsing and checking of the records in the textfile was done within two minutes. Sending the records to the database did add another half hour to an hour to the whole process. And there wasn't much that could be done about it. If I would just write it to another textfile, the whole application would run within 3 to 4 minutes...

Databases are powerful but there's always a cost. And this cost is the speed...

But okay, a quick option might be to build a large SQL query. For every record you need to add, add an "Insert into xxx (...) values (...);" to a string. This way, build a string with as many insert statements as you have records. Then execute that single query... This might or might not speed things up a bit.
0
 
Greg RowlandSoftware Designer, SysDBA, WebMaster OwnerCommented:
You could try using ClientDataSets and replacing your Arrays with DBGrids “whether or not they are visible”.
Then Apply updates periodically at appropriate moments during processing.
The nice thing ClientDatasets.ApplyUpdates will marshal the changed data for you. Although ClientDataSets can be a little tricky at first, surely the crew here at EE can help there.

Typically I would start with and ADOConnection component, connected to an ADOQuery, or ADOTable then a DataSetProvider connected to a ClientDataSet, finally a DataSource and DBGrid. The grids data can be index/enumerated just like an array; multiple grids can be layered to provided multi-dimensionality. Because ClientDataSets are in memory the performance should be pretty good. StringGrids also work real well if you what to view the processing results, and don’t mind doing type conversions on the fly.

If you do try this, pay careful attention the individual fields provider flags. While very useful limit the number of fields in the “where clause”.
“Date” data types also have some rather interesting anomalies when used with ClientDataSet.

Just an Idea.
 


0
 
Dennis_TreacyAuthor Commented:
Thanks for your comments.

huferry - the extra time taken is with the Append/Edit funtion, not Post. I expect what you have suggested will take the same time.

Alex - you have confirmed what I thought was the case. I was hoping I was wrong.

Surfer Joe - thanks. Worth a try.

Split points ok?
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now