Insert and Index Overhead

tonypatton21
tonypatton21 used Ask the Experts™
on
I am working on a file import of 43 million records on a table that is truncated and reloaded with this import each week. There is an index on one of the fields.  If I drop the index and then reload the table , and re-create the index, the process finishes in 10 mins.  If I don't drop the index first and try to insert, the process doesn't finish in an acceptable timeframe. Where is all the extra overhead for the insert when the index is left there? Is this purely a disk bottleneck or more than that?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Commented:
>Where is all the extra overhead for the insert when the index is left there?

The extra overhead comes from random out-of-sequence inserts to the index that fragments the index whilst it is built.  Adding the index later is a sequential process that sorts before building the index.

You may get some additional benefit by using bulk logging during the import, if that's feasible.  .

 

 

Commented:
Also, it it's a clustered index and the import is out-of-sequence, you are continuosly rearranging the data pages during the load.

Author

Commented:
Yes, I changed the recovery model to bulk logged and the bulk insert is functioning pretty well.  So should I leave the import as-is where I programmatically drop index, insert data, and rebuild index?

Commented:
Yes, unless you have some compelling reason not to.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial