• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 841
  • Last Modified:

What's more efficient? Manually inserting 1 million rows or SSIS?

Let's say we have 1 million rows worth of data in a C# program we need to get into the database.  Roughly, is it more efficient to do 1 million inserts in a loop or to write the data to a flat text file (.csv) on disk and import using SSIS?

The target database table has a few indexes on it and we're finding that the indexing is taking a significant amount of time.

We think this is a "pay now or pay later" situation.  I would guess that there are economies of scale that come into play that make the SSIS overall much more efficient, even though the net indexing is the same either way.

Any guesses on if doing SSIS is 10% more efficient?  50% more efficient?

Comments?
0
ZuZuPetals
Asked:
ZuZuPetals
  • 3
3 Solutions
 
chapmandewCommented:
if there is no data in the table already, then it is more effecient to do a bulk insert statement (which can be done either way).

If users are using the table, then smaller inserts of records to prevent locking the table.
0
 
idmedellinCommented:
I suggest you to use insert sentences, but before, alter the indexes in the table defining the fill factor to about 70%.

The porcentage of efficiency depends from the structure of the table and indexes (number of fields, types, number of indexes, etc.)
0
 
KentMarshCommented:
My $.02

1 BCP is far more efficient than 1 Million calls between the .NET application and the dbms.

MOST Efficient way I know:
1. Turn off access to the table from End Users
2. Remove all Indexes  (if possible)
3. Use a BCP process to add data
4. Restore indexes
5. Turn access back on.

There are other strategies that you may consider depending on your constraints. If the table has insert triggers then that will cause more complications and slow downs. If the data is organized in a particular fashion there are ways of adding Blocks of data at a time - say your data is all for the most recent date and all of the existing data is for previous dates.

0
 
KentMarshCommented:
My other $.02
SSIS was specifically designed to be an efficient ETL solution.
Doing 1 million C# inserts from a .NET application will in no way be faster than importing a CSV file into SSIS.
SSIS will also give you a lot of powerful features that are easier to implement than rolling your own in a C# program.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now