Improve company productivity with a Business Account.Sign Up

x
?
Solved

What's more efficient?  Manually inserting 1 million rows or SSIS?

Posted on 2011-03-23
5
Medium Priority
?
843 Views
Last Modified: 2012-05-11
Let's say we have 1 million rows worth of data in a C# program we need to get into the database.  Roughly, is it more efficient to do 1 million inserts in a loop or to write the data to a flat text file (.csv) on disk and import using SSIS?

The target database table has a few indexes on it and we're finding that the indexing is taking a significant amount of time.

We think this is a "pay now or pay later" situation.  I would guess that there are economies of scale that come into play that make the SSIS overall much more efficient, even though the net indexing is the same either way.

Any guesses on if doing SSIS is 10% more efficient?  50% more efficient?

Comments?
0
Comment
Question by:ZuZuPetals
  • 3
5 Comments
 
LVL 60

Accepted Solution

by:
chapmandew earned 668 total points
ID: 35199468
if there is no data in the table already, then it is more effecient to do a bulk insert statement (which can be done either way).

If users are using the table, then smaller inserts of records to prevent locking the table.
0
 
LVL 5

Assisted Solution

by:idmedellin
idmedellin earned 664 total points
ID: 35199488
I suggest you to use insert sentences, but before, alter the indexes in the table defining the fill factor to about 70%.

The porcentage of efficiency depends from the structure of the table and indexes (number of fields, types, number of indexes, etc.)
0
 
LVL 2

Assisted Solution

by:KentMarsh
KentMarsh earned 668 total points
ID: 35207426
My $.02

1 BCP is far more efficient than 1 Million calls between the .NET application and the dbms.

MOST Efficient way I know:
1. Turn off access to the table from End Users
2. Remove all Indexes  (if possible)
3. Use a BCP process to add data
4. Restore indexes
5. Turn access back on.

There are other strategies that you may consider depending on your constraints. If the table has insert triggers then that will cause more complications and slow downs. If the data is organized in a particular fashion there are ways of adding Blocks of data at a time - say your data is all for the most recent date and all of the existing data is for previous dates.

0
 
LVL 2

Expert Comment

by:KentMarsh
ID: 35207521
My other $.02
SSIS was specifically designed to be an efficient ETL solution.
Doing 1 million C# inserts from a .NET application will in no way be faster than importing a CSV file into SSIS.
SSIS will also give you a lot of powerful features that are easier to implement than rolling your own in a C# program.
0
 
LVL 2

Expert Comment

by:KentMarsh
ID: 35207554
0

Featured Post

What Kind of Coding Program is Right for You?

There are many ways to learn to code these days. From coding bootcamps like Flatiron School to online courses to totally free beginner resources. The best way to learn to code depends on many factors, but the most important one is you. See what course is best for you.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

A Stored Procedure in Microsoft SQL Server is a powerful feature that it can be used to execute the Data Manipulation Language (DML) or Data Definition Language (DDL). Depending on business requirements, a single Stored Procedure can return differe…
An alternative to the "For XML" way of pivoting and concatenating result sets into strings, and an easy introduction to "common table expressions" (CTEs). Being someone who is always looking for alternatives to "work your data", I came across this …
Via a live example, show how to extract information from SQL Server on Database, Connection and Server properties
Using examples as well as descriptions, and references to Books Online, show the documentation available for datatypes, explain the available data types and show how data can be passed into and out of variables.

595 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question