Solved

What's more efficient?  Manually inserting 1 million rows or SSIS?

Posted on 2011-03-23
5
793 Views
Last Modified: 2012-05-11
Let's say we have 1 million rows worth of data in a C# program we need to get into the database.  Roughly, is it more efficient to do 1 million inserts in a loop or to write the data to a flat text file (.csv) on disk and import using SSIS?

The target database table has a few indexes on it and we're finding that the indexing is taking a significant amount of time.

We think this is a "pay now or pay later" situation.  I would guess that there are economies of scale that come into play that make the SSIS overall much more efficient, even though the net indexing is the same either way.

Any guesses on if doing SSIS is 10% more efficient?  50% more efficient?

Comments?
0
Comment
Question by:ZuZuPetals
  • 3
5 Comments
 
LVL 60

Accepted Solution

by:
chapmandew earned 167 total points
Comment Utility
if there is no data in the table already, then it is more effecient to do a bulk insert statement (which can be done either way).

If users are using the table, then smaller inserts of records to prevent locking the table.
0
 
LVL 5

Assisted Solution

by:idmedellin
idmedellin earned 166 total points
Comment Utility
I suggest you to use insert sentences, but before, alter the indexes in the table defining the fill factor to about 70%.

The porcentage of efficiency depends from the structure of the table and indexes (number of fields, types, number of indexes, etc.)
0
 
LVL 2

Assisted Solution

by:KentMarsh
KentMarsh earned 167 total points
Comment Utility
My $.02

1 BCP is far more efficient than 1 Million calls between the .NET application and the dbms.

MOST Efficient way I know:
1. Turn off access to the table from End Users
2. Remove all Indexes  (if possible)
3. Use a BCP process to add data
4. Restore indexes
5. Turn access back on.

There are other strategies that you may consider depending on your constraints. If the table has insert triggers then that will cause more complications and slow downs. If the data is organized in a particular fashion there are ways of adding Blocks of data at a time - say your data is all for the most recent date and all of the existing data is for previous dates.

0
 
LVL 2

Expert Comment

by:KentMarsh
Comment Utility
My other $.02
SSIS was specifically designed to be an efficient ETL solution.
Doing 1 million C# inserts from a .NET application will in no way be faster than importing a CSV file into SSIS.
SSIS will also give you a lot of powerful features that are easier to implement than rolling your own in a C# program.
0
 
LVL 2

Expert Comment

by:KentMarsh
Comment Utility
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Join & Write a Comment

Everyone has problem when going to load data into Data warehouse (EDW). They all need to confirm that data quality is good but they don't no how to proceed. Microsoft has provided new task within SSIS 2008 called "Data Profiler Task". It solve th…
Ever needed a SQL 2008 Database replicated/mirrored/log shipped on another server but you can't take the downtime inflicted by initial snapshot or disconnect while T-logs are restored or mirror applied? You can use SQL Server Initialize from Backup…
Via a live example, show how to backup a database, simulate a failure backup the tail of the database transaction log and perform the restore.
Viewers will learn how to use the SELECT statement in SQL and will be exposed to the many uses the SELECT statement has.

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now