Solved

SSIS Foreach loop Slow and crashes Visual Studio when inserting records.

Posted on 2010-11-19
6
1,268 Views
Last Modified: 2013-11-30
I am inserting records by using foreach loop in SSIS.I am using a foreach loop within a for each loop the first foreach loop gets id from a table and based on that id inserts rows in another table, then the other foreach loop uses the id from the old table and the id from the new table to insert records in another table the number of times that id is referenced in another table and grabbing some records from that table as well.
The problem is the number of original records are arouund 20,000 and these records are referenced around 20 times in another table. so 20,000 x 20 = 4,00,000 records needs to be created total in the table in the innermost loop. But when it starts to insert records it is very fast and as the time progresses it becomes slower and slower and eventually it crashes. What should I do? Is there a property in foreach loop that after every n number of inserts it flushes it's memory after making an insert?
0
Comment
Question by:nkapur
6 Comments
 
LVL 7

Accepted Solution

by:
rmm2001 earned 250 total points
ID: 34177743
I'm a little lost on all of your loops but I think it's something that could be a bit faster using some sql. You can use the OUTPUT command to send your newly inserted ids to a temp table (or table variable).

Here's what this is.

You have a SourceTable. It contains some words.
You have a Table1. Which you're inserting some records from SourceTable into based on some criteria. You create a temp table to hold the values that you're going to be inserting that come from SourceTable (IDList1). Then you have a temp table to hold the values that came from the insert from Table1. Finally you insert the values from your filtered table (IDList1) into Table1 and output the rows inserted into, yet another temp table (IDList2).

Doing it this way you can keep control of your ids just as well as you would in a foreach container but it'll be much faster. You'll probably need a few more temp tables than what's in here but it's a start. you can put this in an execute sql task and run it in the package.
CREATE TABLE #SourceTable (ID INT IDENTITY, Words VARCHAR(50))
INSERT #SourceTable (Words)
SELECT 'asdf' UNION
SELECT'hjkl' UNION
SELECT '12345'

CREATE TABLE #Table1 (Table1ID INT IDENTITY, ID INT, Words VARCHAR(50))

DECLARE @IDList1 TABLE ( ID INT, Words VARCHAR(50) )
INSERT @IDList1
SELECT ID, Words
FROM #SourceTable
--WHERE {your criteria}

DECLARE @IDList2 TABLE (Table1ID INT, ID INT, Words VARCHAR(50) )

--(Table1ID is assuemd to be identity)
INSERT #Table1 (ID, Words)
OUTPUT inserted.* INTO @IDList2
SELECT ID, Words
FROM @IDList1


SELECT * FROM @IDList2

Open in new window

0
 
LVL 30

Expert Comment

by:Reza Rad
ID: 34178514
why you do INSERT in control flow task with Foreach ?
why you don't use Data flow task which is dedicated to data transfer?
what is your source and destination?

tell us what exactly you want to do , and we tell you right way
0
 

Author Comment

by:nkapur
ID: 34180084
okay let me elaborate. There is a product table from where product ID needs to be grabbed and this product ID will be inserted in a new product table. Then the new product ID and the old product ID will be used to insert records in a pricing table. Since the pricing table has some values from the new pricing table and some from the old pricing table. So the old product ID and the new product ID should correspond to one another, there is no column common in the old product table and the new product table to relate the new them, so I used a loop which will insert records and match the new product id and the old product id together using new product id and the old product id to grab values from those tables.
But this is not it the pricing table will have records according to the catalog. So if there are six catalog the product will appear in six catalogs and the price for each catalog will vary for the same product. Now there will be another loop within this for each loop in which we will grab each catalog id and insert the new product id , the old product id, the catalog id and insert records in the pricing table. There is a price column in the pricing table in which I have written a query that will calculate the price based on the catalog id.
0
Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

 
LVL 7

Expert Comment

by:rmm2001
ID: 34180112
Did you try putting this using queries yet?
0
 
LVL 30

Expert Comment

by:Reza Rad
ID: 34180265
It's really hard to understand for me!
could you put some screenshots?
0
 
LVL 21

Expert Comment

by:Alpesh Patel
ID: 34185863
May be slow due the system configuration is not sufficient.
0

Featured Post

NAS Cloud Backup Strategies

This article explains backup scenarios when using network storage. We review the so-called “3-2-1 strategy” and summarize the methods you can use to send NAS data to the cloud

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Get the latest status 8 32
SQL Server 2012 r2 - Varible Table 3 24
What's wrong with this T-SQL Foreign Key? 7 45
SQL Query assistance 16 25
Everyone has problem when going to load data into Data warehouse (EDW). They all need to confirm that data quality is good but they don't no how to proceed. Microsoft has provided new task within SSIS 2008 called "Data Profiler Task". It solve th…
Why is this different from all of the other step by step guides?  Because I make a living as a DBA and not as a writer and I lived through this experience. Defining the name: When I talk to people they say different names on this subject stuff l…
Using examples as well as descriptions, and references to Books Online, show the documentation available for date manipulation functions and by using a select few of these functions, show how date based data can be manipulated with these functions.
Via a live example, show how to extract insert data into a SQL Server database table using the Import/Export option and Bulk Insert.

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question