?
Solved

SSIS Foreach loop Slow and crashes Visual Studio when inserting records.

Posted on 2010-11-19
6
Medium Priority
?
1,293 Views
Last Modified: 2013-11-30
I am inserting records by using foreach loop in SSIS.I am using a foreach loop within a for each loop the first foreach loop gets id from a table and based on that id inserts rows in another table, then the other foreach loop uses the id from the old table and the id from the new table to insert records in another table the number of times that id is referenced in another table and grabbing some records from that table as well.
The problem is the number of original records are arouund 20,000 and these records are referenced around 20 times in another table. so 20,000 x 20 = 4,00,000 records needs to be created total in the table in the innermost loop. But when it starts to insert records it is very fast and as the time progresses it becomes slower and slower and eventually it crashes. What should I do? Is there a property in foreach loop that after every n number of inserts it flushes it's memory after making an insert?
0
Comment
Question by:nkapur
6 Comments
 
LVL 7

Accepted Solution

by:
rmm2001 earned 1000 total points
ID: 34177743
I'm a little lost on all of your loops but I think it's something that could be a bit faster using some sql. You can use the OUTPUT command to send your newly inserted ids to a temp table (or table variable).

Here's what this is.

You have a SourceTable. It contains some words.
You have a Table1. Which you're inserting some records from SourceTable into based on some criteria. You create a temp table to hold the values that you're going to be inserting that come from SourceTable (IDList1). Then you have a temp table to hold the values that came from the insert from Table1. Finally you insert the values from your filtered table (IDList1) into Table1 and output the rows inserted into, yet another temp table (IDList2).

Doing it this way you can keep control of your ids just as well as you would in a foreach container but it'll be much faster. You'll probably need a few more temp tables than what's in here but it's a start. you can put this in an execute sql task and run it in the package.
CREATE TABLE #SourceTable (ID INT IDENTITY, Words VARCHAR(50))
INSERT #SourceTable (Words)
SELECT 'asdf' UNION
SELECT'hjkl' UNION
SELECT '12345'

CREATE TABLE #Table1 (Table1ID INT IDENTITY, ID INT, Words VARCHAR(50))

DECLARE @IDList1 TABLE ( ID INT, Words VARCHAR(50) )
INSERT @IDList1
SELECT ID, Words
FROM #SourceTable
--WHERE {your criteria}

DECLARE @IDList2 TABLE (Table1ID INT, ID INT, Words VARCHAR(50) )

--(Table1ID is assuemd to be identity)
INSERT #Table1 (ID, Words)
OUTPUT inserted.* INTO @IDList2
SELECT ID, Words
FROM @IDList1


SELECT * FROM @IDList2

Open in new window

0
 
LVL 30

Expert Comment

by:Reza Rad
ID: 34178514
why you do INSERT in control flow task with Foreach ?
why you don't use Data flow task which is dedicated to data transfer?
what is your source and destination?

tell us what exactly you want to do , and we tell you right way
0
 

Author Comment

by:nkapur
ID: 34180084
okay let me elaborate. There is a product table from where product ID needs to be grabbed and this product ID will be inserted in a new product table. Then the new product ID and the old product ID will be used to insert records in a pricing table. Since the pricing table has some values from the new pricing table and some from the old pricing table. So the old product ID and the new product ID should correspond to one another, there is no column common in the old product table and the new product table to relate the new them, so I used a loop which will insert records and match the new product id and the old product id together using new product id and the old product id to grab values from those tables.
But this is not it the pricing table will have records according to the catalog. So if there are six catalog the product will appear in six catalogs and the price for each catalog will vary for the same product. Now there will be another loop within this for each loop in which we will grab each catalog id and insert the new product id , the old product id, the catalog id and insert records in the pricing table. There is a price column in the pricing table in which I have written a query that will calculate the price based on the catalog id.
0
Receive 1:1 tech help

Solve your biggest tech problems alongside global tech experts with 1:1 help.

 
LVL 7

Expert Comment

by:rmm2001
ID: 34180112
Did you try putting this using queries yet?
0
 
LVL 30

Expert Comment

by:Reza Rad
ID: 34180265
It's really hard to understand for me!
could you put some screenshots?
0
 
LVL 21

Expert Comment

by:Alpesh Patel
ID: 34185863
May be slow due the system configuration is not sufficient.
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Recently we ran in to an issue while running some SQL jobs where we were trying to process the cubes.  We got an error saying failure stating 'NT SERVICE\SQLSERVERAGENT does not have access to Analysis Services. So this is a way to automate that wit…
Ready to get certified? Check out some courses that help you prepare for third-party exams.
Using examples as well as descriptions, and references to Books Online, show the documentation available for date manipulation functions and by using a select few of these functions, show how date based data can be manipulated with these functions.
SQL Database Recovery Software repairs the MDF & NDF Files, corrupted due to hardware related issues or software related errors. Provides preview of recovered database objects and allows saving in either MSSQL, CSV, HTML or XLS format. Ensures recov…

598 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question