Solved

SSIS Foreach loop Slow and crashes Visual Studio when inserting records.

Posted on 2010-11-19
6
1,273 Views
Last Modified: 2013-11-30
I am inserting records by using foreach loop in SSIS.I am using a foreach loop within a for each loop the first foreach loop gets id from a table and based on that id inserts rows in another table, then the other foreach loop uses the id from the old table and the id from the new table to insert records in another table the number of times that id is referenced in another table and grabbing some records from that table as well.
The problem is the number of original records are arouund 20,000 and these records are referenced around 20 times in another table. so 20,000 x 20 = 4,00,000 records needs to be created total in the table in the innermost loop. But when it starts to insert records it is very fast and as the time progresses it becomes slower and slower and eventually it crashes. What should I do? Is there a property in foreach loop that after every n number of inserts it flushes it's memory after making an insert?
0
Comment
Question by:nkapur
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 7

Accepted Solution

by:
rmm2001 earned 250 total points
ID: 34177743
I'm a little lost on all of your loops but I think it's something that could be a bit faster using some sql. You can use the OUTPUT command to send your newly inserted ids to a temp table (or table variable).

Here's what this is.

You have a SourceTable. It contains some words.
You have a Table1. Which you're inserting some records from SourceTable into based on some criteria. You create a temp table to hold the values that you're going to be inserting that come from SourceTable (IDList1). Then you have a temp table to hold the values that came from the insert from Table1. Finally you insert the values from your filtered table (IDList1) into Table1 and output the rows inserted into, yet another temp table (IDList2).

Doing it this way you can keep control of your ids just as well as you would in a foreach container but it'll be much faster. You'll probably need a few more temp tables than what's in here but it's a start. you can put this in an execute sql task and run it in the package.
CREATE TABLE #SourceTable (ID INT IDENTITY, Words VARCHAR(50))
INSERT #SourceTable (Words)
SELECT 'asdf' UNION
SELECT'hjkl' UNION
SELECT '12345'

CREATE TABLE #Table1 (Table1ID INT IDENTITY, ID INT, Words VARCHAR(50))

DECLARE @IDList1 TABLE ( ID INT, Words VARCHAR(50) )
INSERT @IDList1
SELECT ID, Words
FROM #SourceTable
--WHERE {your criteria}

DECLARE @IDList2 TABLE (Table1ID INT, ID INT, Words VARCHAR(50) )

--(Table1ID is assuemd to be identity)
INSERT #Table1 (ID, Words)
OUTPUT inserted.* INTO @IDList2
SELECT ID, Words
FROM @IDList1


SELECT * FROM @IDList2

Open in new window

0
 
LVL 30

Expert Comment

by:Reza Rad
ID: 34178514
why you do INSERT in control flow task with Foreach ?
why you don't use Data flow task which is dedicated to data transfer?
what is your source and destination?

tell us what exactly you want to do , and we tell you right way
0
 

Author Comment

by:nkapur
ID: 34180084
okay let me elaborate. There is a product table from where product ID needs to be grabbed and this product ID will be inserted in a new product table. Then the new product ID and the old product ID will be used to insert records in a pricing table. Since the pricing table has some values from the new pricing table and some from the old pricing table. So the old product ID and the new product ID should correspond to one another, there is no column common in the old product table and the new product table to relate the new them, so I used a loop which will insert records and match the new product id and the old product id together using new product id and the old product id to grab values from those tables.
But this is not it the pricing table will have records according to the catalog. So if there are six catalog the product will appear in six catalogs and the price for each catalog will vary for the same product. Now there will be another loop within this for each loop in which we will grab each catalog id and insert the new product id , the old product id, the catalog id and insert records in the pricing table. There is a price column in the pricing table in which I have written a query that will calculate the price based on the catalog id.
0
Online Training Solution

Drastically shorten your training time with WalkMe's advanced online training solution that Guides your trainees to action. Forget about retraining and skyrocket knowledge retention rates.

 
LVL 7

Expert Comment

by:rmm2001
ID: 34180112
Did you try putting this using queries yet?
0
 
LVL 30

Expert Comment

by:Reza Rad
ID: 34180265
It's really hard to understand for me!
could you put some screenshots?
0
 
LVL 21

Expert Comment

by:Alpesh Patel
ID: 34185863
May be slow due the system configuration is not sufficient.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

JSON is being used more and more, besides XML, and you surely wanted to parse the data out into SQL instead of doing it in some Javascript. The below function in SQL Server can do the job for you, returning a quick table with the parsed data.
In this article we will learn how to fix  “Cannot install SQL Server 2014 Service Pack 2: Unable to install windows installer msi file” error ?
Via a live example, show how to backup a database, simulate a failure backup the tail of the database transaction log and perform the restore.
Viewers will learn how to use the INSERT statement to insert data into their tables. It will also introduce the NULL statement, to show them what happens when no value is giving for any given column.

689 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question