Solved

SSIS Foreach loop Slow and crashes Visual Studio when inserting records.

Posted on 2010-11-19
6
1,263 Views
Last Modified: 2013-11-30
I am inserting records by using foreach loop in SSIS.I am using a foreach loop within a for each loop the first foreach loop gets id from a table and based on that id inserts rows in another table, then the other foreach loop uses the id from the old table and the id from the new table to insert records in another table the number of times that id is referenced in another table and grabbing some records from that table as well.
The problem is the number of original records are arouund 20,000 and these records are referenced around 20 times in another table. so 20,000 x 20 = 4,00,000 records needs to be created total in the table in the innermost loop. But when it starts to insert records it is very fast and as the time progresses it becomes slower and slower and eventually it crashes. What should I do? Is there a property in foreach loop that after every n number of inserts it flushes it's memory after making an insert?
0
Comment
Question by:nkapur
6 Comments
 
LVL 7

Accepted Solution

by:
rmm2001 earned 250 total points
ID: 34177743
I'm a little lost on all of your loops but I think it's something that could be a bit faster using some sql. You can use the OUTPUT command to send your newly inserted ids to a temp table (or table variable).

Here's what this is.

You have a SourceTable. It contains some words.
You have a Table1. Which you're inserting some records from SourceTable into based on some criteria. You create a temp table to hold the values that you're going to be inserting that come from SourceTable (IDList1). Then you have a temp table to hold the values that came from the insert from Table1. Finally you insert the values from your filtered table (IDList1) into Table1 and output the rows inserted into, yet another temp table (IDList2).

Doing it this way you can keep control of your ids just as well as you would in a foreach container but it'll be much faster. You'll probably need a few more temp tables than what's in here but it's a start. you can put this in an execute sql task and run it in the package.
CREATE TABLE #SourceTable (ID INT IDENTITY, Words VARCHAR(50))
INSERT #SourceTable (Words)
SELECT 'asdf' UNION
SELECT'hjkl' UNION
SELECT '12345'

CREATE TABLE #Table1 (Table1ID INT IDENTITY, ID INT, Words VARCHAR(50))

DECLARE @IDList1 TABLE ( ID INT, Words VARCHAR(50) )
INSERT @IDList1
SELECT ID, Words
FROM #SourceTable
--WHERE {your criteria}

DECLARE @IDList2 TABLE (Table1ID INT, ID INT, Words VARCHAR(50) )

--(Table1ID is assuemd to be identity)
INSERT #Table1 (ID, Words)
OUTPUT inserted.* INTO @IDList2
SELECT ID, Words
FROM @IDList1


SELECT * FROM @IDList2

Open in new window

0
 
LVL 30

Expert Comment

by:Reza Rad
ID: 34178514
why you do INSERT in control flow task with Foreach ?
why you don't use Data flow task which is dedicated to data transfer?
what is your source and destination?

tell us what exactly you want to do , and we tell you right way
0
 

Author Comment

by:nkapur
ID: 34180084
okay let me elaborate. There is a product table from where product ID needs to be grabbed and this product ID will be inserted in a new product table. Then the new product ID and the old product ID will be used to insert records in a pricing table. Since the pricing table has some values from the new pricing table and some from the old pricing table. So the old product ID and the new product ID should correspond to one another, there is no column common in the old product table and the new product table to relate the new them, so I used a loop which will insert records and match the new product id and the old product id together using new product id and the old product id to grab values from those tables.
But this is not it the pricing table will have records according to the catalog. So if there are six catalog the product will appear in six catalogs and the price for each catalog will vary for the same product. Now there will be another loop within this for each loop in which we will grab each catalog id and insert the new product id , the old product id, the catalog id and insert records in the pricing table. There is a price column in the pricing table in which I have written a query that will calculate the price based on the catalog id.
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 7

Expert Comment

by:rmm2001
ID: 34180112
Did you try putting this using queries yet?
0
 
LVL 30

Expert Comment

by:Reza Rad
ID: 34180265
It's really hard to understand for me!
could you put some screenshots?
0
 
LVL 21

Expert Comment

by:Alpesh Patel
ID: 34185863
May be slow due the system configuration is not sufficient.
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Introduction SQL Server Integration Services can read XML files, that’s known by every BI developer.  (If you didn’t, don’t worry, I’m aiming this article at newcomers as well.) But how far can you go?  When does the XML Source component become …
Ever needed a SQL 2008 Database replicated/mirrored/log shipped on another server but you can't take the downtime inflicted by initial snapshot or disconnect while T-logs are restored or mirror applied? You can use SQL Server Initialize from Backup…
Via a live example combined with referencing Books Online, show some of the information that can be extracted from the Catalog Views in SQL Server.
This videos aims to give the viewer a basic demonstration of how a user can query current session information by using the SYS_CONTEXT function

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now