Solved

SSIS Foreach loop Slow and crashes Visual Studio when inserting records.

Posted on 2010-11-19
6
1,272 Views
Last Modified: 2013-11-30
I am inserting records by using foreach loop in SSIS.I am using a foreach loop within a for each loop the first foreach loop gets id from a table and based on that id inserts rows in another table, then the other foreach loop uses the id from the old table and the id from the new table to insert records in another table the number of times that id is referenced in another table and grabbing some records from that table as well.
The problem is the number of original records are arouund 20,000 and these records are referenced around 20 times in another table. so 20,000 x 20 = 4,00,000 records needs to be created total in the table in the innermost loop. But when it starts to insert records it is very fast and as the time progresses it becomes slower and slower and eventually it crashes. What should I do? Is there a property in foreach loop that after every n number of inserts it flushes it's memory after making an insert?
0
Comment
Question by:nkapur
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 7

Accepted Solution

by:
rmm2001 earned 250 total points
ID: 34177743
I'm a little lost on all of your loops but I think it's something that could be a bit faster using some sql. You can use the OUTPUT command to send your newly inserted ids to a temp table (or table variable).

Here's what this is.

You have a SourceTable. It contains some words.
You have a Table1. Which you're inserting some records from SourceTable into based on some criteria. You create a temp table to hold the values that you're going to be inserting that come from SourceTable (IDList1). Then you have a temp table to hold the values that came from the insert from Table1. Finally you insert the values from your filtered table (IDList1) into Table1 and output the rows inserted into, yet another temp table (IDList2).

Doing it this way you can keep control of your ids just as well as you would in a foreach container but it'll be much faster. You'll probably need a few more temp tables than what's in here but it's a start. you can put this in an execute sql task and run it in the package.
CREATE TABLE #SourceTable (ID INT IDENTITY, Words VARCHAR(50))
INSERT #SourceTable (Words)
SELECT 'asdf' UNION
SELECT'hjkl' UNION
SELECT '12345'

CREATE TABLE #Table1 (Table1ID INT IDENTITY, ID INT, Words VARCHAR(50))

DECLARE @IDList1 TABLE ( ID INT, Words VARCHAR(50) )
INSERT @IDList1
SELECT ID, Words
FROM #SourceTable
--WHERE {your criteria}

DECLARE @IDList2 TABLE (Table1ID INT, ID INT, Words VARCHAR(50) )

--(Table1ID is assuemd to be identity)
INSERT #Table1 (ID, Words)
OUTPUT inserted.* INTO @IDList2
SELECT ID, Words
FROM @IDList1


SELECT * FROM @IDList2

Open in new window

0
 
LVL 30

Expert Comment

by:Reza Rad
ID: 34178514
why you do INSERT in control flow task with Foreach ?
why you don't use Data flow task which is dedicated to data transfer?
what is your source and destination?

tell us what exactly you want to do , and we tell you right way
0
 

Author Comment

by:nkapur
ID: 34180084
okay let me elaborate. There is a product table from where product ID needs to be grabbed and this product ID will be inserted in a new product table. Then the new product ID and the old product ID will be used to insert records in a pricing table. Since the pricing table has some values from the new pricing table and some from the old pricing table. So the old product ID and the new product ID should correspond to one another, there is no column common in the old product table and the new product table to relate the new them, so I used a loop which will insert records and match the new product id and the old product id together using new product id and the old product id to grab values from those tables.
But this is not it the pricing table will have records according to the catalog. So if there are six catalog the product will appear in six catalogs and the price for each catalog will vary for the same product. Now there will be another loop within this for each loop in which we will grab each catalog id and insert the new product id , the old product id, the catalog id and insert records in the pricing table. There is a price column in the pricing table in which I have written a query that will calculate the price based on the catalog id.
0
Resolve Critical IT Incidents Fast

If your data, services or processes become compromised, your organization can suffer damage in just minutes and how fast you communicate during a major IT incident is everything. Learn how to immediately identify incidents & best practices to resolve them quickly and effectively.

 
LVL 7

Expert Comment

by:rmm2001
ID: 34180112
Did you try putting this using queries yet?
0
 
LVL 30

Expert Comment

by:Reza Rad
ID: 34180265
It's really hard to understand for me!
could you put some screenshots?
0
 
LVL 21

Expert Comment

by:Alpesh Patel
ID: 34185863
May be slow due the system configuration is not sufficient.
0

Featured Post

Free learning courses: Active Directory Deep Dive

Get a firm grasp on your IT environment when you learn Active Directory best practices with Veeam! Watch all, or choose any amount, of this three-part webinar series to improve your skills. From the basics to virtualization and backup, we got you covered.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have a large data set and a SSIS package. How can I load this file in multi threading?
In this article we will learn how to fix  “Cannot install SQL Server 2014 Service Pack 2: Unable to install windows installer msi file” error ?
Via a live example combined with referencing Books Online, show some of the information that can be extracted from the Catalog Views in SQL Server.
Using examples as well as descriptions, and references to Books Online, show the different Recovery Models available in SQL Server and explain, as well as show how full, differential and transaction log backups are performed

696 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question