[x]
Posted via EE Mobile

Search, ask, and monitor your questions on the go with EE Mobile. Visit Experts Exchange from your mobile device and never be out of touch again.

Question
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

8.4

SSIS upsert in incremental data warehouse load

Asked by jflanner in SQL Server 2005, MS SQL DTS, SQL Server 2008

Tags: SSIS SQL Server

I have what I believe shoud be a typical ETL problem.  I am incrementally loading a data warehouse.  My load data is approximately 100K rows and is an SSIS in memory object at the point I want to load it.  My base (Fact) table is 45,000K rows.

My problem is my load data contains both inserts and updates - so I need to upsert it.  My question is what is the best way to approach.

I have tried an SSIS lookup on the base table - but I observe SSIS cachiing the data up until it reaches 12 million rows - at which point the package crashes.

I have considered an SSIS merge - but the documentation I am reading tells me this is to merge in memory objects, not one in memory object into an existing table.  (And per the above - 45 million rows in memory does not appear to be a good idea.)

I have considered defining a real load table, and upserting using T-SQL - but that violates the premise "you will never have to write a line of code with SSIS."  (Not that Microsoft has never lied to us before.)

If anyone has some solid guidence on how to approach this, I would appreciate it.  Please be as specific as possible.

Thanks in advance.  
[+][-]11/05/09 11:25 AM, ID: 25752952Accepted Solution

View this solution now by starting your 30-day free trial. Setting up your free trial is quick, easy, and secure. We will return you to this solution, unlocked, when you're done.

About this solution

Zones: SQL Server 2005, MS SQL DTS, SQL Server 2008
Tags: SSIS SQL Server
Sign Up Now!
Solution Provided By: CGLuttrell
Participating Experts: 1
Solution Grade: B
 
[+][-]11/05/09 11:27 AM, ID: 25752969Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
 
Loading Advertisement...
20091111-EE-VQP-92 - Hierarchy / EE_QW_3_20080625