Solved

SSIS package

Posted on 2006-11-14
4
512 Views
Last Modified: 2008-03-06
Hello Expert:

Since SSIS is one of SQL Server 2005's feature, I post my question here. I am sorry if it is not fitting in this topic area.

I have an assignment for creating a SSIS package which loading around 20 million rows data into a fact table in the data mart. My question is: what is the better solution when the package failed duing the loading data period (let say, 5 million rows data has been loaded in the fact table when the SSIS package(ETL tool in SQL Server 2005) failed. Should I have to truncate the fact table then reload it? I think this is not a better solution?

Any inputs will be appreciated!
0
Comment
Question by:bluesky6045
  • 2
  • 2
4 Comments
 
LVL 11

Expert Comment

by:regbes
ID: 17944849
Hi bluesky6045,

What i do is in my psckage is

Truncate target table
Drop indexes on target table if they exsist
load data into target table
create indexes on target table if they dont exsist

this way whenever you run the package you dont care what is in the target table or wether the previous execution passed or failed

HTH

R.
0
 

Author Comment

by:bluesky6045
ID: 17947596
Hi regbes,

Thank you for your response. Yes, the one you provided  is one of good solutions for this issue. But if I don’t want to truncate the fact table since the window for loading fact table data is just limited (2-3 hours). Is there still a way to load the remaining data to the fact table? Thank you!
0
 
LVL 11

Accepted Solution

by:
regbes earned 200 total points
ID: 17948965
3 other ways

1 to use "stageing" tables
what this means is that you do not load the data directly into you fact table but into a stageing table only once the data you need is fully loaded into the stageing table do you transfer it from the stageing table into your fact table, if your loads are done on a logical breal i.e. date (month) you can use these stageing tables a additional horizontal partitions.

2. make our load process intelegent i.e. check your max id in the fact table then only load data with id's with an id higher than that from your source with this method you need to be sure the data is loaded sequentialy i.e. dont load id 10 before 5 but you gain he benifit of having a robust process that can run almost without careing what happend to the previous load

the third option is touse a hybred of both  
0
 

Author Comment

by:bluesky6045
ID: 17949062
Excellent! I really like the second solution. Thank you so much! I know my question is more related to data warehouse. But we dont have DW channel in EE community. I am quite lucky because I can get you to answer this question. Can I still post any DW (SSIS) related question in SQL Server channel?
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Suggested Solutions

Introduction SQL Server Integration Services can read XML files, that’s known by every BI developer.  (If you didn’t, don’t worry, I’m aiming this article at newcomers as well.) But how far can you go?  When does the XML Source component become …
JSON is being used more and more, besides XML, and you surely wanted to parse the data out into SQL instead of doing it in some Javascript. The below function in SQL Server can do the job for you, returning a quick table with the parsed data.
Via a live example, show how to backup a database, simulate a failure backup the tail of the database transaction log and perform the restore.
Via a live example, show how to shrink a transaction log file down to a reasonable size.

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now