Solved

What would be the best way to load a data warehouse with star schema from a staging database that has been loaded from a  flat file(delimited text file) IN SSIS 2012

Posted on 2014-03-18
4
185 Views
Last Modified: 2016-02-10
Hi!

My production datawarehouse on one server has a star schema structure, where all the dimension and fact tables get loaded on a monthly basis, from the staging tables on a different server.

The data gets dumped directly into the staging tables, as is, from a file on a shared drive, without any transformations.

The staging tables get truncated and loaded every month. But we cannot do the same with the tables on our data warehouse as the key values get changed in the fact table, if the fact table gets truncated every month as the foreign keys of the fact table are mapped to the primary keys in the dimension tables.

We need to either insert new records or do an upsert or the like, which would make the fact table eventually grow in size. Could you please suggest the best approach to follow, in order to load the above? what would be the best approach to truncate and reload the full table or insert or upsert or the like?

Thanks a million in advance! I greatly appreciate it! Any code/logic is greatly appreciated.

I need a quick help! It's really, really urgent!
0
Comment
Question by:amukta
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 9

Expert Comment

by:Sreedhar Vengala
ID: 39938725
Is your monthly loads or 'Delta' (incremental) loads or eveytime are you getting whole data and deleting and reloading the DWH?

If it is incremental load, once the Staging Area is loaded you can use T-SQL Merge with your destination DWH Table.  And it depends on the type of Dimension table (SCD1, SCD2).

And coming to Fact Table there shouldnt be any deletes ( only in execptional business cases you may require update / delete) as this is the source of historical truth.

For further details on using MERGE look at this http://www.purplefrogsystems.com/blog/2012/01/using-t-sql-merge-to-load-data-warehouse-dimensions/
0
 

Author Comment

by:amukta
ID: 39940083
Currently, all the data from staging is being loaded into the DWH without deleting anything, with a date field  showing the date the data was loaded.

My question is, would it be a good idea to truncate the fact table and load it every month, by using the SCD concept for dimensions or would it be a good idea to add the records to the fact table every month as it's being currently?

Appreciate your help! Thanks!
0
 
LVL 17

Accepted Solution

by:
MIKE earned 500 total points
ID: 40387347
Use an Incremental SNAPSHOT type of process, meaning, update the FACT and DIM tables EACH DAY and on the LAST DAY of the Month, the data will be a MONTH END SNAPSHOT for the month.
0
 

Author Closing Comment

by:amukta
ID: 40556979
Thank you !
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Many to one in one row 2 39
T-SQL to copy a database 37 66
SQL Session Remains Open After ReportViewerControl Form is Closed. 8 33
Sql server Error message 3 16
The Delta outage: 650 cancelled flights, more than 1200 delayed flights, thousands of frustrated customers, tens of millions of dollars in damages – plus untold reputational damage to one of the world’s most trusted airlines. All due to a catastroph…
Ever wondered why sometimes your SQL Server is slow or unresponsive with connections spiking up but by the time you go in, all is well? The following article will show you how to install and configure a SQL job that will send you email alerts includ…
This video shows, step by step, how to configure Oracle Heterogeneous Services via the Generic Gateway Agent in order to make a connection from an Oracle session and access a remote SQL Server database table.
Viewers will learn how to use the UPDATE and DELETE statements to change or remove existing data from their tables. Make a table: Update a specific column given a specific row using the UPDATE statement: Remove a set of values using the DELETE s…

756 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question