amukta
asked on
What would be the best way to load a data warehouse with star schema from a staging database that has been loaded from a flat file(delimited text file) IN SSIS 2012
Hi!
My production datawarehouse on one server has a star schema structure, where all the dimension and fact tables get loaded on a monthly basis, from the staging tables on a different server.
The data gets dumped directly into the staging tables, as is, from a file on a shared drive, without any transformations.
The staging tables get truncated and loaded every month. But we cannot do the same with the tables on our data warehouse as the key values get changed in the fact table, if the fact table gets truncated every month as the foreign keys of the fact table are mapped to the primary keys in the dimension tables.
We need to either insert new records or do an upsert or the like, which would make the fact table eventually grow in size. Could you please suggest the best approach to follow, in order to load the above? what would be the best approach to truncate and reload the full table or insert or upsert or the like?
Thanks a million in advance! I greatly appreciate it! Any code/logic is greatly appreciated.
I need a quick help! It's really, really urgent!
My production datawarehouse on one server has a star schema structure, where all the dimension and fact tables get loaded on a monthly basis, from the staging tables on a different server.
The data gets dumped directly into the staging tables, as is, from a file on a shared drive, without any transformations.
The staging tables get truncated and loaded every month. But we cannot do the same with the tables on our data warehouse as the key values get changed in the fact table, if the fact table gets truncated every month as the foreign keys of the fact table are mapped to the primary keys in the dimension tables.
We need to either insert new records or do an upsert or the like, which would make the fact table eventually grow in size. Could you please suggest the best approach to follow, in order to load the above? what would be the best approach to truncate and reload the full table or insert or upsert or the like?
Thanks a million in advance! I greatly appreciate it! Any code/logic is greatly appreciated.
I need a quick help! It's really, really urgent!
ASKER
Currently, all the data from staging is being loaded into the DWH without deleting anything, with a date field showing the date the data was loaded.
My question is, would it be a good idea to truncate the fact table and load it every month, by using the SCD concept for dimensions or would it be a good idea to add the records to the fact table every month as it's being currently?
Appreciate your help! Thanks!
My question is, would it be a good idea to truncate the fact table and load it every month, by using the SCD concept for dimensions or would it be a good idea to add the records to the fact table every month as it's being currently?
Appreciate your help! Thanks!
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thank you !
If it is incremental load, once the Staging Area is loaded you can use T-SQL Merge with your destination DWH Table. And it depends on the type of Dimension table (SCD1, SCD2).
And coming to Fact Table there shouldnt be any deletes ( only in execptional business cases you may require update / delete) as this is the source of historical truth.
For further details on using MERGE look at this http://www.purplefrogsystems.com/blog/2012/01/using-t-sql-merge-to-load-data-warehouse-dimensions/