Freeze portion of datamart

Experts,

We have a data mart that contains data going back 6 to 7 years. Up until now we usually roll off the earliest year. Users have requested not to roll off data any longer as well as going back earlier in time and adding more data. The older data is static. Currently our data mart process is a drop and replace. We use all surrogate keys.

My question is what are the thoughts for the best method for not having to refresh the fact and dimension tables for that older static data so as to keep the build time manageable.

Sorry in advance for the newbie methods we have thought about implementing. One idea was to build  the DM as normal with all the required data to satisfy user requirements. We thought about  finding the max Identity (PK) in each fact and dimension for records that would remain static and also set a flag on those records. Then each month delete the years of data that would need to get refreshed and perhaps reseed the table based on the highest value of the static data. Then just process the newer set of data and appending to the fact and dimensions.

We are probably way off base here. So please excuse the ignorance on our part.

Any insight is appreciated.
LVL 1
jvoconnellAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

lcohanDatabase AnalystCommented:
I believe you should be reading a bit first to decide what method is the best and good advice can be found here:

https://dwbi.org/etl/etl/54-incremental-loading-for-dimension-table
https://dwbi.org/etl/etl/53-methods-of-incremental-loading-in-data-warehouse

and different method like "UPSERT" here: https://www.experts-exchange.com/questions/27401511/Fact-table-incremental-insert-update-best-practice.html

Also depending on the volumes you need to refresh every time, it may be easier just to do incremental population on (very) large tables and you could also use the SQL ROWVERSION data type to identify only rows that changed for a incremental population.

https://www.codeproject.com/Articles/698025/Rowversion-datatype-in-SQL-Server-Track-which-rows
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
jvoconnellAuthor Commented:
This is great. This wasn't the type of question that I would expect to get an answer I could immediately implement. I do want people to know that some research was done prior to my post but our searches didn't return much valuable. We can take what you have provided and know discuss what will work best for us. Thank you so much! You guys are great.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server

From novice to tech pro — start learning today.