Avatar of jvoconnell
jvoconnell
 asked on

Freeze portion of datamart

Experts,

We have a data mart that contains data going back 6 to 7 years. Up until now we usually roll off the earliest year. Users have requested not to roll off data any longer as well as going back earlier in time and adding more data. The older data is static. Currently our data mart process is a drop and replace. We use all surrogate keys.

My question is what are the thoughts for the best method for not having to refresh the fact and dimension tables for that older static data so as to keep the build time manageable.

Sorry in advance for the newbie methods we have thought about implementing. One idea was to build  the DM as normal with all the required data to satisfy user requirements. We thought about  finding the max Identity (PK) in each fact and dimension for records that would remain static and also set a flag on those records. Then each month delete the years of data that would need to get refreshed and perhaps reseed the table based on the highest value of the static data. Then just process the newer set of data and appending to the fact and dimensions.

We are probably way off base here. So please excuse the ignorance on our part.

Any insight is appreciated.
Microsoft SQL Server

Avatar of undefined
Last Comment
jvoconnell

8/22/2022 - Mon
ASKER CERTIFIED SOLUTION
lcohan

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
jvoconnell

ASKER
This is great. This wasn't the type of question that I would expect to get an answer I could immediately implement. I do want people to know that some research was done prior to my post but our searches didn't return much valuable. We can take what you have provided and know discuss what will work best for us. Thank you so much! You guys are great.
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy