We help IT Professionals succeed at work.

BI Archive

123 Views
Last Modified: 2014-12-19
1.5TB MSSQL2k8r2 with 2years of sales data.  

We want to do a one time back up and restore of the entire database to a (read only) BI database. Then maintain only 6 months of data in our production environment, with data older than 6 months being archived in a (read only) BI database.

How would you recommend to setup the daily differential updates for new entries from Production db to BI db, only backing up the Inserts and updates and not the delete statements. Then delete the copied data from production db.
Comment
Watch Question

Phillip BurtonDirector, Practice Manager and Computing Consultant
Awarded 2014
Top Expert 2014

Commented:
Have you considered partition switching?

Maybe see this previous answer: https://www.experts-exchange.com/Database/MS-SQL-Server/SQL-Server-2005/Q_28297008.html
Phillip BurtonDirector, Practice Manager and Computing Consultant
Awarded 2014
Top Expert 2014

Commented:
and this one: https://www.experts-exchange.com/Database/MS-SQL-Server/Q_28412426.html

Basically, you use the ALTER TABLE SWITCH statement.

Or am I missing the point of the question?
Senior DBA
CERTIFIED EXPERT
Most Valuable Expert 2018
Distinguished Expert 2019
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
ScottPletcher- The Software vendor setup the database 2years ago and no one has touched it since. About a year ago we hired a DBA and his manager told him not to touch the database (you break it, you bought it mentality).

Phillip Burton - Altering the production tables is not an option.
Scott PletcherSenior DBA
CERTIFIED EXPERT
Most Valuable Expert 2018
Distinguished Expert 2019

Commented:
OK, that provides some background, but it doesn't answer these qs:

Is your existing sales table clustered on sales date?

If not, can you create a nonclustered index on sales date?

If you can't even add a nonclustered index, I don't know how you're reasonably quickly going to pull each day's data to archive it as it reaches the threshold date, 6 mos or whatever it is.
Phillip BurtonDirector, Practice Manager and Computing Consultant
Awarded 2014
Top Expert 2014

Commented:
At least you can use the partition switching to import the new data into the (read only) BI database, which will speed to importing up.

As for the export, if you have to use kid gloves and go carefully, so be it - Scott's answer for that bit is a good one.
Vitor MontalvãoIT Engineer
CERTIFIED EXPERT
Distinguished Expert 2017

Commented:
we hired a DBA and his manager told him not to touch the database (you break it, you bought it mentality).
The worst nightmare for a DBA, but I've been there and did that already. He can't change the schema but he can touch the data, right?
I personally like the partition idea but if you can't do that then I recommend to implement a Snapshot Replication. Mind that this solution is not good for very large databases since kills the performance. Do you want to backup all data or some tables only?

Author

Commented:
Vitor Montalvão - The business can't decide what data is important at this point so they want the entire Database backed up.
Vitor MontalvãoIT Engineer
CERTIFIED EXPERT
Distinguished Expert 2017

Commented:
Well, the business can't decide what data is important and a DBA can't touch the database. So, you are running out of options here.
How big is the database?
Scott PletcherSenior DBA
CERTIFIED EXPERT
Most Valuable Expert 2018
Distinguished Expert 2019

Commented:
>> At least you can use the partition switching to import the new data into the (read only) BI database, which will speed to importing up. <<

I don't see how that would even be possible.  Partition switches must be within the same filegroup.  How, then, could you possibly do partition switching between different dbs??
Phillip BurtonDirector, Practice Manager and Computing Consultant
Awarded 2014
Top Expert 2014

Commented:
Please correct me if I'm wrong, but:

1. In Source DB, create another table (Table 2) in Source DB to switch into,
2. Partition Switch into Table 2 in Source DB.
3. Use SSIS to export that into the Target DB into Table 2.
4. Partition Switch into Table in Target DB.

Obviously, not in this case if he can't touch the Source DB.
Scott PletcherSenior DBA
CERTIFIED EXPERT
Most Valuable Expert 2018
Distinguished Expert 2019
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
Phillip BurtonDirector, Practice Manager and Computing Consultant
Awarded 2014
Top Expert 2014

Commented:
Because if it's a partition per day, then it may make queries quicker?
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.