• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 204
  • Last Modified:

Data archiving and replicating database

Hi,

I need advise with regards  to archiving old data from our prod database
since db size has increased
All tables in the database do not have a date column but they do relate to the tables
which have date columns
Users might require to access this old database for reporting or data analysis
once in a while, so it needs to exist on a remote server for them to access

Currently this prod database is getting replicated to a  reporting server.
So now once the data is archived we still would require to replicate both the archived and
the current prod data to the reporting server on a single db

Please suggest how this can be achieved??

Thanks
0
SP_2018 .
Asked:
SP_2018 .
  • 2
1 Solution
 
Ryan McCauleyData and Analytics ManagerCommented:
For starters, does historic data change at all? If so, how far back? I ask because, if historic data is static, you don't need to worry about ongoing maintenance - once it's been replicated to the reporting server, you no longer need to keep it syncronized and you can delete it from the production server (or move it to the archive server you're referencing). Also, are you actually using SQL Replication, or are you using "Replication" to mean some kind of ongoing ETL process that moves older data to your reporting server?

If you're using SQL Replication, then both copies of the database (production and reporting) need to be exactly the same, data-wise. If they're the same, then why do you need to archive server? It seems like you'd just be maintaining a second copy of the data that's already on your reporting server.

If you're using some kind of ETL process, then you're free to delete data from your source server without worrying about the impact on the reporting server, so you can go forward with the Archive server you're talking about. You can set up a second ETL process (or extend the existing one) to also copy historic data to the archive server, and then delete it from your source server. As long as the records your working with have a date you can attach to them (through a join, it sounds like), then you can trim old data from your production server and move it to your archive.

Since the data has already been moved from production to the reporting server previously, you don't need to worry about syncing up anything between the archive and reporting servers.
0
 
SP_2018 .IT ConsultantAuthor Commented:
Hi,

The historic data would not change but we need to archive this old data to archive server
as well replicate the old and the current data to the reporting server with some kind of
ETL strategy. Currently we are using transactional repication to replicate the data from
Prod server to the reporting server.
Do you think if this could be achieved using  merge replication between the production data and the archived data.

Also not all the tables on the prod server has date column in it , but the ones which do not
have date column are referencing the tables which do have date column, so can
we still archive the data from all tables.

Is their some kind of reference you could provide which explains such a solution.

Thanks
0
 
Ryan McCauleyData and Analytics ManagerCommented:
If you're doing custom ETL, then you can filter it however you want - if you have a date in your table, that's great, but you can filter by a date in another table that you can join to if you want to do that.

The problem with using standard transactional replication to get your data from the production to the reporting server is that, when you move the rows to the archive server and delete them from production, you'll also replicate that delete to the reporting server and remove them that server too. However, you can configure the subscription to not push delete statements to the subscribers:

http://www.sqlservercentral.com/articles/Replication/3202/

If you do that, then you're safe to delete data on your source database and not have it disappear from your reporting server (assuming deletes aren't a normal part of application activity that you actually require to be replicated for proper activity).
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now