aspguy123
asked on
Data Warehouse Sync
Hello all,
I haven't posted on these boards in a long time. I have come to the point where I really need the outlook of other smarter people than I. So basically the situation is this:
We have a DTS package right now that runs every 10 minutes in a production environment. This was developed years ago and definitely is wreaking havoc on our system due to table level locking. I just came onboard with this system so I have limited knowledge of it thus far, but I do know what the DTS package does.
We have two databases-- one for sales and one for our warehouse. These systems were, unfortunately, developed at different times by different groups of people. This is the root of our problem, we now have to maintain duplicate data in two locations. Right now, they use DTS to map the fields and also utilize the ActiveX (VBScript) portion of DTS for some business logic. Since this locks the table, it also locks people out of the systems when they are trying to access the data.
My thoughts on it was that we are abusing DTS. Really, DTS is a data-pump. We need data into our tables, we can do that easily with DTS. However, we are using it for syncing and mapping (with some added business logic). So my question is this:
What would be the best mechanism for keeping the data synched between tables? They are different databases, with different field names, and some fields require additional business logic. Initially, I was thinking of putting in triggers for the INSERT, UPDATE, and DELETE and then it would keep the data synched that way instead of a DTS package. I have pretty vast knowledge of MS SQL, but am no DBA guru, I'm a developer.
Thanks for your help,
aspguy123
I haven't posted on these boards in a long time. I have come to the point where I really need the outlook of other smarter people than I. So basically the situation is this:
We have a DTS package right now that runs every 10 minutes in a production environment. This was developed years ago and definitely is wreaking havoc on our system due to table level locking. I just came onboard with this system so I have limited knowledge of it thus far, but I do know what the DTS package does.
We have two databases-- one for sales and one for our warehouse. These systems were, unfortunately, developed at different times by different groups of people. This is the root of our problem, we now have to maintain duplicate data in two locations. Right now, they use DTS to map the fields and also utilize the ActiveX (VBScript) portion of DTS for some business logic. Since this locks the table, it also locks people out of the systems when they are trying to access the data.
My thoughts on it was that we are abusing DTS. Really, DTS is a data-pump. We need data into our tables, we can do that easily with DTS. However, we are using it for syncing and mapping (with some added business logic). So my question is this:
What would be the best mechanism for keeping the data synched between tables? They are different databases, with different field names, and some fields require additional business logic. Initially, I was thinking of putting in triggers for the INSERT, UPDATE, and DELETE and then it would keep the data synched that way instead of a DTS package. I have pretty vast knowledge of MS SQL, but am no DBA guru, I'm a developer.
Thanks for your help,
aspguy123
You are right, ActiveX in DTS is generally badly written and inefficient and unneccesary.
As you have already alluded, I suggest that you copy data locally to the warehouse database into staging tables, and perform differential processing on the warehouse system using T-SQL.
Adding triggers to your source system is far better than using ActiveX in DTS, but it still adds load to the source server. It does make things simpler for replication/synchronisatio
Basically you need a new synchronisation methodology. Two important questions are:
1. Are you just synchronising 'master' tables (lists of stuff) or are your synchronsing transaction tables (containing dates and amounts).
2. If you are synchronusing transaction tables, can the users 'post back' to prior periods - i.e. can data that is 2 years old change? If you can guarantee that periods are 'closed' and cannot be changed, then that makes it a lot easier to synchronise your tables because you can target recent periods only rather than needlessly processing years of static data.
As you have already alluded, I suggest that you copy data locally to the warehouse database into staging tables, and perform differential processing on the warehouse system using T-SQL.
Adding triggers to your source system is far better than using ActiveX in DTS, but it still adds load to the source server. It does make things simpler for replication/synchronisatio
Basically you need a new synchronisation methodology. Two important questions are:
1. Are you just synchronising 'master' tables (lists of stuff) or are your synchronsing transaction tables (containing dates and amounts).
2. If you are synchronusing transaction tables, can the users 'post back' to prior periods - i.e. can data that is 2 years old change? If you can guarantee that periods are 'closed' and cannot be changed, then that makes it a lot easier to synchronise your tables because you can target recent periods only rather than needlessly processing years of static data.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
@nmcdermaid:
Thanks for your input, its very helpful. Right now our system is a one-way sync-- in that sense I mean we simply use the DTS package to insert records from one-table to the other. We do not run updates or deletes to my knowledge. Its a very simple and primitive "sync" process. Not even really sure if you can call it a sync, more of a data dump from one table to the other.
@Zberteoc:
I was thinking about replication as well, except since we have business logic I figured we could rule that out. Thanks for the info.
Thanks for your input, its very helpful. Right now our system is a one-way sync-- in that sense I mean we simply use the DTS package to insert records from one-table to the other. We do not run updates or deletes to my knowledge. Its a very simple and primitive "sync" process. Not even really sure if you can call it a sync, more of a data dump from one table to the other.
@Zberteoc:
I was thinking about replication as well, except since we have business logic I figured we could rule that out. Thanks for the info.
I was more talking about replacing the inefficient ActiveX code. If it is just a one way dump if data then there should be any need for ActiveX code at all.
ASKER
Its not merely a data-dump, it has business logic. Sorry wrong terminology-- its not a two-way sync, merely a one way (just INSERT).
I would do what I said in my first post. Applications and/or scripts like ActiveX in a DTS are the worst way performance wise to deal with this problem. Replicate the tables that you need to the warehouse server and then deal with business logic on it with tored procedure that will be executed periodically in sql jobs.
I meant "...with stored procedures that..."
ASKER
Thanks for your advice. I was thinking this, just wanted to confirm with experts.
ASKER