Data Warehouse Sync

Hello all,

I haven't posted on these boards in a long time.  I have come to the point where I really need the outlook of other smarter people than I.  So basically the situation is this:

We have a DTS package right now that runs every 10 minutes in a production environment.  This was developed years ago and definitely is wreaking havoc on our system due to table level locking.  I just came onboard with this system so I have limited knowledge of it thus far, but I do know what the DTS package does.

We have two databases-- one for sales and one for our warehouse.  These systems were, unfortunately, developed at different times by different groups of people.  This is the root of our problem, we now have to maintain duplicate data in two locations.  Right now, they use DTS to map the fields and also utilize the ActiveX (VBScript) portion of DTS for some business logic.  Since this locks the table, it also locks people out of the systems when they are trying to access the data.

My thoughts on it was that we are abusing DTS.  Really, DTS is a data-pump.  We need data into our tables, we can do that easily with DTS.  However, we are using it for syncing and mapping (with some added business logic).  So my question is this:

What would be the best mechanism for keeping the data synched between tables?  They are different databases, with different field names, and some fields require additional business logic.  Initially, I was thinking of putting in triggers for the INSERT, UPDATE, and DELETE and then it would keep the data synched that way instead of a DTS package.  I have pretty vast knowledge of MS SQL, but am no DBA guru, I'm a developer.

Thanks for your help,
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

aspguy123Author Commented:
I have increased the points, I figured it would help increase the chances of a quick response.
You are right, ActiveX in DTS is generally badly written and inefficient and unneccesary.
As you have already alluded, I suggest that you copy data locally to the warehouse database into staging tables, and perform differential processing on the warehouse system using T-SQL.
Adding triggers to your source system is far better than using ActiveX in DTS, but it still adds load to the source server. It does make things simpler for replication/synchronisation though.
Basically you need a new synchronisation methodology. Two important questions are:
1. Are you just synchronising 'master' tables (lists of stuff) or are your synchronsing transaction tables (containing dates and amounts).
2. If you are synchronusing transaction tables, can the users 'post back' to prior periods - i.e. can data that is 2 years old change? If you can guarantee that periods are 'closed' and cannot be changed, then that makes it a lot easier to synchronise your tables because you can target recent periods only rather than needlessly processing years of static data.
Normally for a synch scenario between transactional database and warehouse/reporting database the best solution would be to use replication. In a replication you can handpick tables you need to replicate and add filters to the replication if you don't just simply copy tables from one server to the other.

The problem in your case is that you are using that business login within the DTS package that would not be possible to apply using replication. A solution to that would be to just replicate the tables you need from prod to warehouse server and then build some stored procedure that will take care of the business logic. Those sps could be scheduled in jobs on the warehouse server to make dthe modifications that are needed.

The replication process is not trivial but it is not rocket science either and once understood works very nice, fast and with no much impact on the resources. It will imply a learning process.

Here is an overview article about replication:

and here is a step by step article that can guide you:

You can find a lot of info about this on the net.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

aspguy123Author Commented:

Thanks for your input, its very helpful.  Right now our system is a one-way sync-- in that sense I mean we simply use the DTS package to insert records from one-table to the other.  We do not run updates or deletes to my knowledge.  Its a very simple and primitive "sync" process.  Not even really sure if you can call it a sync, more of a data dump from one table to the other.


I was thinking about replication as well, except since we have business logic I figured we could rule that out.  Thanks for the info.
I was more talking about replacing the inefficient ActiveX code. If it is just a one way dump if data then there should be any need for ActiveX code at all.
aspguy123Author Commented:
Its not merely a data-dump, it has business logic.  Sorry wrong terminology-- its not a two-way sync, merely a one way (just INSERT).
I would do what I said in my first post. Applications and/or scripts like ActiveX in a DTS are the worst way performance wise to deal with this problem. Replicate the tables that you need to the warehouse server and then deal with business logic on it with tored procedure that will be executed periodically in sql jobs.
I meant "...with stored procedures that..."
aspguy123Author Commented:
Thanks for your advice.  I was thinking this, just wanted to confirm with experts.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server

From novice to tech pro — start learning today.