• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 944
  • Last Modified:

Data Warehouse Sync

Hello all,

I haven't posted on these boards in a long time.  I have come to the point where I really need the outlook of other smarter people than I.  So basically the situation is this:

We have a DTS package right now that runs every 10 minutes in a production environment.  This was developed years ago and definitely is wreaking havoc on our system due to table level locking.  I just came onboard with this system so I have limited knowledge of it thus far, but I do know what the DTS package does.

We have two databases-- one for sales and one for our warehouse.  These systems were, unfortunately, developed at different times by different groups of people.  This is the root of our problem, we now have to maintain duplicate data in two locations.  Right now, they use DTS to map the fields and also utilize the ActiveX (VBScript) portion of DTS for some business logic.  Since this locks the table, it also locks people out of the systems when they are trying to access the data.

My thoughts on it was that we are abusing DTS.  Really, DTS is a data-pump.  We need data into our tables, we can do that easily with DTS.  However, we are using it for syncing and mapping (with some added business logic).  So my question is this:

What would be the best mechanism for keeping the data synched between tables?  They are different databases, with different field names, and some fields require additional business logic.  Initially, I was thinking of putting in triggers for the INSERT, UPDATE, and DELETE and then it would keep the data synched that way instead of a DTS package.  I have pretty vast knowledge of MS SQL, but am no DBA guru, I'm a developer.

Thanks for your help,
aspguy123
0
aspguy123
Asked:
aspguy123
  • 4
  • 3
  • 2
1 Solution
 
aspguy123Author Commented:
I have increased the points, I figured it would help increase the chances of a quick response.
0
 
nmcdermaidCommented:
You are right, ActiveX in DTS is generally badly written and inefficient and unneccesary.
As you have already alluded, I suggest that you copy data locally to the warehouse database into staging tables, and perform differential processing on the warehouse system using T-SQL.
Adding triggers to your source system is far better than using ActiveX in DTS, but it still adds load to the source server. It does make things simpler for replication/synchronisation though.
Basically you need a new synchronisation methodology. Two important questions are:
1. Are you just synchronising 'master' tables (lists of stuff) or are your synchronsing transaction tables (containing dates and amounts).
2. If you are synchronusing transaction tables, can the users 'post back' to prior periods - i.e. can data that is 2 years old change? If you can guarantee that periods are 'closed' and cannot be changed, then that makes it a lot easier to synchronise your tables because you can target recent periods only rather than needlessly processing years of static data.
0
 
ZberteocCommented:
Normally for a synch scenario between transactional database and warehouse/reporting database the best solution would be to use replication. In a replication you can handpick tables you need to replicate and add filters to the replication if you don't just simply copy tables from one server to the other.

The problem in your case is that you are using that business login within the DTS package that would not be possible to apply using replication. A solution to that would be to just replicate the tables you need from prod to warehouse server and then build some stored procedure that will take care of the business logic. Those sps could be scheduled in jobs on the warehouse server to make dthe modifications that are needed.

The replication process is not trivial but it is not rocket science either and once understood works very nice, fast and with no much impact on the resources. It will imply a learning process.

Here is an overview article about replication:
http://www.devarticles.com/c/a/SQL-Server/Replication-SQL-Server-2000--Part-1/

and here is a step by step article that can guide you:
http://www.databasejournal.com/features/mssql/article.php/1438201/Setting-Up-Transactional-Replication-A-Step-by-step-Guide.htm

You can find a lot of info about this on the net.
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
aspguy123Author Commented:
@nmcdermaid:

Thanks for your input, its very helpful.  Right now our system is a one-way sync-- in that sense I mean we simply use the DTS package to insert records from one-table to the other.  We do not run updates or deletes to my knowledge.  Its a very simple and primitive "sync" process.  Not even really sure if you can call it a sync, more of a data dump from one table to the other.

@Zberteoc:

I was thinking about replication as well, except since we have business logic I figured we could rule that out.  Thanks for the info.
0
 
nmcdermaidCommented:
I was more talking about replacing the inefficient ActiveX code. If it is just a one way dump if data then there should be any need for ActiveX code at all.
0
 
aspguy123Author Commented:
Its not merely a data-dump, it has business logic.  Sorry wrong terminology-- its not a two-way sync, merely a one way (just INSERT).
0
 
ZberteocCommented:
I would do what I said in my first post. Applications and/or scripts like ActiveX in a DTS are the worst way performance wise to deal with this problem. Replicate the tables that you need to the warehouse server and then deal with business logic on it with tored procedure that will be executed periodically in sql jobs.
0
 
ZberteocCommented:
I meant "...with stored procedures that..."
0
 
aspguy123Author Commented:
Thanks for your advice.  I was thinking this, just wanted to confirm with experts.
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

  • 4
  • 3
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now