Solved

SSIS Incremental update help needed

Posted on 2016-10-19
4
24 Views
Last Modified: 2016-11-08
Hi Guys,
In our source database we have got a booking table. I am planning for incremental load. I have a got a staging layer which i want to populate from OLTP source. I have already have an initial load done. My question is that in the booking table which contains 3 millions records at the moment and is growing quickly, bookings can be potentially updated. What is the most efficient way to track the updates and inserts from OLTP and populate my staging layer.
There is DateCreated field and DateUpdated field in the table. The dateupdated filed is null when there is no change in the booking.
 I am not allowed to apply CDC on transactional database.
0
Comment
Question by:shah36
  • 2
  • 2
4 Comments
 
LVL 12

Accepted Solution

by:
Arifhusen Ansari earned 500 total points
ID: 41851452
As you have mentioned that you have filed like dateupdated in oltp database.

There are three scenario

1) If record is not updated in OLTP so dateupdated  will be null. And yes you can simply insert the data.

It means initial load

2) Record is already there. and also you have some data in dateupdated  in OLTP. You can compare this column from your staging table and OLTP database. it they do not match it means that their is an update and you can update this data in staging.

This code can be used in the scenario where you have some data in dateupdated  field in staging and you got new date in dateupdated filed in OLTP. so your data in staging must be updated and this can be achieved by  comparing both.

You can use below logic.


First use the look up between OLTP and staging table based on some id column between than If data is not available in staging simply insert.

If data is available means either data is updated or not updated.

This you can verify by just verifying dateupdated column in staging and dateupdated  column in OLTP.

You can do so by using Conditional split.

If there is difference means you have to update that.

Hope it will help you.
0
 

Author Comment

by:shah36
ID: 41853866
Hi Ahsan,

Thanks a lot for your response. I didn't have a chance to look into it. Will let you know when i do this.

regards,

Ali
0
 
LVL 12

Expert Comment

by:Arifhusen Ansari
ID: 41862013
Hi,

Did you get a chance to look into this??
0
 

Author Comment

by:shah36
ID: 41868130
Hi Arif,

Sorry been tied up to other things. I might look into it today.

regards
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Performance is the key factor for any successful data integration project, knowing the type of transformation that you’re using is the first step on optimizing the SSIS flow performance, by utilizing the correct transformation or the design alternat…
In couple weeks ago, I encountered an extremely difficult problem while deploying 2008 SSIS packages to a new environment (SQL Server 2014 standard).  My scenario is: We have one C# application that is calling 2008R2 SSIS packages to load text fi…
This is used to tweak the memory usage for your computer, it is used for servers more so than workstations but just be careful editing registry settings as it may cause irreversible results. I hold no responsibility for anything you do to the regist…
Internet Business Fax to Email Made Easy - With  eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, f…

896 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now