Solved

SSIS Incremental update help needed

Posted on 2016-10-19
4
22 Views
Last Modified: 2016-11-08
Hi Guys,
In our source database we have got a booking table. I am planning for incremental load. I have a got a staging layer which i want to populate from OLTP source. I have already have an initial load done. My question is that in the booking table which contains 3 millions records at the moment and is growing quickly, bookings can be potentially updated. What is the most efficient way to track the updates and inserts from OLTP and populate my staging layer.
There is DateCreated field and DateUpdated field in the table. The dateupdated filed is null when there is no change in the booking.
 I am not allowed to apply CDC on transactional database.
0
Comment
Question by:shah36
  • 2
  • 2
4 Comments
 
LVL 12

Accepted Solution

by:
Arifhusen Ansari earned 500 total points
ID: 41851452
As you have mentioned that you have filed like dateupdated in oltp database.

There are three scenario

1) If record is not updated in OLTP so dateupdated  will be null. And yes you can simply insert the data.

It means initial load

2) Record is already there. and also you have some data in dateupdated  in OLTP. You can compare this column from your staging table and OLTP database. it they do not match it means that their is an update and you can update this data in staging.

This code can be used in the scenario where you have some data in dateupdated  field in staging and you got new date in dateupdated filed in OLTP. so your data in staging must be updated and this can be achieved by  comparing both.

You can use below logic.


First use the look up between OLTP and staging table based on some id column between than If data is not available in staging simply insert.

If data is available means either data is updated or not updated.

This you can verify by just verifying dateupdated column in staging and dateupdated  column in OLTP.

You can do so by using Conditional split.

If there is difference means you have to update that.

Hope it will help you.
0
 

Author Comment

by:shah36
ID: 41853866
Hi Ahsan,

Thanks a lot for your response. I didn't have a chance to look into it. Will let you know when i do this.

regards,

Ali
0
 
LVL 12

Expert Comment

by:Arifhusen Ansari
ID: 41862013
Hi,

Did you get a chance to look into this??
0
 

Author Comment

by:shah36
ID: 41868130
Hi Arif,

Sorry been tied up to other things. I might look into it today.

regards
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

In a previous article I've shown you how to import data from an Excel sheet using the OPENROWSET() function (http://www.experts-exchange.com/A_3025.html).  And I concluded by stating that it's not the best option when automating your data import. …
Lessons learned during ten years of interviewing for SQL Server Integration Services (SSIS) and other Extract-Transform-Load (ETL) contract roles and two years of staff manager interviewing contractors.
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…
This video shows how to remove a single email address from the Outlook 2010 Auto Suggestion memory. NOTE: For Outlook 2016 and 2013 perform the exact same steps. Open a new email: Click the New email button in Outlook. Start typing the address: …

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now