Solved

SSIS Incremental update help needed

Posted on 2016-10-19
4
28 Views
Last Modified: 2016-11-08
Hi Guys,
In our source database we have got a booking table. I am planning for incremental load. I have a got a staging layer which i want to populate from OLTP source. I have already have an initial load done. My question is that in the booking table which contains 3 millions records at the moment and is growing quickly, bookings can be potentially updated. What is the most efficient way to track the updates and inserts from OLTP and populate my staging layer.
There is DateCreated field and DateUpdated field in the table. The dateupdated filed is null when there is no change in the booking.
 I am not allowed to apply CDC on transactional database.
0
Comment
Question by:shah36
  • 2
  • 2
4 Comments
 
LVL 12

Accepted Solution

by:
Arifhusen Ansari earned 500 total points
ID: 41851452
As you have mentioned that you have filed like dateupdated in oltp database.

There are three scenario

1) If record is not updated in OLTP so dateupdated  will be null. And yes you can simply insert the data.

It means initial load

2) Record is already there. and also you have some data in dateupdated  in OLTP. You can compare this column from your staging table and OLTP database. it they do not match it means that their is an update and you can update this data in staging.

This code can be used in the scenario where you have some data in dateupdated  field in staging and you got new date in dateupdated filed in OLTP. so your data in staging must be updated and this can be achieved by  comparing both.

You can use below logic.


First use the look up between OLTP and staging table based on some id column between than If data is not available in staging simply insert.

If data is available means either data is updated or not updated.

This you can verify by just verifying dateupdated column in staging and dateupdated  column in OLTP.

You can do so by using Conditional split.

If there is difference means you have to update that.

Hope it will help you.
0
 

Author Comment

by:shah36
ID: 41853866
Hi Ahsan,

Thanks a lot for your response. I didn't have a chance to look into it. Will let you know when i do this.

regards,

Ali
0
 
LVL 12

Expert Comment

by:Arifhusen Ansari
ID: 41862013
Hi,

Did you get a chance to look into this??
0
 

Author Comment

by:shah36
ID: 41868130
Hi Arif,

Sorry been tied up to other things. I might look into it today.

regards
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In a previous article I've shown you how to import data from an Excel sheet using the OPENROWSET() function (http://www.experts-exchange.com/A_3025.html).  And I concluded by stating that it's not the best option when automating your data import. …
Here's a requirements document template for an integration project (also known as Extract-Transform-Load or ETL) based on my development experience as an SQL Server Information Services (SSIS) developer over the years.
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…

803 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question