Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

SSIS Incremental update help needed

Posted on 2016-10-19
4
Medium Priority
?
43 Views
Last Modified: 2016-11-08
Hi Guys,
In our source database we have got a booking table. I am planning for incremental load. I have a got a staging layer which i want to populate from OLTP source. I have already have an initial load done. My question is that in the booking table which contains 3 millions records at the moment and is growing quickly, bookings can be potentially updated. What is the most efficient way to track the updates and inserts from OLTP and populate my staging layer.
There is DateCreated field and DateUpdated field in the table. The dateupdated filed is null when there is no change in the booking.
 I am not allowed to apply CDC on transactional database.
0
Comment
Question by:shah36
  • 2
  • 2
4 Comments
 
LVL 13

Accepted Solution

by:
Arifhusen Ansari earned 2000 total points
ID: 41851452
As you have mentioned that you have filed like dateupdated in oltp database.

There are three scenario

1) If record is not updated in OLTP so dateupdated  will be null. And yes you can simply insert the data.

It means initial load

2) Record is already there. and also you have some data in dateupdated  in OLTP. You can compare this column from your staging table and OLTP database. it they do not match it means that their is an update and you can update this data in staging.

This code can be used in the scenario where you have some data in dateupdated  field in staging and you got new date in dateupdated filed in OLTP. so your data in staging must be updated and this can be achieved by  comparing both.

You can use below logic.


First use the look up between OLTP and staging table based on some id column between than If data is not available in staging simply insert.

If data is available means either data is updated or not updated.

This you can verify by just verifying dateupdated column in staging and dateupdated  column in OLTP.

You can do so by using Conditional split.

If there is difference means you have to update that.

Hope it will help you.
0
 

Author Comment

by:shah36
ID: 41853866
Hi Ahsan,

Thanks a lot for your response. I didn't have a chance to look into it. Will let you know when i do this.

regards,

Ali
0
 
LVL 13

Expert Comment

by:Arifhusen Ansari
ID: 41862013
Hi,

Did you get a chance to look into this??
0
 

Author Comment

by:shah36
ID: 41868130
Hi Arif,

Sorry been tied up to other things. I might look into it today.

regards
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

My client has a dictionary table. They're defining a list of standard naming convention. Now, they are requiring my team to provide us a mechanism how to match new incoming data with existing data in their system.
Here's a requirements document template for an integration project (also known as Extract-Transform-Load or ETL) based on my development experience as an SQL Server Information Services (SSIS) developer over the years.
Integration Management Part 2
Are you ready to place your question in front of subject-matter experts for more timely responses? With the release of Priority Question, Premium Members, Team Accounts and Qualified Experts can now identify the emergent level of their issue, signal…

885 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question