Solved

SSIS Incremental update help needed

Posted on 2016-10-19
4
37 Views
Last Modified: 2016-11-08
Hi Guys,
In our source database we have got a booking table. I am planning for incremental load. I have a got a staging layer which i want to populate from OLTP source. I have already have an initial load done. My question is that in the booking table which contains 3 millions records at the moment and is growing quickly, bookings can be potentially updated. What is the most efficient way to track the updates and inserts from OLTP and populate my staging layer.
There is DateCreated field and DateUpdated field in the table. The dateupdated filed is null when there is no change in the booking.
 I am not allowed to apply CDC on transactional database.
0
Comment
Question by:shah36
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
4 Comments
 
LVL 13

Accepted Solution

by:
Arifhusen Ansari earned 500 total points
ID: 41851452
As you have mentioned that you have filed like dateupdated in oltp database.

There are three scenario

1) If record is not updated in OLTP so dateupdated  will be null. And yes you can simply insert the data.

It means initial load

2) Record is already there. and also you have some data in dateupdated  in OLTP. You can compare this column from your staging table and OLTP database. it they do not match it means that their is an update and you can update this data in staging.

This code can be used in the scenario where you have some data in dateupdated  field in staging and you got new date in dateupdated filed in OLTP. so your data in staging must be updated and this can be achieved by  comparing both.

You can use below logic.


First use the look up between OLTP and staging table based on some id column between than If data is not available in staging simply insert.

If data is available means either data is updated or not updated.

This you can verify by just verifying dateupdated column in staging and dateupdated  column in OLTP.

You can do so by using Conditional split.

If there is difference means you have to update that.

Hope it will help you.
0
 

Author Comment

by:shah36
ID: 41853866
Hi Ahsan,

Thanks a lot for your response. I didn't have a chance to look into it. Will let you know when i do this.

regards,

Ali
0
 
LVL 13

Expert Comment

by:Arifhusen Ansari
ID: 41862013
Hi,

Did you get a chance to look into this??
0
 

Author Comment

by:shah36
ID: 41868130
Hi Arif,

Sorry been tied up to other things. I might look into it today.

regards
0

Featured Post

Ready to get started with anonymous questions?

It's easy! Check out this step-by-step guide for asking an anonymous question on Experts Exchange.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Everyone has problem when going to load data into Data warehouse (EDW). They all need to confirm that data quality is good but they don't no how to proceed. Microsoft has provided new task within SSIS 2008 called "Data Profiler Task". It solve th…
My client has a dictionary table. They're defining a list of standard naming convention. Now, they are requiring my team to provide us a mechanism how to match new incoming data with existing data in their system.
Monitoring a network: why having a policy is the best policy? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the enormous benefits of having a policy-based approach when monitoring medium and large networks. Software utilized in this v…
Michael from AdRem Software outlines event notifications and Automatic Corrective Actions in network monitoring. Automatic Corrective Actions are scripts, which can automatically run upon discovery of a certain undesirable condition in your network.…
Suggested Courses

631 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question