an incremental load on a dimension when there is no modified date on source table

Does anyone have any ideas how to do an incremental load on a dimension when there is no modified date on source table and without loading all source data again?
eluceroAsked:
Who is Participating?
 
Arifhusen AnsariBusiness Intelligence Developer and AnalystCommented:
Hello,
You can achieve this. Let's say your dimension table name is "dbo.Employee" and column names are
1) EmpID -- Primary Key
2) Emplyee Name - Very less chance to be updated.
3) Designation --  High chances to be updated.

If you have ssis package to load your dimension. You can follow below steps.

1) Get the data from Source in Data flow task.
2) Use look up control and based on Primary key column "EmpID" lookup Dimension and get column that you think that can be updated. Like "Designation".
3) Add Conditional Split control and compare new value for source to old value you got from look up transformation.
4) If you find changes means there is an update. Send these rows to other flow and update the table, using primary key and new value.

You can refer below link.
https://www.experts-exchange.com/questions/28631371/SSIS-How-to-Create-Package-to-INSERT-and-or-UPDATE-a-table's-Columns-if-they-CHANGE.html


If you don't use SSIS package and loading query using simple sql query.

You can check merge statement in the SQL query.

You can do check for new data or existing with update or existing without update in single statement.

Try Below query it will give you idea.

CREATE TABLE #Employee
(

	EmpID INT
	,FullName VARCHAR(100)
	,Designation VARCHAR(100)

)

INSERT INTO #Employee
VALUES(1,'Arif','Developer')
,(2,'Axar','Jr.Developer')
,(3,'Amit','Project Manager')


SELECT * FROM #Employee

-- Sample New Data

MERGE #Employee Target
USING ( VALUES (1,'Arif','Jr.Developer'),(4,'New Record','Tester'),(3,'Amit','Project Manager')) AS Source(EmpID,FullName,Designation)
on Target.EmpId=source.EmpID

WHEN MATCHED AND Target.Designation<>Source.Designation THEN 

UPDATE SET Target.Designation=Source.Designation

WHEN NOT MATCHED BY TARGET THEN 

INSERT (EmpID,FullName,Designation)
VALUES (Source.EmpID,Source.FullName,Source.Designation);

SELECT * FROM #Employee

DROP TABLE #Employee

Open in new window


Screenshot shows Update and new entry.

Untitled.png
0
 
eluceroAuthor Commented:
Thank you
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.