an incremental load on a dimension when there is no modified date on source table

Does anyone have any ideas how to do an incremental load on a dimension when there is no modified date on source table and without loading all source data again?
eluceroAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Arifhusen AnsariBusiness Intelligence Developer and AnalystCommented:
Hello,
You can achieve this. Let's say your dimension table name is "dbo.Employee" and column names are
1) EmpID -- Primary Key
2) Emplyee Name - Very less chance to be updated.
3) Designation --  High chances to be updated.

If you have ssis package to load your dimension. You can follow below steps.

1) Get the data from Source in Data flow task.
2) Use look up control and based on Primary key column "EmpID" lookup Dimension and get column that you think that can be updated. Like "Designation".
3) Add Conditional Split control and compare new value for source to old value you got from look up transformation.
4) If you find changes means there is an update. Send these rows to other flow and update the table, using primary key and new value.

You can refer below link.
https://www.experts-exchange.com/questions/28631371/SSIS-How-to-Create-Package-to-INSERT-and-or-UPDATE-a-table's-Columns-if-they-CHANGE.html


If you don't use SSIS package and loading query using simple sql query.

You can check merge statement in the SQL query.

You can do check for new data or existing with update or existing without update in single statement.

Try Below query it will give you idea.

CREATE TABLE #Employee
(

	EmpID INT
	,FullName VARCHAR(100)
	,Designation VARCHAR(100)

)

INSERT INTO #Employee
VALUES(1,'Arif','Developer')
,(2,'Axar','Jr.Developer')
,(3,'Amit','Project Manager')


SELECT * FROM #Employee

-- Sample New Data

MERGE #Employee Target
USING ( VALUES (1,'Arif','Jr.Developer'),(4,'New Record','Tester'),(3,'Amit','Project Manager')) AS Source(EmpID,FullName,Designation)
on Target.EmpId=source.EmpID

WHEN MATCHED AND Target.Designation<>Source.Designation THEN 

UPDATE SET Target.Designation=Source.Designation

WHEN NOT MATCHED BY TARGET THEN 

INSERT (EmpID,FullName,Designation)
VALUES (Source.EmpID,Source.FullName,Source.Designation);

SELECT * FROM #Employee

DROP TABLE #Employee

Open in new window


Screenshot shows Update and new entry.

Untitled.png
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
eluceroAuthor Commented:
Thank you
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Databases

From novice to tech pro — start learning today.