nutnut
asked on
SSIS Data Flow Task EXCLUDE Duplicate Data
Hi,
I am very new to SSIS. I have a data flow task.
My Source is: ServerA.TableA,
My Destination: ServerB.TableB.
I wish to only bring data from ServerA.TableA where it is NOT already in ServerB.TableB.
How do I do this please?
Thnaks
nutnut
I am very new to SSIS. I have a data flow task.
My Source is: ServerA.TableA,
My Destination: ServerB.TableB.
I wish to only bring data from ServerA.TableA where it is NOT already in ServerB.TableB.
How do I do this please?
Thnaks
nutnut
Is there other data in table B or is it just an "older" copy of table A if it is you could just clear the contents before the transfer in table A data
ASKER
Hi yes there is data already in tableB and so unfortunately I cannot delete it
are you using dts or ssis
ASKER
SSIS
if you create the sql for the data transfer based on an NOT EXISTS command
INSERT INTO dbo.Target
SELECT *
FROM dbo.Source AS s
WHERE NOT EXISTS
(
SELECT 1
FROM dbo.Target AS t
WHERE t.title_id = s.title_id
)
this should add only new data
ASKER
They are on different servers though, how would I build this into a Data flow Task
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thats great thanks but cannot get it to work, I think the prob is with my Condition Split, can you elaborate on what I should be doing within it please, what conditions should I be putting within it?
ok
Consider
ServerA.TableA have ID (PK) and Description1 , Description2 these are the fileds....
same in ServerB.TableB
In lookup map both primary keys.then
you are selecteing ServerB.TableB Description1 , Description2 rename it as Description1_out , Description2_out as shown in the image attached...
then in the condition split add the condition like
DESCRIPTION1 != TRIM(DESCRIPTION1_out) && DESCRIPTION2 != TRIM(DESCRIPTION2_out)
as shown in second image
then connect it to destination
condition.JPG
lookup.JPG
Consider
ServerA.TableA have ID (PK) and Description1 , Description2 these are the fileds....
same in ServerB.TableB
In lookup map both primary keys.then
you are selecteing ServerB.TableB Description1 , Description2 rename it as Description1_out , Description2_out as shown in the image attached...
then in the condition split add the condition like
DESCRIPTION1 != TRIM(DESCRIPTION1_out) && DESCRIPTION2 != TRIM(DESCRIPTION2_out)
as shown in second image
then connect it to destination
condition.JPG
lookup.JPG