Link to home
Start Free TrialLog in
Avatar of nutnut
nutnutFlag for United Kingdom of Great Britain and Northern Ireland

asked on

SSIS Data Flow Task EXCLUDE Duplicate Data

Hi,

I am very new to SSIS.  I have a data flow task.  
My Source is: ServerA.TableA,
My Destination: ServerB.TableB.  

I wish to only bring data from ServerA.TableA where it is NOT already in ServerB.TableB.

How do I do this please?

Thnaks

nutnut
Avatar of bradleys40
bradleys40

Is there other data in table B or is it just an "older" copy of table A if it is you could just clear the contents before the transfer in table A data
Avatar of nutnut

ASKER

Hi yes there is data already in tableB and so unfortunately I cannot delete it
are you using dts or ssis
Avatar of nutnut

ASKER

SSIS

if you create the sql for the data transfer based on an  NOT EXISTS command
INSERT INTO dbo.Target
SELECT *
FROM dbo.Source AS s
WHERE NOT EXISTS
(
      SELECT 1
      FROM dbo.Target AS t
      WHERE t.title_id = s.title_id
)

this should add only new data
Avatar of nutnut

ASKER

They are on different servers though, how would I build this into a Data flow Task
ASKER CERTIFIED SOLUTION
Avatar of Pratima
Pratima
Flag of India image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of nutnut

ASKER

Thats great thanks but cannot get it to work, I think the prob is with my Condition Split, can you elaborate on what I should be doing within it please, what conditions should I be putting within it?
ok

Consider

ServerA.TableA  have ID (PK) and Description1 , Description2  these are the fileds....
same in ServerB.TableB

In lookup map both primary keys.then

you are selecteing ServerB.TableB Description1 , Description2 rename it as Description1_out , Description2_out as shown in the image attached...

then in the condition split add the condition like
DESCRIPTION1 != TRIM(DESCRIPTION1_out) && DESCRIPTION2 != TRIM(DESCRIPTION2_out)

as shown in second image

then connect it to destination
condition.JPG
lookup.JPG