SSIS and duplicate rows

Posted on 2014-07-17
Medium Priority
Last Modified: 2016-02-11
I download an XML file from the internet every 30 minutes, this XML file is then split up into 5 different tables. This all works fine, however I have now created 5 new tables mirroring the other 5 with _U appended to the name. What I want to do is every time I do this download, I want to wipe the original tables, load the data in and then only move the records that don't exist in the 5 tables with _U into them, so I only want any data duplication in the _U tables.

What is the easiest way to do this with a bit of an example please.
Question by:CaptainGiblets
1 Comment
LVL 18

Accepted Solution

Jerry Miller earned 2000 total points
ID: 40202287
I would do the insert through either one stored procedure that contains the five insert statements or five stored procedures. It really depends on how you want to separate the inserts. But if you do a NOT EXISTS, it will only insert the records that are not duplicates. Which is what I am assuming that you want. You said in the question that you want the duplicates. If that is the case and you want the duplicates, simply remove the NOT from the criteria.

INSERT <tableName_U> (<fieldNames>)
SELECT <fieldnames>
FROM <tableName> a
   NOT EXISTS (SELECT * FROM <tableName_U> b
              WHERE a.uniqueID = b.uniqueID)

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
When trying to connect from SSMS v17.x to a SQL Server Integration Services 2016 instance or previous version, you get the error “Connecting to the Integration Services service on the computer failed with the following error: 'The specified service …
This video shows, step by step, how to configure Oracle Heterogeneous Services via the Generic Gateway Agent in order to make a connection from an Oracle session and access a remote SQL Server database table.
Viewers will learn how to use the UPDATE and DELETE statements to change or remove existing data from their tables. Make a table: Update a specific column given a specific row using the UPDATE statement: Remove a set of values using the DELETE s…
Suggested Courses

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question