SSIS and duplicate rows

Posted on 2014-07-17
Last Modified: 2016-02-11
I download an XML file from the internet every 30 minutes, this XML file is then split up into 5 different tables. This all works fine, however I have now created 5 new tables mirroring the other 5 with _U appended to the name. What I want to do is every time I do this download, I want to wipe the original tables, load the data in and then only move the records that don't exist in the 5 tables with _U into them, so I only want any data duplication in the _U tables.

What is the easiest way to do this with a bit of an example please.
Question by:CaptainGiblets
    1 Comment
    LVL 18

    Accepted Solution

    I would do the insert through either one stored procedure that contains the five insert statements or five stored procedures. It really depends on how you want to separate the inserts. But if you do a NOT EXISTS, it will only insert the records that are not duplicates. Which is what I am assuming that you want. You said in the question that you want the duplicates. If that is the case and you want the duplicates, simply remove the NOT from the criteria.

    INSERT <tableName_U> (<fieldNames>)
    SELECT <fieldnames>
    FROM <tableName> a
       NOT EXISTS (SELECT * FROM <tableName_U> b
                  WHERE a.uniqueID = b.uniqueID)

    Featured Post

    How your wiki can always stay up-to-date

    Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
    - Increase transparency
    - Onboard new hires faster
    - Access from mobile/offline

    Join & Write a Comment

    Today, still in the boom of Apple, PC's and products, nearly 50% of the computer users use Windows as graphical operating systems. If you are among those users who love windows, but are grappling to keep the system's hard drive optimized, then you s…
    The Delta outage: 650 cancelled flights, more than 1200 delayed flights, thousands of frustrated customers, tens of millions of dollars in damages – plus untold reputational damage to one of the world’s most trusted airlines. All due to a catastroph…
    Using examples as well as descriptions, and references to Books Online, show the documentation available for date manipulation functions and by using a select few of these functions, show how date based data can be manipulated with these functions.
    Using examples as well as descriptions, and references to Books Online, show the different Recovery Models available in SQL Server and explain, as well as show how full, differential and transaction log backups are performed

    754 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    21 Experts available now in Live!

    Get 1:1 Help Now