I have a DTS package that imports several tables from an ODBC compilant DB into my SQL data warehouse. The rows are deleted in the destination tables before import.
A few of the tables are large (over 1,000,000 rows). These are history tables such as INVOICE_HISTORY_DETAILS. I want to speed up the daily import by only brining in data that doesen't exist in the destination table or at a minimum use a query to minimize the date range (I don't want to import the last 6 years of history every day).
Of course if I do import every row where inv_date > "some recent date" how do I make sure that only data that doesn't exist is appended to the table?