I've coded an EXE which opens an ADO connection to various OLEDB/ODBC data sources and transfers the records, depending on the SQL statement used record at a time into a SQL server database.
Problem is, its very slow!
A typical example of the SQL used is, "SELECT * WHERE rowDate = TODAY", and this statement will be executed every half hour so that the table is populated Intra-Day.
We've executed DTS packages to do the same thing and have noticed the significant speed advantages you can get with DTS, problem is, the DTS job is successful the first time, but will always error out the second time as some records are already in the database (and we need them to update every half hour).
Deleting and re-importing isnt an option, some of the tables are huge and we just get SQL Timeouts (even at 300 seconds).
Is there a way to use DTS to transfer the data and ignore the records with a Primary Key violation (ie duplicates)?
Or can we say do an insert into SQL Database where Unique identifer not in table? Ie. SELECT * FROM (database A) where ID not in (Database B)? However I cant see how this will work as all the datasources are completely different and none of the primary keys will be the same.
Any help would be greatly appreciated.!