Solved

Use DTS in VB6 EXE to transfer data when some records already in Table (PK Voilation)

Posted on 2004-09-29
10
309 Views
Last Modified: 2011-10-03
Hi

I've coded an EXE which opens an ADO connection to various OLEDB/ODBC data sources and transfers the records, depending on the SQL statement used record at a time into a SQL server database.

Problem is, its very slow!

A typical example of the SQL used is, "SELECT * WHERE rowDate = TODAY", and this statement will be executed every half hour so that the table is populated Intra-Day.

We've executed DTS packages to do the same thing and have noticed the significant speed advantages you can get with DTS, problem is, the DTS job is successful the first time, but will always error out the second time as some records are already in the database (and we need them to update every half hour).

Deleting and re-importing isnt an option, some of the tables are huge and we just get SQL Timeouts (even at 300 seconds).

Is there a way to use DTS to transfer the data and ignore the records with a Primary Key violation (ie duplicates)?

Or can we say do an insert into SQL Database where Unique identifer not in table?  Ie. SELECT * FROM (database A) where ID not in (Database B)? However I cant see how this will work as all the datasources are completely different and none of the primary keys will be the same.

Any help would be greatly appreciated.!

Thanks experts!
Matt
0
Comment
Question by:razorhazor
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
10 Comments
 
LVL 6

Expert Comment

by:mcp111
ID: 12183455
Use an Activexscript
check for id in the destination table
If dtsdestination("id") exists
set the dtstransformstat_skipinsert flag to skip inserting that row.
0
 
LVL 34

Expert Comment

by:arbert
ID: 12184090
You wouldn't even have to use ActiveX script.  Code ExecuteSQL statements that use INSERT/UPDATE queries based upon the key already existing in the table .
0
 
LVL 34

Expert Comment

by:arbert
ID: 12184103
Several different ways you could do it--to just find not existing records:

SELECT * WHERE rowDate = TODAY"
and yourkey not exist in (select 1 from yourtable table1 inner join yourtable  table2 where table1.primarykey=table2.primarykey)
0
MIM Survival Guide for Service Desk Managers

Major incidents can send mastered service desk processes into disorder. Systems and tools produce the data needed to resolve these incidents, but your challenge is getting that information to the right people fast. Check out the Survival Guide and begin bringing order to chaos.

 
LVL 5

Expert Comment

by:MichaelSFuller
ID: 12186946
For each table transformation implement this type of sql statement replacing the Composite Members with your primary key combination.
 
Select <unnull-column>,... from TableA left outer join TableB
on A.CompositeMember1= B.CompositeMember1 and
on A.CompositeMember2= B.CompositeMember2 and
on A.CompositeMember3= B.CompositeMember3 where
<unnull-column> is Null

This should alleviate your problems unless the primary keys are duplicated from source to source. If this is the case add a column to table to specify the source. Then when you are transforming your information into the destination table, set the source using an alias such as select 1 as iSource, ...from...table name.

Late,
0
 

Author Comment

by:razorhazor
ID: 12187783
Hi guys,

thanks for that, problem with joins though is that the data would be in seperate databases on seperate servers so I wouldnt have thought I would be able to code that in the same SQL statement?

Are we suggesting that I put the data into a temp table to begin with, then run the SQL statement to see whats not there?

Any chance of some example code to get my head round it?!?

Thanks for all your help
Matt
0
 
LVL 34

Accepted Solution

by:
arbert earned 250 total points
ID: 12189641
You can code the join (either via linked server or openrowset), but you need to check the query plan to see if the performance is good enough.
0
 
LVL 5

Assisted Solution

by:MichaelSFuller
MichaelSFuller earned 250 total points
ID: 12189867
Example:

(Select <unnull-column>,... from Table) as A left outer join
(SELECT a.*
FROM OPENROWSET('SQLOLEDB','Servername';'username';'password',
'SELECT * FROM TableB'))
AS B
on A.CompositeMember1= B.CompositeMember1 and
on A.CompositeMember2= B.CompositeMember2 and
on A.CompositeMember3= B.CompositeMember3 where
A.<unnull-column> is Null

0

Featured Post

NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article explains how to reset the password of the sa account on a Microsoft SQL Server.  The steps in this article work in SQL 2005, 2008, 2008 R2, 2012, 2014 and 2016.
In this article we will learn how to fix  “Cannot install SQL Server 2014 Service Pack 2: Unable to install windows installer msi file” error ?
Via a live example, show how to backup a database, simulate a failure backup the tail of the database transaction log and perform the restore.
Viewers will learn how to use the UPDATE and DELETE statements to change or remove existing data from their tables. Make a table: Update a specific column given a specific row using the UPDATE statement: Remove a set of values using the DELETE s…

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question