Solved

-- Need to improve the current bcp solution --

Posted on 2004-08-17
6
528 Views
Last Modified: 2012-06-27
Hi,
Currently I'm copying data from multiple tables on one server to a different server using bcp, which works fine. However because I have to make sure that each record that is copied it's not a duplicate that already exists on my second server and if I find one I write it to a file so I know what duplicates I got. Anyhow, because bcp does not support duplicate record error checking which would give me the duplicate record itself, in order to do this using bcp I have to check each record one at the time for any error messages  like "bcp copy in failed" which would indicate to me that I have an dup. record. (using ksh).
Everything works fine, but the problem is speed, often I have to copy 50,000 records every day and it takes about 3 hours to do that using my current method. I need something that will do the job in less than an hour.
I can not use bulk insert because it is disabled on the servers and I don't have the rights to change it.
Basically what I need is to copy records from one server to another, and any duplicate records should be written to a file. Any suggestions are welcomed.

Thanks
0
Comment
Question by:danidak
6 Comments
 
LVL 14

Accepted Solution

by:
Jan_Franek earned 250 total points
ID: 11828541
I can imagine 2 ways:

1. bcp out data from destination table and use some text processing tools (diff ?) to compare this file with bcp dump of source table. It should not be too difficult to create file with duplicate records

2. bcp in your source data into some temporary table on destination server and then compare this temporary table with destination table to identify duplicate records - then you can bcp out just duplicate records from temporary table to create your file.
0
 
LVL 2

Expert Comment

by:mansoor_a_khan
ID: 11831936
I would suggest create a unique index on the temp table and then BCP the data in, and if you are about to insert dup keys in the table u will get the dup error and u can trace with the no rows inserted count to track whe the dup key is.

However this solution combined with DIFF will to the trick for u.
0
 
LVL 6

Assisted Solution

by:ChrisKing
ChrisKing earned 250 total points
ID: 11832002
I agree with Jan_Frank's solutions, but would like to add:

3. Use CIS for cross server access. Register the first server on the second server and then you can directly access both source and destination tables with a series of statements like:

insert into table1 (a,b,c)
select a,b,c
from remotesvr.db..table1 src
where not exists (select * from table1 dest where dest.pk = src.pk)

etc
0
 
LVL 10

Expert Comment

by:bret
ID: 11835834
I would also recommend the CIS approach if the servers are on the same network.

-bret
0

Featured Post

Active Directory Webinar

We all know we need to protect and secure our privileges, but where to start? Join Experts Exchange and ManageEngine on Tuesday, April 11, 2017 10:00 AM PDT to learn how to track and secure privileged users in Active Directory.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Giving access to ESXi shell console is always an issue for IT departments to other Teams, or Projects. We need to find a way so that teams can use ESXTOP for their POCs, or tests without giving them the access to ESXi host shell console with a root …
This article describes how to reset your Windows 10 password when you've forgotten it.
Two types of users will appreciate AOMEI Backupper Pro: 1 - Those with PCIe drives (and haven't found cloning software that works on them). 2 - Those who want a fast clone of their boot drive (no re-boots needed) and it can clone your drive wh…
The Email Laundry PDF encryption service allows companies to send confidential encrypted  emails to anybody. The PDF document can also contain attachments that are embedded in the encrypted PDF. The password is randomly generated by The Email Laundr…

828 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question