Solved

-- Need to improve the current bcp solution --

Posted on 2004-08-17
6
524 Views
Last Modified: 2012-06-27
Hi,
Currently I'm copying data from multiple tables on one server to a different server using bcp, which works fine. However because I have to make sure that each record that is copied it's not a duplicate that already exists on my second server and if I find one I write it to a file so I know what duplicates I got. Anyhow, because bcp does not support duplicate record error checking which would give me the duplicate record itself, in order to do this using bcp I have to check each record one at the time for any error messages  like "bcp copy in failed" which would indicate to me that I have an dup. record. (using ksh).
Everything works fine, but the problem is speed, often I have to copy 50,000 records every day and it takes about 3 hours to do that using my current method. I need something that will do the job in less than an hour.
I can not use bulk insert because it is disabled on the servers and I don't have the rights to change it.
Basically what I need is to copy records from one server to another, and any duplicate records should be written to a file. Any suggestions are welcomed.

Thanks
0
Comment
Question by:danidak
6 Comments
 
LVL 14

Accepted Solution

by:
Jan_Franek earned 250 total points
Comment Utility
I can imagine 2 ways:

1. bcp out data from destination table and use some text processing tools (diff ?) to compare this file with bcp dump of source table. It should not be too difficult to create file with duplicate records

2. bcp in your source data into some temporary table on destination server and then compare this temporary table with destination table to identify duplicate records - then you can bcp out just duplicate records from temporary table to create your file.
0
 
LVL 2

Expert Comment

by:mansoor_a_khan
Comment Utility
I would suggest create a unique index on the temp table and then BCP the data in, and if you are about to insert dup keys in the table u will get the dup error and u can trace with the no rows inserted count to track whe the dup key is.

However this solution combined with DIFF will to the trick for u.
0
 
LVL 6

Assisted Solution

by:ChrisKing
ChrisKing earned 250 total points
Comment Utility
I agree with Jan_Frank's solutions, but would like to add:

3. Use CIS for cross server access. Register the first server on the second server and then you can directly access both source and destination tables with a series of statements like:

insert into table1 (a,b,c)
select a,b,c
from remotesvr.db..table1 src
where not exists (select * from table1 dest where dest.pk = src.pk)

etc
0
 
LVL 10

Expert Comment

by:bret
Comment Utility
I would also recommend the CIS approach if the servers are on the same network.

-bret
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Suggested Solutions

Possible fixes for Windows 7 and Windows Server 2008 updating problem. Solutions mentioned are from Microsoft themselves. I started a case with them from our Microsoft Silver Partner option to open a case and get direct support from Microsoft. If s…
Veeam Backup & Replication has added a new integration – Veeam Backup for Microsoft Office 365.  In this blog, we will discuss how you can benefit from Office 365 email backup with the Veeam’s new product and try to shed some light on the needs and …
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…
When you create an app prototype with Adobe XD, you can insert system screens -- sharing or Control Center, for example -- with just a few clicks. This video shows you how. You can take the full course on Experts Exchange at http://bit.ly/XDcourse.

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now