?
Solved

-- Need to improve the current bcp solution --

Posted on 2004-08-17
6
Medium Priority
?
531 Views
Last Modified: 2012-06-27
Hi,
Currently I'm copying data from multiple tables on one server to a different server using bcp, which works fine. However because I have to make sure that each record that is copied it's not a duplicate that already exists on my second server and if I find one I write it to a file so I know what duplicates I got. Anyhow, because bcp does not support duplicate record error checking which would give me the duplicate record itself, in order to do this using bcp I have to check each record one at the time for any error messages  like "bcp copy in failed" which would indicate to me that I have an dup. record. (using ksh).
Everything works fine, but the problem is speed, often I have to copy 50,000 records every day and it takes about 3 hours to do that using my current method. I need something that will do the job in less than an hour.
I can not use bulk insert because it is disabled on the servers and I don't have the rights to change it.
Basically what I need is to copy records from one server to another, and any duplicate records should be written to a file. Any suggestions are welcomed.

Thanks
0
Comment
Question by:danidak
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 14

Accepted Solution

by:
Jan Franek earned 1000 total points
ID: 11828541
I can imagine 2 ways:

1. bcp out data from destination table and use some text processing tools (diff ?) to compare this file with bcp dump of source table. It should not be too difficult to create file with duplicate records

2. bcp in your source data into some temporary table on destination server and then compare this temporary table with destination table to identify duplicate records - then you can bcp out just duplicate records from temporary table to create your file.
0
 
LVL 2

Expert Comment

by:mansoor_a_khan
ID: 11831936
I would suggest create a unique index on the temp table and then BCP the data in, and if you are about to insert dup keys in the table u will get the dup error and u can trace with the no rows inserted count to track whe the dup key is.

However this solution combined with DIFF will to the trick for u.
0
 
LVL 6

Assisted Solution

by:ChrisKing
ChrisKing earned 1000 total points
ID: 11832002
I agree with Jan_Frank's solutions, but would like to add:

3. Use CIS for cross server access. Register the first server on the second server and then you can directly access both source and destination tables with a series of statements like:

insert into table1 (a,b,c)
select a,b,c
from remotesvr.db..table1 src
where not exists (select * from table1 dest where dest.pk = src.pk)

etc
0
 
LVL 10

Expert Comment

by:bret
ID: 11835834
I would also recommend the CIS approach if the servers are on the same network.

-bret
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this blog post, we’ll look at how using thread_statistics can cause high memory usage.
We are witnesses that everyone is saying that our children shouldn't "play" with a technology because it is dangerous. This article is going to prove that they are wrong.
This tutorial will teach you the special effect of super speed similar to the fictional character Wally West aka "The Flash" After Shake : http://www.videocopilot.net/presets/after_shake/ All lightning effects with instructions : http://www.mediaf…
How to fix incompatible JVM issue while installing Eclipse While installing Eclipse in windows, got one error like above and unable to proceed with the installation. This video describes how to successfully install Eclipse. How to solve incompa…
Suggested Courses

801 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question