Solved

-- Need to improve the current bcp solution --

Posted on 2004-08-17
6
529 Views
Last Modified: 2012-06-27
Hi,
Currently I'm copying data from multiple tables on one server to a different server using bcp, which works fine. However because I have to make sure that each record that is copied it's not a duplicate that already exists on my second server and if I find one I write it to a file so I know what duplicates I got. Anyhow, because bcp does not support duplicate record error checking which would give me the duplicate record itself, in order to do this using bcp I have to check each record one at the time for any error messages  like "bcp copy in failed" which would indicate to me that I have an dup. record. (using ksh).
Everything works fine, but the problem is speed, often I have to copy 50,000 records every day and it takes about 3 hours to do that using my current method. I need something that will do the job in less than an hour.
I can not use bulk insert because it is disabled on the servers and I don't have the rights to change it.
Basically what I need is to copy records from one server to another, and any duplicate records should be written to a file. Any suggestions are welcomed.

Thanks
0
Comment
Question by:danidak
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 14

Accepted Solution

by:
Jan_Franek earned 250 total points
ID: 11828541
I can imagine 2 ways:

1. bcp out data from destination table and use some text processing tools (diff ?) to compare this file with bcp dump of source table. It should not be too difficult to create file with duplicate records

2. bcp in your source data into some temporary table on destination server and then compare this temporary table with destination table to identify duplicate records - then you can bcp out just duplicate records from temporary table to create your file.
0
 
LVL 2

Expert Comment

by:mansoor_a_khan
ID: 11831936
I would suggest create a unique index on the temp table and then BCP the data in, and if you are about to insert dup keys in the table u will get the dup error and u can trace with the no rows inserted count to track whe the dup key is.

However this solution combined with DIFF will to the trick for u.
0
 
LVL 6

Assisted Solution

by:ChrisKing
ChrisKing earned 250 total points
ID: 11832002
I agree with Jan_Frank's solutions, but would like to add:

3. Use CIS for cross server access. Register the first server on the second server and then you can directly access both source and destination tables with a series of statements like:

insert into table1 (a,b,c)
select a,b,c
from remotesvr.db..table1 src
where not exists (select * from table1 dest where dest.pk = src.pk)

etc
0
 
LVL 10

Expert Comment

by:bret
ID: 11835834
I would also recommend the CIS approach if the servers are on the same network.

-bret
0

Featured Post

Free Tool: Postgres Monitoring System

A PHP and Perl based system to collect and display usage statistics from PostgreSQL databases.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article was originally published on Monitis Blog, you can check it  here . If you have responsibility for software in production, I bet you’d like to know more about it. I don’t mean that you’d like an extra peek into the bowels of the sour…
This article was initially published on Monitis Blog, you can read it here . When it comes to deciding which approach to website performance monitoring is best for your business, unfortunately, like so many options in life . . . it depends. In t…
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…

730 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question