Solved

-- Need to improve the current bcp solution --

Posted on 2004-08-17
6
530 Views
Last Modified: 2012-06-27
Hi,
Currently I'm copying data from multiple tables on one server to a different server using bcp, which works fine. However because I have to make sure that each record that is copied it's not a duplicate that already exists on my second server and if I find one I write it to a file so I know what duplicates I got. Anyhow, because bcp does not support duplicate record error checking which would give me the duplicate record itself, in order to do this using bcp I have to check each record one at the time for any error messages  like "bcp copy in failed" which would indicate to me that I have an dup. record. (using ksh).
Everything works fine, but the problem is speed, often I have to copy 50,000 records every day and it takes about 3 hours to do that using my current method. I need something that will do the job in less than an hour.
I can not use bulk insert because it is disabled on the servers and I don't have the rights to change it.
Basically what I need is to copy records from one server to another, and any duplicate records should be written to a file. Any suggestions are welcomed.

Thanks
0
Comment
Question by:danidak
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 14

Accepted Solution

by:
Jan_Franek earned 250 total points
ID: 11828541
I can imagine 2 ways:

1. bcp out data from destination table and use some text processing tools (diff ?) to compare this file with bcp dump of source table. It should not be too difficult to create file with duplicate records

2. bcp in your source data into some temporary table on destination server and then compare this temporary table with destination table to identify duplicate records - then you can bcp out just duplicate records from temporary table to create your file.
0
 
LVL 2

Expert Comment

by:mansoor_a_khan
ID: 11831936
I would suggest create a unique index on the temp table and then BCP the data in, and if you are about to insert dup keys in the table u will get the dup error and u can trace with the no rows inserted count to track whe the dup key is.

However this solution combined with DIFF will to the trick for u.
0
 
LVL 6

Assisted Solution

by:ChrisKing
ChrisKing earned 250 total points
ID: 11832002
I agree with Jan_Frank's solutions, but would like to add:

3. Use CIS for cross server access. Register the first server on the second server and then you can directly access both source and destination tables with a series of statements like:

insert into table1 (a,b,c)
select a,b,c
from remotesvr.db..table1 src
where not exists (select * from table1 dest where dest.pk = src.pk)

etc
0
 
LVL 10

Expert Comment

by:bret
ID: 11835834
I would also recommend the CIS approach if the servers are on the same network.

-bret
0

Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

There are times when we need to generate a report on the inbox rules, where users have set up forwarding externally in their mailbox. In this article, I will be sharing a script I wrote to generate the report in CSV format.
This article provides a convenient collection of links to Microsoft provided Security Patches for operating systems that have reached their End of Life support cycle. Included operating systems covered by this article are Windows XP,  Windows Server…
In this brief tutorial Pawel from AdRem Software explains how you can quickly find out which services are running on your network, or what are the IP addresses of servers responsible for each service. Software used is freeware NetCrunch Tools (https…
Monitoring a network: how to monitor network services and why? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the philosophy behind service monitoring and why a handshake validation is critical in network monitoring. Software utilized …

696 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question