Solved

SQL to SQL Data transfer speed

Posted on 1998-07-23
4
261 Views
Last Modified: 2010-03-19
Background:
I am converting a huge DBase db into SQL.  What I have done is extracted data from DBase into non-delimited files and BCPed the data into a SQL db - HMS(Note that this db has the same table structure as the one in DBase).  I have a db in SQL - RxOne where some of the table structures match with the DBase db.  The aim is to get the data from HMS to RxOne(both the dbs are on the same server) in the right order i.e maintaining the FK relationships etc.  So here is what I am doing.

1. Read data from HMS using Stored Procedures
2. Map the fields in VB
3. Insert data into RxOne using Stored Procedures

Now to do this for 20000 records it takes about 70 minutes.  The other thing I have tried is export into a tab delimited file so that I can BCP from there, this takes about 60 mins.

Question:
I have a few million records to convert, does anyone know how to speed up this process?
0
Comment
Question by:arvindp
  • 3
4 Comments
 

Author Comment

by:arvindp
ID: 1089185
Edited text of question
0
 

Author Comment

by:arvindp
ID: 1089186
Edited text of question
0
 
LVL 9

Accepted Solution

by:
cymbolic earned 100 total points
ID: 1089187
SQL Server is disappointingly slow when creating records, particularly if you are coming from a .DBF environment, where it happens much faster.  However, slow inserts are characteristic of relational databases that hold and enforce constraints and foreign keys during the insert process.

But there are a few things that will give you the best throughput of what is available.  You've already taken a big bite out of performance by using stored procedures, here are a few other things that might help:

1) try usng transactions in your vb migration app.  Using .begintrans/.committrans methods allows you to vary the number of rows dumped to the Transaction log before SQL Server processes them into the database.  Varying the number of records between .committrans methods can improve your throughput to SQL Server.

2) Periodically perform Update Statistics for the database during the migration.  This will reoptimize your query paths for your stored procedures as the database grows.

3) Use VB5. RDO or ADO, SP3.  Consider using SQLOLE (SQL-DMO) as an alternative to speed up the access.  SQL-DMO is available by including it in your project as a reference, and the object model plus samples are on MSDN.

4) if using BCP, note that there is a database option on SQL Server for Fast BCP, that bypasses the use of the transaction log


0
 

Author Comment

by:arvindp
ID: 1089188
Although the answer didn't really help me, since you have responded first and gave me some direction on the problem I would like to award these points to you.

Thanks for your help.
0

Featured Post

Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I wrote this interesting script that really help me find jobs or procedures when working in a huge environment. I could I have written it as a Procedure but then I would have to have it on each machine or have a link to a server-related search that …
Ever wondered why sometimes your SQL Server is slow or unresponsive with connections spiking up but by the time you go in, all is well? The following article will show you how to install and configure a SQL job that will send you email alerts includ…
This videos aims to give the viewer a basic demonstration of how a user can query current session information by using the SYS_CONTEXT function
Viewers will learn how to use the SELECT statement in SQL and will be exposed to the many uses the SELECT statement has.

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

27 Experts available now in Live!

Get 1:1 Help Now