Solved

SQL to SQL Data transfer speed

Posted on 1998-07-23
4
271 Views
Last Modified: 2010-03-19
Background:
I am converting a huge DBase db into SQL.  What I have done is extracted data from DBase into non-delimited files and BCPed the data into a SQL db - HMS(Note that this db has the same table structure as the one in DBase).  I have a db in SQL - RxOne where some of the table structures match with the DBase db.  The aim is to get the data from HMS to RxOne(both the dbs are on the same server) in the right order i.e maintaining the FK relationships etc.  So here is what I am doing.

1. Read data from HMS using Stored Procedures
2. Map the fields in VB
3. Insert data into RxOne using Stored Procedures

Now to do this for 20000 records it takes about 70 minutes.  The other thing I have tried is export into a tab delimited file so that I can BCP from there, this takes about 60 mins.

Question:
I have a few million records to convert, does anyone know how to speed up this process?
0
Comment
Question by:arvindp
  • 3
4 Comments
 

Author Comment

by:arvindp
ID: 1089185
Edited text of question
0
 

Author Comment

by:arvindp
ID: 1089186
Edited text of question
0
 
LVL 9

Accepted Solution

by:
cymbolic earned 100 total points
ID: 1089187
SQL Server is disappointingly slow when creating records, particularly if you are coming from a .DBF environment, where it happens much faster.  However, slow inserts are characteristic of relational databases that hold and enforce constraints and foreign keys during the insert process.

But there are a few things that will give you the best throughput of what is available.  You've already taken a big bite out of performance by using stored procedures, here are a few other things that might help:

1) try usng transactions in your vb migration app.  Using .begintrans/.committrans methods allows you to vary the number of rows dumped to the Transaction log before SQL Server processes them into the database.  Varying the number of records between .committrans methods can improve your throughput to SQL Server.

2) Periodically perform Update Statistics for the database during the migration.  This will reoptimize your query paths for your stored procedures as the database grows.

3) Use VB5. RDO or ADO, SP3.  Consider using SQLOLE (SQL-DMO) as an alternative to speed up the access.  SQL-DMO is available by including it in your project as a reference, and the object model plus samples are on MSDN.

4) if using BCP, note that there is a database option on SQL Server for Fast BCP, that bypasses the use of the transaction log


0
 

Author Comment

by:arvindp
ID: 1089188
Although the answer didn't really help me, since you have responded first and gave me some direction on the problem I would like to award these points to you.

Thanks for your help.
0

Featured Post

Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

JSON is being used more and more, besides XML, and you surely wanted to parse the data out into SQL instead of doing it in some Javascript. The below function in SQL Server can do the job for you, returning a quick table with the parsed data.
Ever needed a SQL 2008 Database replicated/mirrored/log shipped on another server but you can't take the downtime inflicted by initial snapshot or disconnect while T-logs are restored or mirror applied? You can use SQL Server Initialize from Backup…
Using examples as well as descriptions, and references to Books Online, show the documentation available for date manipulation functions and by using a select few of these functions, show how date based data can be manipulated with these functions.
This videos aims to give the viewer a basic demonstration of how a user can query current session information by using the SYS_CONTEXT function

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question