[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 295
  • Last Modified:

SQL to SQL Data transfer speed

Background:
I am converting a huge DBase db into SQL.  What I have done is extracted data from DBase into non-delimited files and BCPed the data into a SQL db - HMS(Note that this db has the same table structure as the one in DBase).  I have a db in SQL - RxOne where some of the table structures match with the DBase db.  The aim is to get the data from HMS to RxOne(both the dbs are on the same server) in the right order i.e maintaining the FK relationships etc.  So here is what I am doing.

1. Read data from HMS using Stored Procedures
2. Map the fields in VB
3. Insert data into RxOne using Stored Procedures

Now to do this for 20000 records it takes about 70 minutes.  The other thing I have tried is export into a tab delimited file so that I can BCP from there, this takes about 60 mins.

Question:
I have a few million records to convert, does anyone know how to speed up this process?
0
arvindp
Asked:
arvindp
  • 3
1 Solution
 
arvindpAuthor Commented:
Edited text of question
0
 
arvindpAuthor Commented:
Edited text of question
0
 
cymbolicCommented:
SQL Server is disappointingly slow when creating records, particularly if you are coming from a .DBF environment, where it happens much faster.  However, slow inserts are characteristic of relational databases that hold and enforce constraints and foreign keys during the insert process.

But there are a few things that will give you the best throughput of what is available.  You've already taken a big bite out of performance by using stored procedures, here are a few other things that might help:

1) try usng transactions in your vb migration app.  Using .begintrans/.committrans methods allows you to vary the number of rows dumped to the Transaction log before SQL Server processes them into the database.  Varying the number of records between .committrans methods can improve your throughput to SQL Server.

2) Periodically perform Update Statistics for the database during the migration.  This will reoptimize your query paths for your stored procedures as the database grows.

3) Use VB5. RDO or ADO, SP3.  Consider using SQLOLE (SQL-DMO) as an alternative to speed up the access.  SQL-DMO is available by including it in your project as a reference, and the object model plus samples are on MSDN.

4) if using BCP, note that there is a database option on SQL Server for Fast BCP, that bypasses the use of the transaction log


0
 
arvindpAuthor Commented:
Although the answer didn't really help me, since you have responded first and gave me some direction on the problem I would like to award these points to you.

Thanks for your help.
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now