[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now

x
?
Solved

SQL to SQL Data transfer speed

Posted on 1998-07-23
4
Medium Priority
?
290 Views
Last Modified: 2010-03-19
Background:
I am converting a huge DBase db into SQL.  What I have done is extracted data from DBase into non-delimited files and BCPed the data into a SQL db - HMS(Note that this db has the same table structure as the one in DBase).  I have a db in SQL - RxOne where some of the table structures match with the DBase db.  The aim is to get the data from HMS to RxOne(both the dbs are on the same server) in the right order i.e maintaining the FK relationships etc.  So here is what I am doing.

1. Read data from HMS using Stored Procedures
2. Map the fields in VB
3. Insert data into RxOne using Stored Procedures

Now to do this for 20000 records it takes about 70 minutes.  The other thing I have tried is export into a tab delimited file so that I can BCP from there, this takes about 60 mins.

Question:
I have a few million records to convert, does anyone know how to speed up this process?
0
Comment
Question by:arvindp
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
4 Comments
 

Author Comment

by:arvindp
ID: 1089185
Edited text of question
0
 

Author Comment

by:arvindp
ID: 1089186
Edited text of question
0
 
LVL 9

Accepted Solution

by:
cymbolic earned 300 total points
ID: 1089187
SQL Server is disappointingly slow when creating records, particularly if you are coming from a .DBF environment, where it happens much faster.  However, slow inserts are characteristic of relational databases that hold and enforce constraints and foreign keys during the insert process.

But there are a few things that will give you the best throughput of what is available.  You've already taken a big bite out of performance by using stored procedures, here are a few other things that might help:

1) try usng transactions in your vb migration app.  Using .begintrans/.committrans methods allows you to vary the number of rows dumped to the Transaction log before SQL Server processes them into the database.  Varying the number of records between .committrans methods can improve your throughput to SQL Server.

2) Periodically perform Update Statistics for the database during the migration.  This will reoptimize your query paths for your stored procedures as the database grows.

3) Use VB5. RDO or ADO, SP3.  Consider using SQLOLE (SQL-DMO) as an alternative to speed up the access.  SQL-DMO is available by including it in your project as a reference, and the object model plus samples are on MSDN.

4) if using BCP, note that there is a database option on SQL Server for Fast BCP, that bypasses the use of the transaction log


0
 

Author Comment

by:arvindp
ID: 1089188
Although the answer didn't really help me, since you have responded first and gave me some direction on the problem I would like to award these points to you.

Thanks for your help.
0

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Ever needed a SQL 2008 Database replicated/mirrored/log shipped on another server but you can't take the downtime inflicted by initial snapshot or disconnect while T-logs are restored or mirror applied? You can use SQL Server Initialize from Backup…
When trying to connect from SSMS v17.x to a SQL Server Integration Services 2016 instance or previous version, you get the error “Connecting to the Integration Services service on the computer failed with the following error: 'The specified service …
Via a live example, show how to set up a backup for SQL Server using a Maintenance Plan and how to schedule the job into SQL Server Agent.
Viewers will learn how to use the SELECT statement in SQL to return specific rows and columns, with various degrees of sorting and limits in place.

656 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question