?
Solved

SQL to SQL Data transfer speed

Posted on 1998-07-23
4
Medium Priority
?
286 Views
Last Modified: 2010-03-19
Background:
I am converting a huge DBase db into SQL.  What I have done is extracted data from DBase into non-delimited files and BCPed the data into a SQL db - HMS(Note that this db has the same table structure as the one in DBase).  I have a db in SQL - RxOne where some of the table structures match with the DBase db.  The aim is to get the data from HMS to RxOne(both the dbs are on the same server) in the right order i.e maintaining the FK relationships etc.  So here is what I am doing.

1. Read data from HMS using Stored Procedures
2. Map the fields in VB
3. Insert data into RxOne using Stored Procedures

Now to do this for 20000 records it takes about 70 minutes.  The other thing I have tried is export into a tab delimited file so that I can BCP from there, this takes about 60 mins.

Question:
I have a few million records to convert, does anyone know how to speed up this process?
0
Comment
Question by:arvindp
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
4 Comments
 

Author Comment

by:arvindp
ID: 1089185
Edited text of question
0
 

Author Comment

by:arvindp
ID: 1089186
Edited text of question
0
 
LVL 9

Accepted Solution

by:
cymbolic earned 300 total points
ID: 1089187
SQL Server is disappointingly slow when creating records, particularly if you are coming from a .DBF environment, where it happens much faster.  However, slow inserts are characteristic of relational databases that hold and enforce constraints and foreign keys during the insert process.

But there are a few things that will give you the best throughput of what is available.  You've already taken a big bite out of performance by using stored procedures, here are a few other things that might help:

1) try usng transactions in your vb migration app.  Using .begintrans/.committrans methods allows you to vary the number of rows dumped to the Transaction log before SQL Server processes them into the database.  Varying the number of records between .committrans methods can improve your throughput to SQL Server.

2) Periodically perform Update Statistics for the database during the migration.  This will reoptimize your query paths for your stored procedures as the database grows.

3) Use VB5. RDO or ADO, SP3.  Consider using SQLOLE (SQL-DMO) as an alternative to speed up the access.  SQL-DMO is available by including it in your project as a reference, and the object model plus samples are on MSDN.

4) if using BCP, note that there is a database option on SQL Server for Fast BCP, that bypasses the use of the transaction log


0
 

Author Comment

by:arvindp
ID: 1089188
Although the answer didn't really help me, since you have responded first and gave me some direction on the problem I would like to award these points to you.

Thanks for your help.
0

Featured Post

Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article explains how to reset the password of the sa account on a Microsoft SQL Server.  The steps in this article work in SQL 2005, 2008, 2008 R2, 2012, 2014 and 2016.
JSON is being used more and more, besides XML, and you surely wanted to parse the data out into SQL instead of doing it in some Javascript. The below function in SQL Server can do the job for you, returning a quick table with the parsed data.
This videos aims to give the viewer a basic demonstration of how a user can query current session information by using the SYS_CONTEXT function
Viewers will learn how the fundamental information of how to create a table.
Suggested Courses

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question