Solved

Copy large amount of records via stored procedure?

Posted on 2007-11-26
7
194 Views
Last Modified: 2010-04-21
Hello,

I have a view which returns close to 5 million rows of data.  I need to copy this data into a table of the same structure.

Is it possible to create a stored procedure which copies the data over in chunks (so it doesn't timeout)?

If so can you give an example or a link to a tutorial?

Thanks,
-Torrwin
0
Comment
Question by:Torrwin
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
7 Comments
 
LVL 29

Accepted Solution

by:
QPR earned 500 total points
ID: 20352902
Is this a one off?
You might want to consider creating a SSIS package.
Or use BCP
0
 
LVL 13

Author Comment

by:Torrwin
ID: 20352917
I'm not sure what a one off is, but I need to be able to execute it from a VB.NET app, so I don't think those are options.
0
 
LVL 29

Expert Comment

by:QPR
ID: 20352999
A "one off" is an event you only plan to do once.

http://www.vbrad.com/article.aspx?id=10
Explains the benefits of bcp over using stored procedures (speed) and has some vb class/exe that maybe you could include in your project?
0
Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

 
LVL 13

Author Comment

by:Torrwin
ID: 20353213
No, this will happen several times a month.

Also, BCP appears to need the source to be a file, not a view.
0
 
LVL 12

Expert Comment

by:kselvia
ID: 20353951

select * into newTable from MyView

will transfer all rows into newTable, but 5 million rows is relatively small as far as I'm concerned, and that statement would not "time out" in any case. It will run until it completes, which should be 3-15 seconds depending on the complexity of the view.

The answer is yes you could make a stored procedure return chunks or ranges of data, but the problem probably lies in the view and retrieving ranges from that is probably not going to improve performance, and making a stored procedure return portions of data from it is probably not the best approach.

To get the first 100 rows

select top 1000 * from mytable

to get the 2nd 1000 rows

select top 1000 * from
( select top 2000 * from mytable order by ID desc )
order by ID ASC

where ID is the unique key in the result set

Make 5 versions of that statement, each retrieving 1 million rows.  I can pretty much guarantee that will perform far worse than what you have now though.

Can we see your view? Maybe we can suggest ways to improve the performance.

And yes you can use a view with bcp using the queryout option.

0
 
LVL 29

Expert Comment

by:QPR
ID: 20353969
0
 
LVL 13

Author Closing Comment

by:Torrwin
ID: 31411073
Basically, the BCP suggestion led me to discover the SQLBulkCopy commands which are BCP-based.
0

Featured Post

NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction SQL Server Integration Services can read XML files, that’s known by every BI developer.  (If you didn’t, don’t worry, I’m aiming this article at newcomers as well.) But how far can you go?  When does the XML Source component become …
This article explains how to reset the password of the sa account on a Microsoft SQL Server.  The steps in this article work in SQL 2005, 2008, 2008 R2, 2012, 2014 and 2016.
This videos aims to give the viewer a basic demonstration of how a user can query current session information by using the SYS_CONTEXT function
Via a live example, show how to setup several different housekeeping processes for a SQL Server.

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question