Solved

Loading rows from a view (no index)

Posted on 2013-06-19
7
239 Views
Last Modified: 2016-02-11
hi guys

we have a remote view from which we are pulling data into our local table.

The remote view is :REMOTE_CUSTOMER
and our local table is :LOCAL_CUSTOMER

This is what we are doing right now.

We first delete data from our local table
DELETE Top (50000) FROM LOCAL_CUSTOMER where Id = '209'

Then we do a select to pull the data from the view (the remote view does not have any indexes, can we assign a index to view? Also the remote view doesnt have a primary key)

SELECT FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp FROM remote.dbview.REMOTE_CUSTOMER

then we insert into our local table

INSERT INTO LOCAL_CUSTOMER (FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp,Id)
VALUES(:FirstName, :LastName, :BusinessDescription, :ProductDescription, :ProductSourceMap, :updateBy,:updateTimestamp,:Id)

There are over a million rows in the REMOTE_CUSTOMER view and this process takes around 30 minutes to get the data from the view and populate into our table.

Is there a way to improve the speed of this entire process?

thanks
0
Comment
Question by:royjayd
  • 4
  • 3
7 Comments
 
LVL 1

Expert Comment

by:yechan
ID: 39260305
If you have any indexes on the LOCAL_CUSTOMER table my suggestion is to drop those indexes, load your data, and then re-create those indexes.
0
 

Author Comment

by:royjayd
ID: 39265735
ok, i dropped the index, but it takes the same time to insert the rows into
our LOCAL_CUSTOMER table (around 30 minutes)
Any other advice other than dropping the indexes ?

Can i create multiple threads and do parallel processing in sql server 2008 to increase the speed?

thanks
0
 
LVL 1

Expert Comment

by:yechan
ID: 39265796
Do you have a WHERE clause against the REMOTE_CUSTOMER table/view?  If yes, then I think adding an index would be very beneficial.

Other things to check for is that the LOCAL_CUSTOMER table does not have any triggers.  Change the "Recovery Level" of the database to Simple.  

Using SSMS to create a multiple threads would be difficult unless you open several tabs =)

I think if you want to do some parallel processing, using SSIS is the way to go.
0
Optimizing Cloud Backup for Low Bandwidth

With cloud storage prices going down a growing number of SMBs start to use it for backup storage. Unfortunately, business data volume rarely fits the average Internet speed. This article provides an overview of main Internet speed challenges and reveals backup best practices.

 

Author Comment

by:royjayd
ID: 39265899
hi

actually there is no Where clause in the select statement to query the view.
its just a simple sql

SELECT FirstName,LastName,BusinessDescription,ProductDescription,ProductSourceMap,updateBy,updateTimestamp FROM remote.dbview.REMOTE_CUSTOMER


Also
LOCAL_CUSTOMER  also does not have any triggers
Recovery Model of the database is Simple

Can you tell me how i can do SSIS  for parallel processing?

thanks.
0
 
LVL 1

Accepted Solution

by:
yechan earned 500 total points
ID: 39266645
In SSIS, for example, you could have two "Execute SQL" tasks and each task would have a query that would look something like the following:

DECLARE @myCount AS INT =( SELECT COUNT(*) FROM REMOTE_CUSTOMER );

WITH myCTE as
(
   SELECT 	
      col1, 
      col2, 
      etc,
      rn = ROW_NUMBER() OVER (ORDER BY (SELECT 0))
FROM 
     REMOTE_CUSTOMER
)

INSERT INTO LOCAL_CUSTOMER
SELECT  * FROM mycte
WHERE rn BETWEEN 1  AND 0.5 * @myCount
--WHERE rn BETWEEN 0.5 * @myCount AND @myCount  query to be used in the second SSIS task

Open in new window

0
 

Author Comment

by:royjayd
ID: 39267024
ok thanks

So is the above sql being run by multiple threads ?
Can you explain a little what the sql does?
0
 
LVL 1

Expert Comment

by:yechan
ID: 39270986
What the SQL code does is create a  Common Table Expression (CTE for short) that selects all REMOTE_CUSTOMER records along with a row number using the ROW_NUMBER() function.

After that, the code inserts the first 50% of the records in the myCTE table into the LOCAL_CUSTOMER table.  You can execute this logic by inserting this SQL code into a "Execute SQL Task".
0

Featured Post

Best Practices: Disaster Recovery Testing

Besides backup, any IT division should have a disaster recovery plan. You will find a few tips below relating to the development of such a plan and to what issues one should pay special attention in the course of backup planning.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

My client has a dictionary table. They're defining a list of standard naming convention. Now, they are requiring my team to provide us a mechanism how to match new incoming data with existing data in their system.
This article explains how to reset the password of the sa account on a Microsoft SQL Server.  The steps in this article work in SQL 2005, 2008, 2008 R2, 2012, 2014 and 2016.
This video shows how to use Hyena, from SystemTools Software, to bulk import 100 user accounts from an external text file. View in 1080p for best video quality.
I've attached the XLSM Excel spreadsheet I used in the video and also text files containing the macros used below. https://filedb.experts-exchange.com/incoming/2017/03_w12/1151775/Permutations.txt https://filedb.experts-exchange.com/incoming/201…

679 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question